Data classification method and device, equipment and storage medium
By determining category tags in Internet user data management and using pre-trained models to automatically acquire and classify user data, the problem that traditional manual data classification methods cannot meet the needs of Internet user data management is solved, and efficient and accurate data classification and management are achieved.
Patent Information
- Application Number
- CN202411217831.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-27
AI Technical Summary
Traditional manual data classification methods cannot meet the data management needs of Internet users, especially when the data volume is large and the variety is wide, it is difficult to achieve efficient and accurate data classification and management.
By determining multiple category tags, the pre-trained extraction model is used to automatically obtain user data from the database metadata, and the category tag of user data is determined based on the matching results, so as to realize automatic filtering and classification of user data.
It reduces manual participation, improves the accuracy and efficiency of data classification, and can flexibly manage user data of different categories of tags, meeting various data management needs, such as data management at different security levels.
Smart Images
Figure CN119202823A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a data classification method, device, equipment and storage medium. Background Art
[0002] With the development of the Internet, user data management has become an important part of Internet business operations. Efficient, reasonable and secure user data management is conducive to enterprises providing users with a better user experience. However, with the increase in Internet users and Internet service types, the amount of user data is huge and diverse, and the traditional manual data classification method can no longer meet the management needs of user data. Summary of the invention
[0003] The embodiments of the present invention provide a data classification method, apparatus, device and storage medium to realize automatic data classification and improve user data management efficiency.
[0004] In a first aspect, an embodiment of the present invention provides a data classification method, the method comprising:
[0005] determining a plurality of category labels for classifying the user data;
[0006] Inputting the database metadata into a pre-trained extraction model to output at least one column of user data to be classified through the extraction model;
[0007] Determine a matching result between at least one target user data included in the target column user data and the multiple category labels, wherein the target column user data is any column user data in the at least one column of target user data;
[0008] According to the matching result, a category label of the target column user data is determined.
[0009] In a second aspect, an embodiment of the present invention provides a data classification device, the device comprising:
[0010] A determination module, used to determine a plurality of category labels for classifying user data;
[0011] An extraction module, used for inputting database metadata into a pre-trained extraction model, so as to output at least one column of user data to be classified through the extraction model, each column of user data including at least one user data;
[0012] A classification module is used to determine a matching result between at least one target user data contained in a target column user data and the multiple category labels, wherein the target column user data is any column user data in the at least one column of target user data; and determine a category label of the target column user data according to the matching result.
[0013] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the data classification method described in the first aspect.
[0014] In a fourth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor can at least implement the data classification method described in the first aspect.
[0015] In a fifth aspect, an embodiment of the present invention provides a computer program product, comprising: a computer program, when the computer program is executed by a processor of an electronic device, the processor can at least implement the data classification method as described in the first aspect.
[0016] In the solution provided by the embodiment of the present invention, first, multiple category labels for classifying user data are determined. In practical applications, different user data classification scenarios may correspond to different category labels. Then, the database metadata is input into the pre-trained extraction model to output at least one column of user data to be classified through the extraction model, so as to realize the automatic screening of user data. After that, the matching result between at least one target user data contained in the target column user data and multiple category labels is determined, and the target column user data is any column of user data in the at least one column of target user data. Finally, the category label of the target column user data is determined according to the matching result between each target user data in the target column user data and multiple category labels. This solution automatically obtains user data from the database metadata through the extraction model, and matches each user data in each column of user data obtained with the pre-established category label, which can not only reduce the manual participation in the user data classification process, but also realize accurate and efficient classification and labeling of each column of user data. Based on the labeling results of user data, various data management requirements can be flexibly and conveniently realized, such as: data management of different security levels for user data with different category labels. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0018] Figure 1A flow chart of a data classification method provided by an embodiment of the present invention;
[0019] Figure 2 A schematic diagram of classification and grading of category labels provided by an embodiment of the present invention;
[0020] Figure 3 A flowchart of a method for determining a category label provided by an embodiment of the present invention;
[0021] Figure 4 A schematic diagram of the structure of a data classification device provided by an embodiment of the present invention;
[0022] Figure 5 For Figure 4 A schematic diagram of the structure of an electronic device corresponding to the data classification device provided in the illustrated embodiment. DETAILED DESCRIPTION
[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0024] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0025] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features in the embodiments can be combined with each other. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.
[0026] The data classification method provided in the embodiment of the present invention can be executed by an electronic device, which can be a terminal device such as a PC, a laptop, a smart phone, etc., or a server. The server can be a physical server including an independent host, or a virtual server, or a cloud server or a server cluster.
[0027] Figure 1A flowchart of a data classification method provided by an embodiment of the present invention is as follows: Figure 1 As shown, the following steps may be included:
[0028] 101. Determine multiple category labels for classifying user data.
[0029] 102. Input the database metadata into a pre-trained extraction model to output at least one column of user data to be classified through the extraction model.
[0030] 103. Determine a matching result between at least one target user data included in the target column user data and a plurality of category labels, where the target column user data is any column of user data in the at least one column of target user data.
[0031] 104. Determine the category label of the target column user data according to the matching result.
[0032] The data classification method provided in the embodiment of the present invention is used to classify user data in database metadata. Among them, user data refers to user-related data, including but not limited to user personal information (such as: user mobile phone number, ID number, home address, etc.) and user behavior data generated by users in the process of using the services provided by the platform (such as: website access data, etc.). In the database metadata (Database Metadata), user data is stored in the form of tables, each table contains multiple columns of user data, and each column of user data contains multiple user data. The implementation of the present invention realizes the classification and labeling of user data by determining the category label corresponding to each column of user data.
[0033] It is understandable that before data classification, multiple category labels for classifying user data must be determined first, that is, it is necessary to determine which categories the user data needs to be classified into. In actual applications, for the same user data, in different data classification scenarios, based on different classification purposes, the multiple category labels used may be different. For example, the types of category labels are different, the number of category labels is different, or the hierarchical structure of the label tree of the category labels is different.
[0034] In an optional embodiment, based on the current classification scenario or classification purpose of data classification, multiple category labels that match the current classification scenario or classification purpose can be determined from pre-established category labels to serve as multiple category labels for classifying user data.
[0035] In another optional embodiment, in order to achieve accurate classification of user data, before data classification, data sampling can be performed on the user data to be classified, and multiple category labels for data classification of the user data can be determined based on the sample data obtained by sampling; multiple category labels for data classification of the user data can also be determined based on other sample data related to the classification scenario or classification purpose corresponding to the user data to be classified. For example, multiple category labels for data classification of user data can be determined by manually marking the sample data; or multiple category labels for data classification of user data can be determined by clustering the sample data. In this embodiment, the focus is first on the data classification process, and in subsequent embodiments, the process of determining multiple category labels by clustering the sample data will be described in detail.
[0036] In an embodiment of the present invention, optionally, there are multiple category levels for multiple category tags used to classify user data, and each category tag corresponds to a preset data security level. The data security level is determined according to the sensitivity of the user data. If the sensitivity of the user data is high, it is determined that the category tag to which the user data belongs has a higher data security level. By dividing the user data into category tags and setting the data security level corresponding to the category tags, the user data can be conveniently and quickly classified and graded, effectively preventing data leakage and illegal access.
[0037] For ease of understanding, combined Figure 2 Give an example. Figure 2 A schematic diagram of classification and grading of category labels provided by an embodiment of the present invention, such as Figure 2 As shown, there are three levels of multiple category labels used to classify user data, namely, primary category labels, secondary category labels, and tertiary category labels, wherein a primary category label contains multiple secondary category labels, and a secondary category label contains multiple tertiary category labels, and the multi-level category labels are in a tree structure. Figure 2 As indicated, the first-level category tags include: user information, business data, security operation and maintenance, and business management; the user information category in the first-level category tag includes the following second-level category tags: personal basic information, personal identity information, personal device information, personal identity authentication information, network identity identification, personal Internet access records, personal property information, personal physiological health information, etc.; the second-level category tag personal basic information includes the following third-level category tags: gender, ethnicity, nationality, education, birthday, marital status, user name, etc.
[0038] To facilitate the safe management of user data, Figure 2In the illustrated scenario, the corresponding data security level is set from the dimension of the secondary category label, for example: personal basic information is set to C3 level, personal identity information is set to C4 level, personal device information is set to C3 level, personal identity authentication information is set to C4 level, network identity is set to C4 level, personal Internet records are set to C3 level, personal property information is set to C4 level, personal physiological health information is set to C3 level, etc. Among them, the larger the value corresponding to the level, the higher the security level of the data, and a more secure data access strategy needs to be adopted to avoid data leakage.
[0039] It should be noted that Figure 2 The classification and grading of the category labels shown are only for illustrative purposes. In actual applications, the multiple category labels used to classify user data may have different numbers of category levels, such as four category levels, five category levels, etc. The division of data security levels may also be diversified, with category labels for different category levels. This embodiment does not limit the classification and grading method of category labels.
[0040] In the process of classifying user data, since the user data to be classified is contained in the database metadata, and the database metadata contains other data in addition to the user data, it is necessary to first obtain the database metadata from the database, and then obtain the user data to be classified from the obtained database metadata.
[0041] In the related art, in order to obtain user data in the database metadata from the database, the database is usually scanned and the user data in the database metadata is obtained through a large number of read and write requests. However, the large number of read and write requests generated during the scanning process will affect the normal access requests to the database, and may cause database errors, etc., which is not conducive to the normal use of the database.
[0042] In order to reduce the impact of obtaining user data from database metadata on the database, in this embodiment, optionally, the database metadata in the database can be read into an offline environment first, and then the user data in the metadata can be further obtained in the offline environment. In this process, on the one hand, there is only a read operation on the database, and the frequency of reading and the order of magnitude of metadata read each time can be customized, for example: 100 numbers in a column of a data table are read each time, instead of the full amount of data; on the other hand, obtaining user data from database metadata in an offline environment will not affect the database. Therefore, the impact of obtaining user data on normal access to the database can be effectively reduced.
[0043] The database metadata read from the database may contain repeated redundant data or unrecognizable data. In order to improve the efficiency of obtaining user data from the database metadata, after obtaining the database metadata, the database metadata may be cleaned to remove repeated redundant data in the database metadata (for example, if a name appears repeatedly in multiple tables, the name is retained in only one table after redundancy is removed) and unrecognizable data.
[0044] In addition, in the database metadata, different tables may have different data expressions. For example, in the database metadata, some column names may be in pinyin, some in Chinese, and some in English. In order to facilitate the subsequent acquisition of user data from the database metadata and classification of user data, after acquiring the database metadata, the database metadata may be preprocessed to unify the data expression of the database metadata, for example, the column names of the tables in the database metadata are all expressed in Chinese.
[0045] After cleaning and / or preprocessing the database metadata, the amount of data in the database metadata can be effectively reduced, and the data expression form of the database metadata can be unified, thereby effectively improving the efficiency of obtaining user data in the process of obtaining user data from the database metadata.
[0046] In the embodiment of the present invention, at least one column of user data to be classified is extracted from the database metadata through a pre-trained extraction model to achieve automatic acquisition of user data. Optionally, the extraction model can be a unified information extraction model (Unified Information Extraction, UIM for short) or the like.
[0047] During the specific implementation process, the database metadata is first encoded to convert the text in the database metadata into a numerical vector format that can be understood by the extraction model; then, the encoding result of the database metadata is input into the pre-trained extraction model to output at least one column of user data to be classified through the extraction model.
[0048] Optionally, the database metadata can be converted into a numerical vector through encoding methods such as One-Hot Encoding and Word2Vec; or, the database metadata can be input into a Bidirectional Encoder Representations from Transformers (BERT) to output a numerical vector of the text in the database metadata through BERT, namely, an embedding vector. The bidirectional encoding mechanism of BERT can capture the contextual semantic information of the text in the database metadata.
[0049] In this embodiment, the extraction model is pre-trained to be able to identify important information such as named entities, relationships, events, etc. corresponding to user data. It can be understood that after the numerical vector corresponding to the database metadata is input into the extraction model, the extraction model outputs the recognized numerical vector. Since there is a corresponding relationship between the text of the database metadata and the data, the text corresponding to the numerical vector output by the extraction model is at least one column of user data to be classified.
[0050] Next, any column of user data in at least one column of user data to be classified (referred to as target column of user data) is taken as an example to explain the process of determining the category label of a column of user data.
[0051] In summary, for the target column user data, first determine the matching result between each target user data in at least one target user data included in the target column user data and multiple category labels used to classify the user data, that is, determine the category label corresponding to at least one target user data included in the target column user data; then determine the category label of the target column user data based on at least one matching result corresponding to the at least one target user data.
[0052] Specifically, the matching result between any target user data i in at least one target user data and multiple category labels can be determined based on the matching degree between the target user data i and each category label in the multiple category labels. Specifically, the matching degree between the target user data i and each category label in the multiple category labels is first determined; then, the matching degrees corresponding to the multiple category labels are sorted in descending order; finally, the category label corresponding to the top-k in the matching degree sorting result is determined as the category label corresponding to the target user data i.
[0053] It is understandable that the category labels corresponding to the target user data included in the target column user data may be the same or different. In an optional embodiment, the ratio of the number of target user data corresponding to each category label to the total number of target user data included in the target column user data in the matching results corresponding to the target user data included in the target column user data may be determined first; then, the category label of the target column user data is determined according to the ratio corresponding to each category label in the matching results. The ratio corresponding to the category label of the target column user data is greater than the ratio corresponding to other category labels in the matching results.
[0054] For ease of understanding, for example, assume that the target column user data contains 100 user data, namely target user data 1, target user data 2, ..., target user data 100; the multiple category labels used to classify the user data are category label 1, category label 2, and category label 3. Among them, the matching results corresponding to target user data 1 to target user data 90 respectively indicate that the category label corresponding to the target user data is category label 1, that is, target user data 1 to target user data 90 all correspond to category label 1; the matching results corresponding to target user data 91 to target user data 100 respectively indicate that the category label corresponding to the target user data is category label 2, that is, target user data 91 to target user data 100 all correspond to category label 2.
[0055] Therefore, it can be determined that the number of target user data corresponding to category label 1 is 90, and the ratio to the total number of target user data 100 included in the target column user data is 90%; the number of target user data corresponding to category label 2 is 10, and the ratio to the total number of target user data 100 included in the target column user data is 10%; the number of target user data corresponding to category label 3 is 0, and the ratio to the total number of target user data 100 included in the target column user data is 0. Since the ratio 90% corresponding to category label 1> the ratio 10% corresponding to category label 2> the ratio 0 corresponding to category label 3, it is determined that the category label of the target column user data is category label 1.
[0056] However, in actual applications, there are problems where the above ratios corresponding to multiple category labels are similar. For example, the ratio corresponding to category label 1 is 40%, the ratio corresponding to category label 2 is 30%, and the ratio corresponding to category label 3 is 30%. In this case, using category label 1 as the category label of the target column user data may be inaccurate.
[0057] In order to improve the accuracy of the target column user data classification results, optionally, a threshold value (e.g., 80%) can be set. After determining the above ratios corresponding to multiple category labels, if the maximum value among the ratios corresponding to the multiple category labels is greater than or equal to the set threshold value, the category label of the target column user data is determined to be the category label corresponding to the maximum ratio value; if the maximum value among the ratios corresponding to the multiple category labels is less than the set threshold value, the category label of the target column user data is determined to be the category label corresponding to the maximum ratio value, and a prompt message is output to prompt the user to confirm whether the category label of the target column user data is correct. By means of secondary confirmation, the accuracy of the target column user data classification results can be improved.
[0058] In addition, optionally, when more than one column of user data to be classified is obtained from the database metadata, clustering processing can be performed on the obtained multiple columns of user data to obtain multiple clusters, wherein each cluster includes different columns of user data in the multiple columns of user data. Afterwards, for a certain cluster, the category label of a certain column of user data in the cluster can be determined by the above data classification method, and then the category label of the column of user data is used as the category label of other columns of user data included in the cluster. Optionally, clustering processing can be performed by hierarchical clustering algorithms such as K-means, Agglomerative Hierarchical Clustering, etc.
[0059] In this embodiment, multiple columns of user data obtained from database metadata are first clustered, and then the category label corresponding to a column of user data in a cluster is determined to be the category label corresponding to other columns of user data in the cluster (that is, each column of user data in the same cluster has the same category label), which can effectively improve the classification efficiency of each column of user data.
[0060] In summary, in the solution provided by the embodiment of the present invention, first, multiple category labels for classifying user data are determined. Then, the database metadata is input into the pre-trained extraction model, so that at least one column of user data to be classified is output through the extraction model to realize automatic screening of user data. After that, the matching result between at least one target user data contained in the target column user data and multiple category labels is determined, and the target column user data is any column of user data in the at least one column of target user data. Finally, the category label of the target column user data is determined according to the matching result between each target user data in the target column user data and multiple category labels. This solution automatically obtains user data from the database metadata through the extraction model, and matches each user data in each column of user data obtained with the pre-established category label, which can not only reduce the manual participation in the user data classification process, but also realize accurate and efficient classification and labeling of each column of user data. Based on the labeling results of user data, various data management requirements can be flexibly and conveniently realized, such as: data management of different security levels for user data with different category labels.
[0061] The following is an explanation of the process of determining multiple category labels by performing cluster analysis on sample data.
[0062] Figure 3 A flow chart of a method for determining a category label provided by an embodiment of the present invention is as follows: Figure 3 As shown, the following steps may be included:
[0063] 301. Obtain a user data sample set, where the user data sample set includes multiple columns of user data samples with different column names.
[0064] The user data sample set may be sample data sampled from at least one column of user data to be classified, or may be other sample data related to the classification scenario or classification purpose corresponding to at least one column of user data to be classified.
[0065] 302. Convert multiple columns of user data samples into numerical vectors to obtain word vectors for each of the multiple columns of user data samples.
[0066] Optionally, the text in the multiple columns of user data samples can be converted into numerical vectors through One-Hot Encoding, Word2Vec, BERT and other encoding methods, and each data vector is the word vector of each column of user data samples.
[0067] Optionally, before converting multiple columns of user data samples into numerical vectors, the multiple columns of user data samples may be cleaned and / or preprocessed to remove repeated redundant data and unrecognizable data in the multiple columns of user data samples and unify the data expression of the multiple columns of user data samples.
[0068] 303. Clustering is performed on multiple columns of user data samples according to the similarity between word vectors to obtain multiple clusters and multiple category labels corresponding to the multiple clusters.
[0069] Optionally, the similarity between word vectors can be calculated by cosine similarity, Euclidean distance, Manhattan distance, etc.
[0070] Optionally, clustering processing can be performed using a hierarchical clustering algorithm such as K-means, Agglomerative Hierarchical Clustering, etc.
[0071] Optionally, the word vectors of each of the multiple columns of user data samples include a column noun vector. Clustering the multiple columns of user data samples according to the similarity between the word vectors to obtain multiple clusters and multiple category labels corresponding to the multiple clusters includes: clustering the multiple columns of user data samples according to the similarity between the column noun vectors of each of the multiple columns of user data samples to obtain multiple clusters and multiple category labels corresponding to the multiple clusters. The similarity between the column noun vectors of each column of user data sample in the same cluster is greater than a set similarity threshold.
[0072] 304. Extract common features of word vectors of each column of user data samples included in a target cluster, where the target cluster is any one of the multiple clusters.
[0073] 305. The common features are used as category features of the target category labels corresponding to the target clusters.
[0074] Optionally, in the process of determining multiple category labels corresponding to multiple clusters, the common features of the word vectors of each column of user data samples in the same cluster can also be extracted. The extracted common features can be used as the category features of the target category label corresponding to the target cluster, so as to determine the matching results of at least one target user data in the target column user data and multiple category labels.
[0075] exist Figure 1In the embodiment shown, the data classification method provided by the embodiment of the present invention is generally described. Among them, step 103, determining the matching result of at least one target user data included in the target column user data and multiple category labels, there are many determination methods in actual application, which are described in detail below.
[0076] In an optional embodiment, determining a matching result between at least one target user data included in the target column user data and a plurality of category tags includes: determining a matching result between at least one target user data included in the target column user data and category features of the plurality of category tags.
[0077] Specifically, for a certain target user data j in at least one target user data, the similarity between the target user data j and the category features of multiple category labels can be calculated; the similarity between the target user data j and the category features of each category label in the multiple category labels is used as the matching degree between the target user data j and each category label in the multiple category labels; based on the matching degree between the target user data j and each category label in the multiple category labels, the matching result between the target user data j and the multiple category labels is determined, for example: the matching result is-the category label corresponding to the top-k in the matching degree sorting result is the category label corresponding to the target user data i.
[0078] In another optional embodiment, determining the matching result between at least one target user data included in the target column user data and multiple category tags includes: determining the matching result between at least one target user data included in the target column user data and multiple category tags based on a preset regular expression.
[0079] Specifically, based on a preset regular expression, if it is determined that at least one target user data included in the target column user data includes a target field, then the category label of the at least one target user data is determined to be the category label corresponding to the target field in the regular expression. For example, assuming that the preset regular expression indicates that if there is a "user" or "account" field in the user data, the user data can be matched to the "user information category" label. For a certain target user data k in the at least one target user data, if it is determined that the target user data k includes the target field "user" or "account", then the category label of the target user data k is determined to be "user information category".
[0080] In another optional embodiment, determining the matching result between at least one target user data contained in the target column user data and multiple category tags includes: determining the matching result between at least one target user data and multiple category tags based on the matching result between the at least one target user data contained in the target column user data and the keywords respectively corresponding to the multiple category tags.
[0081] Specifically, a set of keywords is defined for each of the multiple category tags, for example, "user information category" may include keywords "name", "telephone", "address", etc. Then, for a certain target user data h in at least one target user data, the target user data h is matched with the keywords corresponding to each of the multiple category tags; and the matching results of the target user data h and the multiple category tags are determined. For example, if the target user data h contains the keywords "name", "telephone", and "address" of the "user information category" in the multiple category tags, then the category tag of the target user data h is determined to be "user information category".
[0082] In the above embodiment, the matching result between at least one target user data included in the target column user data and multiple category tags is determined by one of category feature matching, regular expression matching, and keyword matching. In practical applications, two or three of the above matching methods can be combined to determine the matching result between at least one target user data included in the target column user data and multiple category tags.
[0083] In an optional implementation, determining a matching result between at least one target user data included in the target column user data and a plurality of category labels includes:
[0084] Based on a preset regular expression, determining a matching result between at least one target user data included in the target column user data and a plurality of category labels;
[0085] If there is other target user data in the target column user data whose category labels cannot be determined by regular expressions, the matching results of the other target user data and the multiple category labels are determined according to the matching results of the keywords respectively corresponding to the other target user data and the multiple category labels.
[0086] In yet another optional embodiment, determining a matching result between at least one target user data included in the target column user data and a plurality of category labels further includes:
[0087] Based on a preset regular expression, determining a matching result between at least one target user data included in the target column user data and a plurality of category labels;
[0088] If there is other target user data in the target column user data whose category labels cannot be determined by regular expressions, the matching results of the other target user data and the multiple category labels are determined according to the matching results of the category features respectively corresponding to the other target user data and the multiple category labels.
[0089] In this embodiment, due to the use of regular expressions, the matching results between at least one target user data included in the target column user data and multiple category tags can be determined more quickly. Therefore, matching is first performed through regular expressions. When there are other target user data whose category tags cannot be determined through regular expressions, the matching results between other target user data and multiple category tags are determined through category feature matching or keyword matching. This can ensure the efficiency of determining the matching results between at least one target user data included in the target column user data and multiple category tags, and the accuracy of the matching results.
[0090] Optionally, the matching result between at least one target user data included in the target column user data and the multiple category labels may also be determined by manual labeling.
[0091] In practical applications, in order to ensure the accuracy of data classification results, the matching results of at least one target user data contained in the target column user data and multiple category labels may be determined by multiple different methods (such as category feature matching, regular expression matching, keyword matching, manual labeling, etc.) at the same time, so as to further determine the category label of the target column user data corresponding to each method according to the matching results. The category labels of the target column user data determined by different methods may be the same or different.
[0092] Optionally, the credibility corresponding to different methods such as category feature matching, regular expression matching, keyword matching, and manual labeling can be preset, for example, the credibility is from large to small, which is manual labeling, regular expression matching, category feature matching, and keyword matching. When there are category labels of target column user data corresponding to multiple different methods at the same time, according to the order of the credibility, the category label of the target column user data corresponding to the method with the highest credibility is determined as the final category label of the target column user data.
[0093] In this embodiment, the category labels of the target column user data are determined respectively through different methods such as category feature matching, regular expression matching, keyword matching, manual labeling, etc., and the category label of the target column user data corresponding to the method with the highest credibility is selected as the final category label of the target column user data, which can effectively improve the accuracy of the category label of the target column user data.
[0094] The data classification device of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will appreciate that these devices can be configured using commercially available hardware components through the steps taught in this solution.
[0095] Figure 4 A schematic diagram of the structure of a data classification device provided by an embodiment of the present invention is shown in FIG. Figure 4As shown, the device includes: a determination module 11, an extraction module 12, and a classification module 13.
[0096] The determination module 11 is used to determine a plurality of category labels for classifying user data.
[0097] The extraction module 12 is used to input the database metadata into a pre-trained extraction model to output at least one column of user data to be classified through the extraction model, each column of user data including at least one user data.
[0098] The classification module 13 is used to determine a matching result between at least one target user data contained in the target column user data and the multiple category labels, wherein the target column user data is any column user data in the at least one column of target user data; and determine a category label of the target column user data according to the matching result.
[0099] Optionally, the determination module 11 is specifically used to obtain a user data sample set, the user data sample set including multiple columns of user data samples with different column names; convert the multiple columns of user data samples into numerical vectors to obtain word vectors of each of the multiple columns of user data samples; and cluster the multiple columns of user data samples according to the similarity between the word vectors to obtain multiple cluster clusters and multiple category labels corresponding to the multiple cluster clusters.
[0100] Optionally, each word vector of the plurality of columns of user data samples includes a column noun vector. The determination module 11 is further specifically configured to perform clustering processing on the plurality of columns of user data samples according to the similarity between the column noun vectors to obtain a plurality of clusters and a plurality of category labels corresponding to the plurality of clusters, wherein the similarity between the column noun vectors of the columns of user data samples in the same cluster is greater than a set similarity threshold.
[0101] Optionally, the determination module 11 is also specifically used to extract common features of word vectors of each column of user data samples contained in a target cluster, where the target cluster is any one of the multiple clusters; and use the common features as category features of a target category label corresponding to the target cluster.
[0102] Optionally, the classification module 13 is specifically configured to determine a matching result between at least one target user data included in the target column user data and category features of the multiple category labels.
[0103] Optionally, the classification module 13 is also specifically used to determine, based on a preset regular expression, a matching result between at least one target user data contained in the target column user data and the multiple category labels; if there are other target user data in the target column user data whose category labels cannot be determined by the regular expression, then the matching results between the other target user data and the multiple category labels are determined based on the matching results of the keywords respectively corresponding to the other target user data and the multiple category labels.
[0104] Optionally, the classification module 13 is further specifically configured to, based on a preset regular expression, determine that a category label of the at least one target user data included in the target column user data includes a target field, then determine that the category label of the at least one target user data is the category label corresponding to the target field in the regular expression.
[0105] Optionally, the classification module 13 is also specifically used to determine the ratio of the number of target user data corresponding to each category label in the matching result to the total number of target user data contained in the target column user data; determine the category label of the target column user data according to the ratio corresponding to each category label in the matching result, and the ratio corresponding to the category label of the target column user data is greater than the ratio corresponding to other category labels in the matching result.
[0106] Optionally, the multiple category tags have multiple category levels, and each category tag corresponds to a preset data security level.
[0107] Figure 4 The device shown can execute the steps introduced in the aforementioned embodiments. For detailed execution process and technical effects, please refer to the description in the aforementioned embodiments, which will not be repeated here.
[0108] In one possible design, the above Figure 4 The structure of the data classification device shown can be implemented as an electronic device, such as Figure 5 As shown, the electronic device may include: a memory 21, a processor 22, and a communication interface 23. The memory 21 stores executable codes, and when the executable codes are executed by the processor 22, the processor 22 can at least implement the data classification method provided in the above embodiments.
[0109] In addition, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the data classification method provided in the aforementioned embodiment.
[0110] An embodiment of the present invention provides a computer program product, including: a computer program, when the computer program is executed by a processor of an electronic device, the processor is caused to execute the data classification method provided in the above-mentioned embodiment.
[0111] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.
[0112] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A data classification method, characterized in that: include: determining a plurality of category labels for classifying the user data; Inputting the database metadata into a pre-trained extraction model to output at least one column of user data to be classified through the extraction model; Determine a matching result between at least one target user data included in the target column user data and the multiple category labels, wherein the target column user data is any column user data in the at least one column of target user data; According to the matching result, a category label of the target column user data is determined.
2. The method according to claim 1, characterized in that The determining of a plurality of category labels for classifying the user data includes: Acquire a user data sample set, wherein the user data sample set includes multiple columns of user data samples with different column names; Convert the multiple columns of user data samples into numerical vectors to obtain word vectors for each of the multiple columns of user data samples; According to the similarity between the word vectors, clustering processing is performed on the multiple columns of user data samples to obtain multiple clusters and multiple category labels corresponding to the multiple clusters.
3. The method according to claim 2, characterized in that The word vectors of each of the multiple columns of user data samples include a column noun vector; the clustering process is performed on the multiple columns of user data samples according to the similarity between the word vectors to obtain multiple clusters and multiple category labels corresponding to the multiple clusters, including: According to the similarity between the column noun vectors, the multiple columns of user data samples are clustered to obtain multiple clusters and multiple category labels corresponding to the multiple clusters, and the similarity between the column noun vectors of each column of user data sample in the same cluster is greater than a set similarity threshold.
4. The method according to claim 2, characterized in that: The method further comprises: Extracting common features of word vectors of each column of user data samples included in a target cluster, wherein the target cluster is any one of the multiple clusters; The common features are used as category features of the target category label corresponding to the target cluster.
5. The method according to claim 4, characterized in that The determining of a matching result between at least one target user data included in the target column user data and the plurality of category labels includes: A matching result between at least one target user data included in the target column user data and the category features of the multiple category labels is determined.
6. The method according to claim 1, characterized in that The determining of a matching result between at least one target user data included in the target column user data and the plurality of category labels includes: Based on a preset regular expression, determining a matching result between at least one target user data included in the target column user data and the multiple category labels; If there is other target user data in the target column user data whose category labels cannot be determined by the regular expression, the matching results of the other target user data and the multiple category labels are determined according to the matching results of the keywords respectively corresponding to the other target user data and the multiple category labels.
7. The method according to claim 6, characterized in that The determining, based on a preset regular expression, a matching result between at least one target user data included in the target column user data and the plurality of category labels includes: Based on a preset regular expression, if it is determined that at least one target user data included in the target column user data includes a target field, the category label of the at least one target user data is determined to be the category label corresponding to the target field in the regular expression.
8. The method according to any one of claims 1 to 7, characterized in that Determining the category label of the target column user data according to the matching result includes: Determine a ratio of the amount of target user data corresponding to each category label in the matching result to the total amount of target user data included in the target column user data; The category label of the target column user data is determined according to the ratio corresponding to each category label in the matching result, and the ratio corresponding to the category label of the target column user data is greater than the ratios corresponding to other category labels in the matching result.
9. The method according to any one of claims 1 to 7, characterized in that The multiple category tags have multiple category levels, and each category tag corresponds to a preset data security level.
10. A data classification device, characterized in that: include: A determination module, used to determine a plurality of category labels for classifying user data; An extraction module, used for inputting database metadata into a pre-trained extraction model, so as to output at least one column of user data to be classified through the extraction model, each column of user data including at least one user data; a classification module, used to determine a matching result between at least one target user data included in the target column user data and the plurality of category labels, wherein the target column user data is any column user data in the at least one column of target user data; According to the matching result, a category label of the target column user data is determined.
11. An electronic device, characterized in that: include: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the data classification method as described in any one of claims 1 to 9.
12. A non-transitory machine-readable storage medium, characterized in that: The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the data classification method according to any one of claims 1 to 9.