Data classification method, apparatus, device and computer-readable medium

Abstract

The present invention discloses a method, apparatus, device and computer-readable medium for data classification, and relates to the field of computer technology. A specific implementation of the method includes: splitting the metadata information of structured data, filtering to obtain the classification terms of the structured data; determining the hit rate of the classification terms in each classification dictionary and the occupancy rate of the classification terms in each classification dictionary according to multiple classification dictionaries of the structured data and the weights of the terms in the classification dictionaries, each of the classification dictionaries corresponding to a preset industry; calculating the confidence that the classification terms belong to the classification dictionary based on the hit rate of each classification dictionary and the occupancy rate of the classification terms in each classification dictionary, and taking the category to which the classification dictionary with the maximum confidence belongs as the category of the structured data. This implementation can be applied to a variety of industries and companies, has a wide range of applicability, and improves data classification efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data classification method, apparatus, device and computer-readable medium. Background Art

[0002] With the advent of the big data era, enterprises are storing vast amounts of data, including user data, financial data, and operational data. Depending on the size of the enterprise, the types of data assets it holds can range from dozens to hundreds. Due to the growing demand for data security, enterprises need to categorize, organize, and classify their stored data.

[0003] In the process of realizing the present invention, the inventors found that there are at least the following problems in the prior art: the differences in data in different industries and companies have a great impact on the classification results, and the application scope of existing data classification methods is narrow, which affects the efficiency of data classification. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus, device, and computer-readable medium for data classification, which can be applied to a variety of industries and companies, have a wide range of applications, and improve data classification efficiency.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for data classification is provided, comprising:

[0006] After splitting the metadata information of the structured data, the classification terms of the structured data are obtained through filtering;

[0007] determining, according to multiple classification dictionaries of the structured data and weights of terms in the classification dictionaries, a hit rate of the classification terms in each classification dictionary and a occupancy rate of the classification terms in each classification dictionary, wherein the occupancy rate is the occupancy rate of the classification terms in the classification dictionary in the classification terms of the structured data, each classification dictionary corresponding to a preset industry;

[0008] The confidence that the classification word belongs to the classification dictionary is calculated according to the hit rate of each classification dictionary and the occupancy rate of the classification word in each classification dictionary, and the category to which the classification dictionary with the maximum confidence belongs is used as the category of the structured data.

[0009] The step of determining the hit rate of the classification word in each classification dictionary and the occupancy rate of the classification word in each classification dictionary according to the plurality of classification dictionaries of the structured data and the weights of the words in the classification dictionaries includes:

[0010] According to the upper-level category to which the structured data belongs, obtaining multiple lower-level classification dictionaries of the structured data;

[0011] According to the plurality of subordinate classification dictionaries of the structured data and the weights of the words in the subordinate classification dictionaries, the hit rate of the classification word in each subordinate classification dictionary and the occupancy rate of the classification word in each subordinate classification dictionary are determined.

[0012] The step of obtaining a plurality of subordinate classification dictionaries of the structured data according to the superior category to which the structured data belongs includes:

[0013] According to the upper-level category to which the structured data belongs and a data classification dictionary template tree, a plurality of lower-level classification dictionaries of the structured data are obtained, wherein the data classification dictionary template tree includes classification dictionaries of multiple levels.

[0014] The categories of the structured data include one or more of the following: industry classification, business category, table level category, and field level category.

[0015] After taking the category of the classification dictionary with the maximum confidence as the category of the structured data, the method further includes:

[0016] Based on the classification words of the structured data, the weights of the words in the classification dictionary are updated.

[0017] The hit rate of the classification word in each classification dictionary is determined by the weight of the hit word in the classification word and the weight of the word in the classification dictionary. The hit word is the classification word that is the same as the word in the classification dictionary.

[0018] The occupancy rate of the classification words in the classification dictionary is determined by the weight of the hit words in the classification words and the weight of the classification words.

[0019] According to a second aspect of an embodiment of the present invention, a data classification apparatus is provided, comprising:

[0020] A processing module, configured to split the metadata information of the structured data and obtain classification terms of the structured data through filtering;

[0021] a determination module, configured to determine, based on multiple classification dictionaries of the structured data and weights of terms in the classification dictionaries, a hit rate of the classification terms in each classification dictionary and a occupancy rate of the classification terms in each classification dictionary, wherein the occupancy rate is the occupancy rate of the classification terms in the classification dictionary in the classification terms of the structured data, and each classification dictionary corresponds to a preset industry;

[0022] The classification module is used to calculate the confidence that the classification words belong to the classification dictionary according to the hit rate of each classification dictionary and the occupancy rate of the classification words in each classification dictionary, and use the category to which the classification dictionary with the maximum confidence belongs as the category of the structured data.

[0023] According to a third aspect of an embodiment of the present invention, there is provided an electronic device for data classification, comprising:

[0024] one or more processors;

[0025] a storage device for storing one or more programs,

[0026] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-described method.

[0027] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0028] One embodiment of the above invention has the following advantages or beneficial effects: after the metadata information of the structured data is split, the classification terms of the structured data are obtained through filtering; according to multiple classification dictionaries of the structured data and the weights of the terms in the classification dictionaries, the hit rate of the classification terms in each classification dictionary and the occupancy rate of the classification terms in each classification dictionary are determined, wherein the occupancy rate is the occupancy rate of the classification terms in the classification dictionary among the classification terms of the structured data, and each classification dictionary corresponds to a preset industry; based on the hit rate of each classification dictionary and the occupancy rate of the classification terms in each classification dictionary, the confidence level that the classification term belongs to the classification dictionary is calculated, and the category to which the classification dictionary with the highest confidence level belongs is used as the category of the structured data. The classification dictionaries correspond to the preset industries and can therefore be applied to a variety of industries and companies, with a wide range of applications and improved data classification efficiency.

[0029] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0031] Figure 1 1 is a schematic diagram of the main flow of a method for data classification according to an embodiment of the present invention;

[0032] Figure 2 is a flow chart of determining the hit rate and occupancy rate of classification words according to an embodiment of the present invention;

[0033] Figure 3 is a schematic diagram of a data classification dictionary template tree according to an embodiment of the present invention;

[0034] Figure 4 is a schematic diagram of the correspondence between categories and words according to an embodiment of the present invention;

[0035] Figure 5 is a schematic diagram of an application of a method for data classification according to an embodiment of the present invention;

[0036] Figure 6 is a schematic diagram of the main structure of a data classification device according to an embodiment of the present invention;

[0037] Figure 7 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0038] Figure 8 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0040] Currently, enterprise data classification can only be used for specific industries, with poor portability and difficulty in customization. Because data types vary across industries, data classification methods designed for specific fields may not be fully applicable to other industries, making them difficult to port.

[0041] Most existing data classification methods are based on machine learning or deep learning, which requires large amounts of data and time to train models. This creates a significant workload and makes it difficult to quickly apply to new scenarios. Furthermore, deep learning methods require the creation of knowledge graphs, extensive data training, and continuous model tuning, making the transition from model design to implementation time-consuming.

[0042] At the same time, the differences in data across different industries and companies may have a significant impact on the results of machine learning. Therefore, when applied to new scenarios, the data needs to be reorganized and trained, which still requires a lot of preparation and takes a lot of time.

[0043] In addition, data classification based on machine learning or deep learning requires identifying the content of the data, which makes the solution difficult to implement when users want the data to be kept confidential or encrypted.

[0044] In summary, data classification schemes based on machine learning or deep learning have a narrow scope of application and low transplantation efficiency.

[0045] In order to solve the problems of narrow application scope and low migration efficiency of data classification solutions, the following technical solutions in the embodiments of the present invention can be adopted.

[0046] See also Figure 1 , Figure 1 This is a schematic diagram of the main process of the data classification method according to an embodiment of the present invention. Based on the classification terms of the structured data, the classification dictionary corresponding to the preset industry is used to determine the category of the structured data. Figure 1 As shown, the specific steps include:

[0047] S101: After the metadata information of the structured data is split, the classification words of the structured data are obtained through filtering.

[0048] In an embodiment of the present invention, the object of data classification is structured data. Structured data refers to data stored in a specific structure, such as data in MySQL, SQL Server, or Hive. In one embodiment of the present invention, metadata information is information that describes the attributes and structure of the data, such as the name, description, and attributes of the database, table, and field in MySQL data types.

[0049] Because structured data involves the data structure but not the actual data content, data classification can be achieved through structured data, making it easier to implement data confidentiality and encryption. In other words, even when data is encrypted or confidential, data classification can still be achieved by utilizing the metadata of structured data.

[0050] In one embodiment of the present invention, a classifier can be used to implement data classification. Specifically, the original text input into the classifier is metadata information for structured data. Taking MySQL as an example, table information and field information are input into the classifier. Table information and field information are structured metadata information. Table information includes the table name and table description, and field information includes the field name and field description. For certain data types that do not have field information, such as MongoDB, field information does not need to be input.

[0051] The metadata of structured data needs to be split into multiple classification terms, and data classification is achieved based on the multiple classification terms. After the metadata of structured data is split and filtered, the classification terms of the structured data are obtained.

[0052] Specifically, the metadata of the structured data is segmented using a pre-set word segmentation dictionary, splitting Chinese characters into words and English sentences into characters or phrases. The word segmentation dictionary can be used to segment Chinese characters, split or combine Chinese sentences or phrases as needed, and can also be used to combine certain English phrases.

[0053] After splitting the metadata of the structured data, we filter out words that are irrelevant to the content using the filter dictionary to obtain the classification terms for the structured data. Words that are irrelevant to the content include, but are not limited to, the following: index ID, data creator, creation time, modification time, and deletion time.

[0054] The filter word dictionary is used to filter some common words or words used to describe metadata, such as "index", "id", "creation time", etc. These words are irrelevant to the data content and may even interfere with classification. Therefore, they need to be filtered before classification.

[0055] S102. Determine, based on multiple classification dictionaries of the structured data and the weights of the words in the classification dictionaries, a hit rate of the classification words in each classification dictionary and a occupancy rate of the classification words in each classification dictionary. The occupancy rate is the occupancy rate of the classification words in the classification dictionary in the classification words of the structured data. Each classification dictionary corresponds to a preset industry.

[0056] Structured data is represented by classification terms. The category to which the classification terms belong is the category to which the structured data belongs. In this embodiment of the present invention, classification terms are classified using classification dictionaries. Each classification dictionary corresponds to a preset industry. For example, industry 1 corresponds to classification dictionary a; industry 2 corresponds to classification dictionary b.

[0057] In an embodiment of the present invention, the classification dictionary can be pre-set or updated. As an example, words in the classification dictionary can be deleted, added, or modified. In this way, the updated classification dictionary can be used in the next data classification process.

[0058] A classification dictionary includes multiple terms, each with a corresponding weight. The weight indicates the importance and discriminatory power of a term. A higher weight indicates a higher probability of the term appearing in this category, while a lower probability of appearing in other categories. For example, the initial weight values ​​are manually set based on the specific situation.

[0059] Furthermore, according to the multiple classification dictionaries of the structured data and the weights of the words in the classification dictionaries, the hit rate of the classification words in each classification dictionary and the occupancy rate of the classification words in each classification dictionary are determined. The occupancy rate is the occupancy rate of the classification words in the classification dictionary in the classification words of the structured data.

[0060] See also Figure 2 , Figure 2This is a flow chart of determining the hit rate and occupancy rate of classification words according to an embodiment of the present invention. Specifically, the following steps are included:

[0061] S201. Acquire multiple subordinate classification dictionaries of the structured data according to the superior category to which the structured data belongs.

[0062] Each classification dictionary corresponds to a preset industry, and classification dictionaries include multiple levels. In other words, a preset industry corresponds to multiple levels of classification dictionaries. As an example, a classification dictionary includes four levels, from high to low: first-level classification dictionary, second-level classification dictionary, third-level classification dictionary, and fourth-level classification dictionary. Each of these four levels of classification dictionaries corresponds to a preset industry.

[0063] Then, according to the upper-level category to which the structured data belongs, multiple lower-level classification dictionaries of the structured data can be obtained. Based on the classification dictionary at each level, the data classification at that level is realized.

[0064] In one embodiment of the present invention, if a preset industry does not have a corresponding classification dictionary, all classification dictionaries within the categories of the preset industry can be used as classification dictionaries for the preset industry. For example, Industry A has three classification dictionaries: Classification Dictionary A, Classification Dictionary B, and Classification Dictionary C. If Industry A does not have a corresponding classification dictionary, Classification Dictionary A, Classification Dictionary B, and Classification Dictionary C can be used as classification dictionaries for Industry A.

[0065] See also Figure 3 , Figure 3 2 is a schematic diagram of a data classification dictionary template tree according to an embodiment of the present invention. Figure 3 It includes multiple classification dictionaries, such as: general classification dictionary, financial industry classification dictionary and medical industry classification dictionary. Among them, the general classification dictionary is applicable to many industries.

[0066] For each industry, the corresponding classification dictionary can be divided into four levels, from highest to lowest: industry, business, table-level, and field-level. For example, an industry-level classification dictionary corresponds to one industry; under the industry category, there are n business-level classification dictionaries; under business category 2, there are n table-level classification dictionaries; and under table category 2, there are n field-level classification dictionaries. n is a pre-set natural number.

[0067] From the attached Figure 3 As can be seen in the previous section, classification dictionaries can be divided into multiple levels. A classification dictionary for an industry can have a business-level classification dictionary under it; a business-level classification dictionary can have table-level classification dictionaries under it; and a table-level classification dictionary can have field-level classification dictionaries under it. As the level decreases, the number of classification dictionaries increases, allowing classification terms to be accurately assigned to classification dictionaries.

[0068] Specifically, the data classification dictionary template tree consists of four levels of content. The first level is industry classification. The second level is business classification. Similar data content or content belonging to the same business line is divided into one category, such as personal information, finance, technology, and company information. The third level is the table level class. Each business classification contains several tables. The table level refers to a data structure similar to the Mysql table level, such as Mysql tables, Hive tables, and MongoBD tables. The fourth level is the field level class. Each table classification contains several fields. The field level refers to a data structure similar to the Mysql field level, such as Mysql fields and Hive fields.

[0069] In one embodiment of the present invention, using Figure 3 The data classification dictionary template tree in the data classification dictionary is used to obtain a classification dictionary. Specifically, multiple sub-classification dictionaries of the structured data are obtained according to the upper-level category of the structured data and the data classification dictionary template tree. The data classification dictionary template tree includes classification dictionaries at multiple levels. As an example, the data classification dictionary template tree includes an industry classification dictionary, a business classification dictionary, a table-level classification dictionary, and a field-level classification dictionary.

[0070] In this embodiment of the present invention, the classification dictionary corresponds to the data classification dictionary template tree. The table or field level is set as the final result of data classification, so classification dictionaries are added to the table and field level classes. A business class can also add a classification dictionary, or use the classification dictionary set of all tables it contains as the business class's classification dictionary.

[0071] As an example, if the upper category of structured data is the financial industry, the following sub-category dictionaries for the structured data are obtained according to the data classification dictionary template tree: the classification dictionary for business category 1, the classification dictionary for business category 2, and the classification dictionary for business category n. The industry to which the structured data belongs is known.

[0072] As another example, the upper category of the structured data is table-level class 2. According to the number of data classification dictionary templates, the lower-level classification dictionaries of the structured data are obtained: the classification dictionary of field-level class 1, the classification dictionary of field-level class 2, and the classification dictionary of field-level class n.

[0073] See also Figure 4 , Figure 4 Schematic diagram of the correspondence between categories and words according to an embodiment of the present invention. Figure 4 Category A and Category B are included. Category A and Category B both belong to the field-level classification dictionary. Category A includes Term 1, Term 2, and Term 3. Term 1 has a weight of 10; Term 2 has a weight of 8; and Term 3 has a weight of 5. Category B includes Term 4. Term 4 has a weight of 4.

[0074] S202: Determine the hit rate of the classification words in each subordinate classification dictionary and the occupancy rate of the classification words in each subordinate classification dictionary according to the multiple subordinate classification dictionaries of the structured data and the weights of the words in the subordinate classification dictionaries.

[0075] For each classification word, it is necessary to determine the hit rate in multiple subordinate classification dictionaries and the occupancy rate of the classification word in each subordinate classification dictionary in the classification word of the structured data.

[0076] In an embodiment of the present invention, the likelihood of a term being in a classification dictionary is measured by its hit rate. The hit rate of a classification term in each classification dictionary is determined by the weight of the hit term in the classification dictionary and the weight of the term in the classification dictionary. The hit term is the same classification term as the term in the classification dictionary.

[0077] Specifically, the hit rate is the weighted proportion of the hit word in each classification dictionary. Repeated hit words are counted only once.

[0078] Set the total number of words in each category to sum, and the weight of each word in the classification dictionary to weightWord n (n=1…sum), the index of the hit word is index, and the weight of the hit word is weightHit index (1≤index≤sum). According to formula 1, the hit rate of the classified words in each classified dictionary is calculated.

[0079] It should be noted that the sum of the hit word weights in Formula 1 includes repeated words. Repeated words are calculated once. As an example, if word 1 appears 3 times, then the hit word weight in Formula 1 is calculated once for word 1.

[0080]

[0081] In an embodiment of the present invention, the likelihood of a classification dictionary in a classification term is measured by its occupancy rate. The occupancy rate of a classification dictionary in a classification term is determined by the weight of the hit terms in the classification term and the weight of the classification term itself. In other words, the occupancy rate of a classification dictionary in a classification term is determined by the weight of the hit terms in the classification term and the weight of the classification term in the structured data.

[0082] Specifically, the occupancy rate is the weighted proportion of the classified words in each classification dictionary among all the classified words. Repeated words can participate in the calculation multiple times.

[0083] Set the number of classification terms to length and the weight of each classification term to weightDate m(m=1...length), weight is the weight of the word in the corresponding classification dictionary, and the weight of the word not included in the word segmentation dictionary is set to 1. After calculation, the occupancy rate of the classification dictionary in the classification words is obtained.

[0084] It should be noted that the sum of all hit word weights in Formula 2 includes repeated words. For repeated words, the number of reappearances is calculated. As an example, if word 1 appears 3 times, then word 1 is repeated 3 times in the total hit word weight in Formula 2.

[0085]

[0086] S103 , calculating the confidence that the classification word belongs to the classification dictionary based on the hit rate of each classification dictionary and the occupancy rate of the classification word in each classification dictionary, and taking the category to which the classification dictionary with the maximum confidence belongs as the category of the structured data.

[0087] For each classification dictionary, after determining the classification dictionary's hit rate and the occupancy rate of the classification terms in the classification dictionary, the confidence level of the classification terms belonging to the classification dictionary can be calculated. In other words, for each classification term, the confidence level of belonging to each classification dictionary can be calculated. Each classification dictionary corresponds to a corresponding confidence level.

[0088] In the embodiment of the present invention, the confidence level of the classification dictionary can be calculated according to Formula 3.

[0089] Confidence of the classification dictionary = hit rate of the classification dictionary × a + occupancy rate of classification words in the classification dictionary × b Formula 3

[0090] Wherein, a+b=1, and a and b can be preset according to experience.

[0091] After the confidence of each classification dictionary is calculated according to Formula 3, the category to which the classification dictionary with the maximum confidence belongs can be used as the category of the structured data.

[0092] It should be noted that when the category of structured data includes multiple levels of categories, Figure 1 In the middle step, after determining the parent category of the structured data, you can also Figure 1 In the middle step, the sub-categories of the structured data are determined.

[0093] As an example, the categories of structured data are divided into four levels from high to low: industry category, business category, table level category and field level category. After knowing that the structured data belongs to the industry category, you can execute Figure 1 In the steps above, determine the business class of structured data; then, follow Figure 1 Steps to determine the table-level class of structured data; finally, follow Figure 1The field-level class of the structured data is determined by the following steps. Finally, the business class, table-level class, and field-level class to which the structured data belongs can be obtained.

[0094] In one embodiment of the present invention, the categories of structured data include one or more of the following: industry classification, business category, table-level category, and field-level category.

[0095] That is to say, by adopting the technical solution in the embodiment of the present invention, one or more of the industry classification, business class, table-level class and field-level class to which the structured data belongs can be obtained.

[0096] In the embodiment of the present invention, the classification dictionary and the weight of the words in the classification dictionary are used to determine the hit rate in the classification dictionary and the occupancy rate of the classification words in the classification dictionary. As the number of structured data categories increases, the words in the classification dictionary change.

[0097] To improve data classification accuracy, the weights of terms in the classification dictionary are updated based on the classification terms in the structured data. As an example, an analyzer is used to update the weights of terms in the classification dictionary based on the classification terms in the structured data. As another example, the categories of the structured data can be manually reviewed before updating the weights of terms in the classification dictionary.

[0098] The Aprior algorithm is used to update the weights of terms in the classification dictionary based on the classification terms in the structured data. Specifically, the Aprior algorithm first filters out existing classification terms and duplicate classification terms. Then, the Aprior algorithm is used to identify frequently occurring classification terms. These are added as new terms in the classification dictionary. In other words, the weights of these new terms in the classification dictionary are updated from zero to non-zero.

[0099] For example, when using the Aprior algorithm for analysis, each categorized word in structured data is considered a set. Multiple sets of structured data of the same category serve as input to the Aprior algorithm. The Aprior algorithm outputs frequently occurring words and highly relevant words—words discovered by the Aprior algorithm that can be added to the categorized dictionary.

[0100] In addition, the TF-IDF algorithm is specifically used to update the weights of words in the classification dictionary. If the updated weight is inconsistent with the preset weight, the weight is set to the updated weight.

[0101] It is understandable that for existing words in the classification dictionary, the TF-IDF algorithm can be used to update the weight of the words in the classification dictionary; for new words in the classification dictionary, the TF-IDF algorithm can be used to update the weight of the words in the classification dictionary.

[0102] In the above-described embodiment of the present invention, after the metadata information of the structured data is split, the classification terms of the structured data are obtained through filtering. According to multiple classification dictionaries of the structured data and the weights of the terms in the classification dictionaries, the hit rate of the classification terms in each classification dictionary and the occupancy rate of the classification terms in each classification dictionary are determined. The occupancy rate is the occupancy rate of the classification terms in the classification dictionary among the classification terms of the structured data. Each classification dictionary corresponds to a preset industry. Based on the hit rate of each classification dictionary and the occupancy rate of the classification terms in each classification dictionary, the confidence level that the classification term belongs to the classification dictionary is calculated, and the category to which the classification dictionary with the highest confidence level belongs is used as the category of the structured data. The classification dictionaries correspond to the preset industries and can therefore be applied to a variety of industries and companies, with a wide range of applications and improved data classification efficiency.

[0103] Moreover, by using metadata information of structured data, data classification can be achieved without understanding the data content, thereby improving user privacy protection.

[0104] See also Figure 5 , Figure 5 2 is a schematic diagram of an application of a method for data classification according to an embodiment of the present invention. Figure 5 The classifier and analyzer are included. The classifier is used to classify the metadata of structured data. The analyzer is used to update the weight of the terms in the classification dictionary based on the classification terms of the structured data.

[0105] The classifier loads a segmentation dictionary to segment the original text. A filter dictionary is loaded to filter out useless words from the segmented original text, resulting in the classification terms for the structured data—the processed text. The original text is the metadata for the structured data. Classification dictionaries at different levels are loaded to classify the classification terms for the structured data and generate classification results.

[0106] In the analyzer, the classification results of the classifier are used to discover new words, and then the weights of the words in the classification dictionary are optimized to achieve the update of the classification dictionary.

[0107] The classifier can use the updated classification dictionary during the next data classification process.

[0108] See also Figure 6 , Figure 6 FIG. 1 is a schematic diagram of the main structure of a data classification device according to an embodiment of the present invention. The data classification device can implement a data classification method, such as Figure 6 As shown, the data classification device specifically includes:

[0109] Processing module 601 is used to split metadata information of structured data and obtain classification terms of the structured data through filtering;

[0110] Determining module 602 is configured to determine, based on multiple classification dictionaries of the structured data and weights of terms in the classification dictionaries, a hit rate of the classification terms in each classification dictionary and a occupancy rate of the classification terms in each classification dictionary, wherein the occupancy rate is the occupancy rate of the classification terms in the classification dictionary in the classification terms of the structured data, and each classification dictionary corresponds to a preset industry;

[0111] The classification module 603 is configured to calculate the confidence that the classification words belong to the classification dictionary according to the hit rate of each classification dictionary and the occupancy rate of the classification words in each classification dictionary, and use the category to which the classification dictionary with the maximum confidence belongs as the category of the structured data.

[0112] In one embodiment of the present invention, the determining module 602 is specifically configured to obtain a plurality of lower-level classification dictionaries of the structured data according to the upper-level category to which the structured data belongs;

[0113] According to the plurality of subordinate classification dictionaries of the structured data and the weights of the words in the subordinate classification dictionaries, the hit rate of the classification word in each subordinate classification dictionary and the occupancy rate of the classification word in each subordinate classification dictionary are determined.

[0114] In one embodiment of the present invention, the determination module 602 is specifically configured to obtain multiple subordinate classification dictionaries of the structured data according to the upper-level category to which the structured data belongs and a data classification dictionary template tree, wherein the data classification dictionary template tree includes classification dictionaries of multiple levels.

[0115] In one embodiment of the present invention, the categories of the structured data include one or more of the following: industry classification, business category, table level category, and field level category.

[0116] In one embodiment of the present invention, the classification module 603 is further configured to update the weights of the terms in the classification dictionary based on the classification terms in the structured data.

[0117] In one embodiment of the present invention, the hit rate of the classification term in each of the classification dictionaries is determined by the weight of the hit term in the classification term and the weight of the term in the classification dictionary, and the hit term is the classification term that is the same as the term in the classification dictionary.

[0118] In one embodiment of the present invention, the occupancy rate of the classification words in the classification dictionary is determined by the weight of the hit words in the classification words and the weight of the classification words.

[0119] Figure 7 An exemplary system architecture 700 is shown to which the method or apparatus for data classification according to an embodiment of the present invention may be applied.

[0120] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, 703, a network 704, and a server 705. Network 704 is used to provide a medium for communication links between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0121] Users can use terminal devices 701, 702, and 703 to interact with server 705 via network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0122] The terminal devices 701 , 702 , and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0123] Server 705 may be a server that provides various services, such as a backend management server (for example only) that supports shopping websites browsed by users using terminal devices 701, 702, and 703. The backend management server may analyze and process received data such as product information query requests, and feed back processing results (for example, target push information and product information—for example only) to the terminal device.

[0124] It should be noted that the data classification method provided in the embodiment of the present invention is generally executed by the server 705 , and accordingly, the data classification device is generally set in the server 705 .

[0125] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0126] Reference below Figure 8 , which shows a schematic structural diagram of a computer system 800 of a terminal device suitable for implementing an embodiment of the present invention. Figure 8 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0127] like Figure 8As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the system 800 are also stored in the RAM 803. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0128] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, and the like; an output section 807 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 808 including a hard disk; and a communication section 809 including a network interface card such as a LAN card or a modem. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 810 as needed, so that computer programs read therefrom can be installed into the storage section 808 as needed.

[0129] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from a removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are performed.

[0130] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0131] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0132] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as follows: a processor includes a processing module, a determination module, and a classification module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the processing module may also be described as "for splitting metadata information of structured data and obtaining classification terms of the structured data through filtering."

[0133] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiments, or may exist independently without being incorporated into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, the device includes:

[0134] After splitting the metadata information of the structured data, the classification terms of the structured data are obtained through filtering;

[0135] determining, according to multiple classification dictionaries of the structured data and weights of terms in the classification dictionaries, a hit rate of the classification terms in each classification dictionary and a occupancy rate of the classification terms in each classification dictionary, wherein the occupancy rate is the occupancy rate of the classification terms in the classification dictionary in the classification terms of the structured data, each classification dictionary corresponding to a preset industry;

[0136] The confidence that the classification word belongs to the classification dictionary is calculated according to the hit rate of each classification dictionary and the occupancy rate of the classification word in each classification dictionary, and the category to which the classification dictionary with the maximum confidence belongs is used as the category of the structured data.

[0137] According to the technical solution of an embodiment of the present invention, the metadata information of the structured data is split and filtered to obtain the classification terms of the structured data; according to multiple classification dictionaries of the structured data and the weights of the terms in the classification dictionaries, the hit rate of the classification terms in each classification dictionary and the occupancy rate of the classification terms in each classification dictionary are determined, and the occupancy rate is the occupancy rate of the classification terms in the classification dictionary in the classification terms of the structured data, and each classification dictionary corresponds to a preset industry; based on the hit rate of each classification dictionary and the occupancy rate of the classification terms in each classification dictionary, the confidence level that the classification terms belong to the classification dictionary is calculated, and the category to which the classification dictionary with the highest confidence level belongs is used as the category of the structured data. The classification dictionaries correspond to the preset industries and can therefore be applied to a variety of industries and companies, with a wide range of applications and improved data classification efficiency.

[0138] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for data classification, characterized in that: include: After splitting the metadata information of the structured data, the classification terms of the structured data are obtained through filtering; determining, according to multiple classification dictionaries of the structured data and weights of terms in the classification dictionaries, a hit rate of the classification terms in each classification dictionary and a occupancy rate of the classification terms in each classification dictionary, wherein the occupancy rate is the occupancy rate of the classification terms in the classification dictionary in the classification terms of the structured data, each classification dictionary corresponding to a preset industry; The confidence that the classification word belongs to the classification dictionary is calculated according to the hit rate of each classification dictionary and the occupancy rate of the classification word in each classification dictionary, and the category to which the classification dictionary with the maximum confidence belongs is used as the category of the structured data.

2. The data classification method according to claim 1, characterized in that: The step of determining the hit rate of the classification word in each classification dictionary and the occupancy rate of the classification word in each classification dictionary according to the plurality of classification dictionaries of the structured data and the weights of the words in the classification dictionaries includes: Obtaining multiple subordinate classification dictionaries of the structured data according to the superior category to which the structured data belongs; According to the plurality of subordinate classification dictionaries of the structured data and the weights of the words in the subordinate classification dictionaries, the hit rate of the classification word in each subordinate classification dictionary and the occupancy rate of the classification word in each subordinate classification dictionary are determined.

3. The data classification method according to claim 2, characterized in that: The step of obtaining a plurality of subordinate classification dictionaries of the structured data according to the superior category to which the structured data belongs includes: According to the upper-level category to which the structured data belongs and a data classification dictionary template tree, a plurality of lower-level classification dictionaries of the structured data are obtained, wherein the data classification dictionary template tree includes classification dictionaries of multiple levels.

4. The data classification method according to claim 1, characterized in that: The categories of the structured data include one or more of the following: industry classification, business category, table level category, and field level category.

5. The data classification method according to claim 1, characterized in that: After taking the category of the classification dictionary with the maximum confidence as the category of the structured data, the method further includes: Based on the classification words of the structured data, the weights of the words in the classification dictionary are updated.

6. The data classification method according to claim 1, characterized in that: The hit rate of the classification word in each classification dictionary is determined by the weight of the hit word in the classification word and the weight of the word in the classification dictionary. The hit word is the classification word that is the same as the word in the classification dictionary.

7. The data classification method according to claim 1, characterized in that: The occupancy rate of the classification words in the classification dictionary is determined by the weight of the hit words in the classification words and the weight of the classification words.

8. A data classification device, characterized in that: include: A processing module, configured to split the metadata information of the structured data and obtain classification terms of the structured data through filtering; a determination module, configured to determine, based on multiple classification dictionaries of the structured data and weights of terms in the classification dictionaries, a hit rate of the classification terms in each classification dictionary and a occupancy rate of the classification terms in each classification dictionary, wherein the occupancy rate is the occupancy rate of the classification terms in the classification dictionary in the classification terms of the structured data, and each classification dictionary corresponds to a preset industry; The classification module is used to calculate the confidence that the classification words belong to the classification dictionary according to the hit rate of each classification dictionary and the occupancy rate of the classification words in each classification dictionary, and use the category to which the classification dictionary with the maximum confidence belongs as the category of the structured data.

9. An electronic device for data classification, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images