Data classification system

Abstract

The invention discloses a data classification system, which relates to the technical field of data processing, and comprises a data management module, a semantic extraction module and a similarity calculation module, the data management module is in communication connection with the semantic extraction module; the semantic extraction module is in communication connection with the similarity calculation module; the data management module is used for sending the target data table and the attribute information of the target data table to the semantic extraction module; the semantic extraction module is used for determining feature vectors of the to-be-classified fields according to the table attribute information and the field attribute information of the to-be-classified fields for each to-be-classified field; the similarity calculation module is used for calculating the similarity between the feature vector of the to-be-classified field and the feature vector of at least one data category for each to-be-classified field; and according to the similarity between the feature vector of the to-be-classified field and the feature vector of at least one data category, determining the data category to which the to-be-classified field belongs. According to the invention, the classification efficiency and the classification accuracy of the data are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data classification system. Background Art

[0002] Traditional data classification and grading schemes mainly rely on predefined rules, regular expressions, keyword matching, data field names and other strategies for data identification, classification and grading. This approach usually requires manually writing rules and applying them to data sets to identify and classify specific types of data.

[0003] With the explosive growth of data volume, traditional rule-based classification and grading methods have been unable to meet the processing needs of large-scale and complex data. Manually defined rules are not only inefficient, but also difficult to handle complex data types such as structured data with contextual relationships and implicit semantic data. Summary of the invention

[0004] The present invention provides a data classification system to improve the efficiency and accuracy of data classification.

[0005] According to one aspect of the present invention, the present invention provides a data classification system, characterized in that the system comprises: a data management module, a semantic extraction module and a similarity calculation module; the data management module is communicatively connected to the semantic extraction module; the semantic extraction module is communicatively connected to the similarity calculation module;

[0006] The data management module is used to send the target data table and the attribute information of the target data table to the semantic extraction module; wherein the target data table includes at least one field to be classified; the attribute information includes the table attribute information of the target data table and the field attribute information of each field to be classified;

[0007] A semantic extraction module, for determining, for each field to be classified, a feature vector of the field to be classified according to table attribute information and field attribute information of the field to be classified;

[0008] A similarity calculation module, used for calculating, for each field to be classified, the similarity between a feature vector of the field to be classified and a feature vector of at least one data category;

[0009] The data category to which the field to be classified belongs is determined according to the similarity between the feature vector of the field to be classified and the feature vector of at least one data category.

[0010] Optionally, determining a feature vector of the field to be classified according to table attribute information and field attribute information of the field to be classified includes:

[0011] Merge the table attribute information and the field attribute information of the field to be classified to obtain attribute fusion information of the field to be classified;

[0012] The attribute fusion information of the field to be classified is semantically extracted to obtain the feature vector of the field to be classified.

[0013] Optionally, the data management module includes a data management unit and a model management unit;

[0014] A data management unit, used for sending a target data table and attribute information of the target data table to a semantic extraction module;

[0015] A model management unit, configured to send first semantic model identification information to a semantic extraction module;

[0016] The semantic extraction module is further used to select a first semantic extraction model from candidate semantic extraction models according to the first semantic model identification information;

[0017] Accordingly, semantic extraction is performed on the attribute fusion information of the field to be classified to obtain the feature vector of the field to be classified, including:

[0018] The first semantic extraction model is used to perform semantic extraction on the attribute fusion information of the field to be classified, and a feature vector of the field to be classified is obtained.

[0019] Optionally, the data management module is also in communication connection with the similarity calculation module;

[0020] The data management unit is further used to send category set identification information to the similarity calculation module;

[0021] The similarity calculation module is further used to query the target category set matching the category set identification information according to the category set identification information; the target category set includes a feature vector of at least one data category;

[0022] Calculate the similarity between the feature vector of the field to be classified and the feature vector of the data category in the target category set;

[0023] Accordingly, determining the data category to which the field to be classified belongs according to the similarity between the feature vector of the field to be classified and the feature vector of at least one data category includes:

[0024] The data category to which the field to be classified belongs is determined based on the similarity between the feature vector of the field to be classified and the feature vectors of each data category in the target category set.

[0025] Optionally, determining the data category to which the field to be classified belongs according to the similarity between the feature vector of the field to be classified and the feature vectors of each data category in the target category set includes:

[0026] Determine the data category with the greatest similarity as the candidate category of the field to be classified;

[0027] The data category to which the field to be classified belongs is determined based on the similarity corresponding to the candidate categories and the preset similarity threshold.

[0028] Optionally, determining the data category to which the field to be classified belongs according to the similarity corresponding to the candidate category and a preset similarity threshold includes:

[0029] If the similarity of the candidate category is greater than or equal to the preset similarity threshold, the candidate category is determined as the data category to which the field to be classified belongs;

[0030] If the similarity of the candidate categories is less than the preset similarity threshold, the data category to which the field to be classified belongs is determined to be an abnormal category. Optionally, the system further includes a storage module; the storage module is in communication connection with the similarity calculation module;

[0031] The similarity calculation module is further used to associate the field to be classified with the data category of the field to be classified for each field to be classified, and send the associated field to be classified and data category to the storage module;

[0032] The storage module is used to store the associated fields to be classified and data categories.

[0033] Optionally, the data management module further includes a category query unit; the category query unit is communicatively connected with the storage module;

[0034] A category query unit, used for sending a field identifier of a field to be queried to the storage module;

[0035] The storage module is also used to query the data category to which the field to be queried belongs according to the field identifier;

[0036] The queried data category is fed back to the category query unit.

[0037] Optionally, the category query unit is further configured to generate an alarm message if the received data category is an abnormal category, so as to prompt the technician to correct the data category of the queried category.

[0038] Optionally, the data management module further includes a category set management unit; the category set management unit is communicatively connected with the semantic extraction module;

[0039] A category set management unit, used to send description information of the category set to the semantic extraction module; the description information of the category set includes description information of at least one data category;

[0040] The semantic extraction module is further used to perform semantic extraction on the description information of each data category in the description information of the category set to obtain a feature vector of the data category;

[0041] Send the feature vector of the data category to the similarity calculation module;

[0042] The similarity calculation module is also used to store the feature vectors of each data category.

[0043] The data classification system of the embodiment of the present invention can perform semantic extraction on the attribute information of each field in the data table through the semantic extraction module to obtain the semantic feature vector of the field; by calculating the similarity between the semantic feature vector of the field and the feature vector of each data category, the category to which the field belongs is automatically determined, thereby improving the efficiency of classification; and by calculating the similarity between the semantic feature vectors, the semantic features of the field and the data category are fully utilized in the classification process, thereby improving the accuracy of classification.

[0044] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 is a structural diagram of a data classification system provided according to Embodiment 1 of the present invention;

[0047] Figure 2 is a structural diagram of a data classification system provided according to Embodiment 1 of the present invention;

[0048] Figure 3 is a structural diagram of a data classification system provided according to Embodiment 1 of the present invention;

[0049] Figure 4 is a structural diagram of a data classification system provided according to Embodiment 1 of the present invention;

[0050] Figure 5 is a structural diagram of a data classification system provided according to Embodiment 1 of the present invention;

[0051] Figure 6 is a structural diagram of a data classification system provided according to Embodiment 1 of the present invention;

[0052] Figure 7 This is a data classification flow chart provided according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0054] It should be noted that the terms "first" and "second" and the like in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] In the technical solution of the embodiment of the present invention, the acquisition, storage and application of the target data table and attribute information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0056] Embodiment 1

[0057] Figure 1 This is a schematic diagram of the structure of a data classification system provided by the first embodiment of the present invention; the embodiment of the present invention is applicable to the case of classifying fields in a data table. Figure 1As shown, the data classification system includes a data module 100, a semantic extraction module 200 and a similarity calculation module 300; the data module 100 is in communication connection with the semantic extraction module 200; the semantic extraction module 200 is in communication connection with the similarity calculation module 300; the data module 100 is used to send the target data table and the attribute information of the target data table to the semantic extraction module 200; wherein the target data table includes at least one field to be classified; the attribute information includes the table attribute information of the target data table and the field attribute information of each field to be classified; the semantic extraction module 200 is used to determine the feature vector of the field to be classified according to the table attribute information and the field attribute information of the field to be classified for each field to be classified; the similarity calculation module 300 is used to calculate the similarity between the feature vector of the field to be classified and the feature vector of at least one data category for each field to be classified; and the data category to which the field to be classified belongs is determined according to the similarity between the feature vector of the field to be classified and the feature vector of at least one data category.

[0058] In this embodiment, the field to be classified may be a field in a data table to be classified; the target data table is a data table including the field to be classified. The table attribute information may be the attribute information of the target data table, which may include but is not limited to the name, structure and source name of the target data table; the field attribute information may be the attribute information of the field to be classified, which may include but is not limited to the name, data type and field size of the field to be classified. The type of data category may be a security category, a business category and a format category, etc., and the present invention does not limit the specific form of the data category. Exemplarily, if the type of data category is a security category, the data category may include but is not limited to a first-level security category, a second-level security category and a third-level security category, etc., wherein the field of the first-level security category does not allow external access; the field of the second-level security category allows external access after encryption; the field of the third-level security category does not need to be encrypted and allows external access. If the type of data category is a business category, the data category may include a business basic information category, a transaction amount information category and a transaction basic information category, etc.

[0059] Specifically, the data model module 100 can send the target data table and the attribute information of the target data table to the semantic extraction module 200; the semantic extraction module 200 can adopt a certain algorithm to determine the feature vector of the field to be classified according to the table attribute information and the field attribute information of the field to be classified for each field to be classified; the similarity calculation module 300 can calculate the cosine similarity between the feature vector of the field to be classified and the feature vector of at least one data category for each field to be classified; adopt a certain algorithm to determine the data category to which the field to be classified belongs according to the similarity between the feature vector of the field to be classified and the feature vector of at least one data category.

[0060] The data classification system of the embodiment of the present invention can perform semantic extraction on the attribute information of each field in the data table through the semantic extraction module 200 to obtain the semantic feature vector of the field; by calculating the similarity between the semantic feature vector of the field and the feature vector of each data category, the category to which the field belongs is automatically determined, thereby improving the efficiency of classification; and by calculating the similarity between the semantic feature vectors, the semantic features of the field and the data category are fully utilized in the classification process, thereby improving the accuracy of classification.

[0061] Optionally, determining a feature vector of the field to be classified based on table attribute information and field attribute information of the field to be classified includes: fusing the table attribute information and the field attribute information of the field to be classified to obtain attribute fusion information of the field to be classified; and performing semantic extraction on the attribute fusion information of the field to be classified to obtain a feature vector of the field to be classified.

[0062] The attribute fusion information may be information obtained by fusing the table attribute information and the field attribute information. The feature vector of the field to be classified is the semantic feature vector obtained by semantically extracting the attribute fusion information of the field to be classified. It should be noted that the present invention does not limit the fusion method between the table attribute information and the field attribute information of the field to be classified, and the fusion method may be, for example, concatenation and random insertion.

[0063] It can be understood that, by adopting the above technical scheme, the table attribute information and the field attribute information of the field to be classified are fused to obtain the attribute fusion information of the field to be classified; the attribute fusion information of the field to be classified is semantically extracted to obtain the feature vector of the field to be classified, and the feature vector containing rich semantic information can be obtained, thereby improving the accuracy of classifying the field to be classified according to the feature vector.

[0064] Optional, Figure 2 It is a structural diagram of a data classification system. Figure 2 As shown, the data module 100 includes a data management unit 101 and a model management unit 102; the data management unit 101 is used to send the target data table and the attribute information of the target data table to the semantic extraction module 200; the model management unit 102 is used to send the first semantic model identification information to the semantic extraction module 200; the semantic extraction module 200 is also used to select a first semantic extraction model from candidate semantic extraction models according to the first semantic model identification information; accordingly, semantic extraction is performed on the attribute fusion information of the field to be classified to obtain a feature vector of the field to be classified, including: through the first semantic extraction model, semantic extraction is performed on the attribute fusion information of the field to be classified to obtain a feature vector of the field to be classified.

[0065] The first semantic model identification information may be used to uniquely identify the candidate semantic extraction model. The first semantic extraction model may be a candidate semantic model that matches the first semantic model identification information. The candidate semantic extraction model may be a network model that performs semantic extraction on attribute fusion information.

[0066] Specifically, the model management unit 102 can send the first semantic model identification information to the semantic extraction module 200; the semantic extraction module 200 can query the candidate semantic extraction model that matches the first semantic model identification information from the candidate semantic extraction models, and determine the queried candidate semantic extraction model as the first semantic extraction model.

[0067] In a specific embodiment, the first semantic model identification information may be associated with the target data table; illustratively, if the target data table is a financial business data table, the first semantic model identification information may be the identification information of the financial business semantic extraction model; if the target data table is a medical business data table, the first semantic model identification information may be the identification information of the medical business semantic extraction model.

[0068] It can be understood that, by adopting the above technical scheme, the data module 100 includes a data management unit 101 and a model management unit 102; the data management unit 101 is used to send the target data table and the attribute information of the target data table to the semantic extraction module 200; the model management unit 102 is used to send the first semantic model identification information to the semantic extraction module 200; the semantic extraction module 200 is also used to select the first semantic extraction model from the candidate semantic extraction models according to the first semantic model identification information; through the first semantic extraction model, the attribute fusion information of the field to be classified is semantically extracted to obtain the feature vector of the field to be classified, and the attribute fusion information of the field to be classified can be accurately semantically extracted through the matching first semantic extraction model, thereby improving the accuracy of the feature vector of the field to be classified, thereby improving the accuracy of classifying the field to be classified.

[0069] Optional, Figure 3 It is a structural diagram of a data classification system. Figure 3As shown, the data management unit 100 is also connected to the similarity calculation module 300 for communication; the data management unit 101 is also used to send category set identification information to the similarity calculation module 300; the similarity calculation module 300 is also used to query the target category set that matches the category set identification information according to the category set identification information; the target category set includes a feature vector of at least one data category; the similarity between the feature vector of the field to be classified and the feature vector of the data category in the target category set is calculated; accordingly, according to the similarity between the feature vector of the field to be classified and the feature vector of at least one data category, the data category to which the field to be classified belongs is determined, including: according to the similarity between the feature vector of the field to be classified and the feature vectors of each data category in the target category set, the data category to which the field to be classified belongs is determined.

[0070] The category set identification information can be used to uniquely identify the category set; the target category set is the category set that matches the category set identification information. The category set may include a feature vector of at least one data category. Different category sets may include different data categories. The category set may include a financial category set, a medical category set, a business category set, etc.

[0071] In a specific implementation, if the target data table is a financial data table, the category set identified by the category set identification information is a financial category set; the data categories in the financial category set may include but are not limited to account basic information categories, transaction amount information categories, and transaction basic information categories, etc.; the data category to which the field to be classified belongs is determined from the data categories included in the financial category set.

[0072] Specifically, the data management unit 101 can send category set identification information to the similarity calculation module 300; the similarity calculation module 300 can query the target category set that matches the category set identification information based on the category set identification information; calculate the similarity between the feature vector of the field to be classified and the feature vector of the data category in the target category set; use a certain algorithm to determine the data category to which the field to be classified belongs based on the similarity between the feature vector of the field to be classified and the feature vectors of each data category in the target category set.

[0073] It can be understood that, by adopting the above technical scheme, the data module 100 is also communicated with the similarity calculation module 300; the data management unit 101 is also used to send category set identification information to the similarity calculation module 300; the similarity calculation module 300 is also used to query the target category set that matches the category set identification information according to the category set identification information; the target category set includes a feature vector of at least one data category; the similarity between the feature vector of the field to be classified and the feature vector of the data category in the target category set is calculated; according to the similarity between the feature vector of the field to be classified and the feature vector of each data category in the target category set, the data category to which the field to be classified belongs is determined, and the data category to which the data to be classified belongs can be determined from the target category set, avoiding determining the data category from a category set that is not related to the data to be classified, thereby improving the accuracy of determining the data category to which the data to be classified belongs.

[0074] Optionally, the data category to which the field to be classified belongs is determined based on the similarity between the feature vector of the field to be classified and the feature vectors of each data category in the target category set, including: determining the data category with the greatest similarity as the alternative category of the field to be classified; and determining the data category to which the field to be classified belongs based on the similarity corresponding to the alternative categories and a preset similarity threshold.

[0075] Among them, the candidate category can be the data category with the greatest similarity. It should be noted that the preset similarity threshold can be independently set by the technician according to actual needs or practical experience, and the present invention does not limit this. Specifically, the data category with the greatest similarity is determined as the candidate category of the field to be classified; a certain algorithm is used to determine the data category to which the field to be classified belongs according to the similarity corresponding to the candidate category and the preset similarity threshold.

[0076] It can be understood that by adopting the above technical solution, the data category with the greatest similarity is determined as the alternative category of the field to be classified; based on the similarity corresponding to the alternative category and the preset similarity threshold, the data category to which the field to be classified belongs is determined, thereby improving the accuracy of the data category to which the field to be classified belongs.

[0077] Optionally, the data category to which the field to be classified belongs is determined based on the similarity corresponding to the alternative categories and a preset similarity threshold, including: if the similarity of the alternative categories is greater than or equal to the preset similarity threshold, the alternative category is determined as the data category to which the field to be classified belongs; if the similarity of the alternative categories is less than the preset similarity threshold, the data category to which the field to be classified belongs is determined to be an abnormal category.

[0078] Among them, the abnormal category can represent that the data category to which the field to be classified belongs cannot be determined from the category set. It can be understood that, by adopting the above technical solution, when the similarity of the candidate categories is less than the preset similarity threshold, the data category to which the field to be classified belongs can be determined to be an abnormal category, thereby avoiding the situation where the data category of the field to be classified is incorrectly classified as an alternative category when the similarity of the candidate categories is still less than the preset similarity, thereby improving the accuracy of the data category of the field to be classified.

[0079] Optional, Figure 4 It is a structural diagram of a data classification system; Figure 4 As shown, the system also includes a storage module 400; the storage module 400 is in communication with the similarity calculation module 300; the similarity calculation module 300 is also used to associate the field to be classified and the data category of the field to be classified for each field to be classified, and send the associated field to be classified and data category to the storage module 400; the storage module 400 is used to store the associated field to be classified and data category.

[0080] It can be understood that by adopting the above-mentioned technical solution, the associated fields to be classified and data categories are stored in the storage module 400, and the classification results can be queried, analyzed, exported and reported through the storage module 400. This makes it possible not only to track and analyze the classification results, but also to facilitate technical personnel to carry out subsequent security management measures, such as access control, encryption protection and desensitizing processing.

[0081] Optional, Figure 5 It is a structural diagram of a data classification system. Figure 5 As shown, the data module 100 also includes a category query unit 103; the category query unit 103 is communicatively connected to the storage module 400; the category query unit 103 is used to send the field identifier of the field to be queried to the storage module 400; the storage module 400 is also used to query the data category to which the field to be queried belongs according to the field identifier; and the queried data category is fed back to the category query unit 103.

[0082] The field to be queried may be a field of the classification result to be queried; the field identifier is used to uniquely identify the field. Specifically, the category query unit 103 is connected to the storage module 400 for communication; the category query unit 103 is used to send the field identifier of the field to be queried to the storage module 400; the storage module 400 is also used to query the data category to which the field to be queried belongs according to the field identifier; and the queried data category is fed back to the category query unit 103.

[0083] It can be understood that, by adopting the above technical scheme, the category query unit 103 is used to send the field identifier of the field to be queried to the storage module 400; the storage module 400 is also used to query the data category to which the field to be queried belongs based on the field identifier; the queried data category is fed back to the category query unit 103, and the data category of the field can be queried from the storage module 400 through the field identifier, so that the technical personnel can take subsequent security management measures according to the data category.

[0084] Optionally, the category query unit 103 is further configured to generate an alarm message if the received data category is an abnormal category, so as to prompt the technician to correct the data category of the query category. It is understandable that, by adopting the above technical solution, an alarm message can be sent to the technician when the data category is an abnormal category, so as to prompt the technician to manually correct the data category of the query field, thereby improving the accuracy of the data category.

[0085] Optional, Figure 6 It is a structural diagram of a data classification system. Figure 6 As shown, the data module 100 also includes a category set management unit 104; the category set management unit 104 is in communication connection with the semantic extraction module 200; the category set management unit 104 is used to send description information of the category set to the semantic extraction module 200; the description information of the category set includes description information of at least one data category; the semantic extraction module 200 is also used to perform semantic extraction on the description information of each data category in the description information of the category set to obtain a feature vector of the data category; the feature vector of the data category is sent to the similarity calculation module 300; the similarity calculation module 300 is also used to store the feature vector of each data category.

[0086] The description information of the data category may be a description text of the data category. For example, if the data category is basic account information, the description information of the basic account information may be "referring to the basic information data of the account, such as account number, account type, margin account mark, account status, account opening institution number, account opening date, account closing date, payment mark (as a substitute value for original transaction elements such as payment account number, used to complete payment transactions in specific scenarios)". In an optional embodiment, the description information may be semantically extracted through a semantic extraction model of category set matching to obtain a feature vector of the data category.

[0087] It can be understood that by adopting the above-mentioned technical scheme, the feature vector corresponding to each data category in the extracted category set can be pre-extracted and stored in the similarity calculation module 300, so as to obtain the feature vector of the data category in a timely manner and calculate the similarity between the feature vector of the field to be classified and the feature vector of the data category, thereby improving the efficiency of calculating the similarity.

[0088] Optional, Figure 7 It is a data classification flow chart; Figure 7 As shown, the classification standard is entered into the category set management unit; wherein the classification standard can be a specific form of the category set; the classification and grading standard includes at least one data category and the description information of each data category; the category set management unit performs data preprocessing on the classification standard, including text cleaning, word segmentation and format standardization. These processed data will be converted into semantic representations recognizable by the semantic extraction model and sent to the semantic extraction module, which performs semantic extraction vectorization processing on the classification standard; the feature vector of the obtained classification standard is stored in the similarity calculation module. Enter metadata of the data to be classified into the data management unit; wherein the data to be classified is the target data table, and the metadata is the field to be classified in the target data table; the data management unit preprocesses the entered data, and sends it to the semantic extraction module to extract semantic features, and obtains vectorized metadata, that is, the feature vector of the field to be classified; the feature vector of the metadata to be classified is sent to the similarity calculation module; the similarity calculation module calculates the similarity between the feature vector of the metadata to be classified and the feature vector of each data category; according to the decision rule of classification and grading, and the similarity between the feature vector of the metadata to be classified and the feature vector of each data category, determine the grading result of the metadata to be classified, that is, the category to which the field to be classified belongs; save the classification and grading result of the metadata to be classified, so as to facilitate the technical personnel to query the classification decomposition result of the metadata to be classified later.

[0089] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0090] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data classification system, characterized in that: The system comprises: a data management module, a semantic extraction module and a similarity calculation module; the data management module is connected in communication with the semantic extraction module; the semantic extraction module is connected in communication with the similarity calculation module; The data management module is used to send a target data table and attribute information of the target data table to the semantic extraction module; wherein the target data table includes at least one field to be classified; the attribute information includes table attribute information of the target data table and field attribute information of each field to be classified; The semantic extraction module is used to determine, for each field to be classified, a feature vector of the field to be classified according to the table attribute information and the field attribute information of the field to be classified; The similarity calculation module is used to calculate the similarity between the feature vector of each field to be classified and the feature vector of at least one data category; The data category to which the field to be classified belongs is determined according to the similarity between the feature vector of the field to be classified and the feature vector of at least one data category.

2. The system according to claim 1, characterized in that The step of determining the feature vector of the field to be classified according to the table attribute information and the field attribute information of the field to be classified includes: Merging the table attribute information and the field attribute information of the field to be classified to obtain attribute fusion information of the field to be classified; Semantic extraction is performed on the attribute fusion information of the field to be classified to obtain a feature vector of the field to be classified.

3. The system according to claim 2, characterized in that The data management module includes a data management unit and a model management unit; The data management unit is used to send the target data table and the attribute information of the target data table to the semantic extraction module; The model management unit is used to send first semantic model identification information to the semantic extraction module; The semantic extraction module is further used to select a first semantic extraction model from candidate semantic extraction models according to the first semantic model identification information; Accordingly, the semantic extraction of the attribute fusion information of the field to be classified to obtain the feature vector of the field to be classified includes: The first semantic extraction model is used to perform semantic extraction on the attribute fusion information of the field to be classified to obtain a feature vector of the field to be classified.

4. The method according to claim 3, characterized in that: The data management module is also in communication connection with the similarity calculation module; The data management unit is further used to send category set identification information to the similarity calculation module; The similarity calculation module is further used to query a target category set matching the category set identification information according to the category set identification information; the target category set includes a feature vector of at least one data category; Calculating the similarity between the feature vector of the field to be classified and the feature vector of the data category in the target category set; Correspondingly, determining the data category to which the field to be classified belongs according to the similarity between the feature vector of the field to be classified and the feature vector of at least one data category includes: The data category to which the field to be classified belongs is determined according to the similarity between the feature vector of the field to be classified and the feature vectors of each data category in the target category set.

5. The method according to claim 4, characterized in that The step of determining the data category to which the field to be classified belongs according to the similarity between the feature vector of the field to be classified and the feature vectors of each data category in the target category set includes: Determine the data category with the greatest similarity as the candidate category of the field to be classified; The data category to which the field to be classified belongs is determined according to the similarity corresponding to the candidate categories and a preset similarity threshold.

6. The system according to claim 5, characterized in that The step of determining the data category to which the field to be classified belongs according to the similarity corresponding to the candidate categories and a preset similarity threshold comprises: If the similarity of the candidate category is greater than or equal to the preset similarity threshold, the candidate category is determined as the data category to which the field to be classified belongs; If the similarity of the candidate categories is less than the preset similarity threshold, it is determined that the data category to which the field to be classified belongs is an abnormal category.

7. The system according to claim 1, characterized in that The system further comprises a storage module; the storage module is communicatively connected with the similarity calculation module; The similarity calculation module is further used to associate the field to be classified with the data category of the field to be classified for each field to be classified, and send the associated field to be classified and data category to the storage module; The storage module is used to store the associated fields to be classified and data categories.

8. The system according to claim 7, characterized in that The data management module further includes a category query unit; the category query unit is in communication connection with the storage module; The category query unit is used to send the field identifier of the field to be queried to the storage module; The storage module is further used to query the data category to which the field to be queried belongs according to the field identifier; The queried data category is fed back to the category query unit.

9. The system according to claim 8, characterized in that The category query unit is further configured to generate an alarm message if the received data category is an abnormal category, so as to prompt the technician to correct the data category of the category to be queried.

10. The system according to claim 1, characterized in that The data management module further comprises a category set management unit; the category set management unit is in communication connection with the semantic extraction module; The category set management unit is used to send description information of the category set to the semantic extraction module; the description information of the category set includes description information of at least one data category; The semantic extraction module is further used to perform semantic extraction on the description information of each data category in the description information of the category set to obtain a feature vector of the data category; Sending the feature vector of the data category to the similarity calculation module; The similarity calculation module is also used to store the feature vectors of each data category.

Images