Data classification method and apparatus, electronic device, storage medium, and program product

Abstract

Embodiments of the present application provide a data classification method and device, electronic equipment, storage medium and program product. Relate to the field of information processing. In the present application, a first classification result is obtained according to text data of to-be-classified data, and a second classification result is obtained according to resource directory data of to-be-classified data, and then the first classification result and the second classification result are combined to determine the subclass to which the to-be-classified data belongs. In the present application, regular expressions are not relied on, and not only structured data can be classified, but also unstructured data can be classified. In the present application, the subclass description text data is obtained according to the subclass description text in the classification hierarchical judgment specification table corresponding to the to-be-classified data. Based on the resource directory data of the to-be-classified data and the subclass description text data, the second classification result of the to-be-classified data is obtained, which can scan all subclasses included in the hierarchical classification judgment specification table to accurately determine the second classification result of the to-be-classified data.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of information processing, and in particular to a data classification method, apparatus, electronic device, storage medium and program product. Background Technology

[0002] In the context of big data, data centers across various industries have accumulated massive amounts of data assets, which are stored in both structured and unstructured formats. Classifying and grading this data enables effective data security management. Specifically, data can be categorized according to different attributes and characteristics, and different levels of protection measures can be applied to different categories.

[0003] Currently, common classification and grading systems primarily employ the following method for structured data: scanning database tables and matching the data using pre-configured regular expressions to classify and grade the structured data. This method performs well with user personal information fields such as name, phone number, and address. However, the standardized definition of classification and grading often extends beyond user personal information to include numerous subcategories. Due to the limitations of regular expressions, this method cannot automate the processing of all subcategories. Furthermore, this method is also unsuitable for processing unstructured data.

[0004] Therefore, there is an urgent need for a simpler and more convenient interaction method. Summary of the Invention

[0005] This invention provides a data classification method, an electronic device, a storage medium, and a computer program product to at least partially solve the problems existing in the related technologies.

[0006] The first aspect of this invention provides a data classification method, the method comprising:

[0007] Process the data to be classified to determine the text data and resource directory data of the data to be classified;

[0008] Based on the text data of the data to be classified and the pre-trained text classification model, the first classification result of the data to be classified is obtained;

[0009] Based on the resource catalog data and subclass description text data of the data to be classified, the second classification result of the data to be classified is obtained; the subclass description text data is obtained based on the subclass description text in the classification and grading determination specification table corresponding to the data to be classified.

[0010] Based on the first classification result and the second classification result, the subclass to which the data to be classified belongs is determined.

[0011] Optionally, if the data to be classified is structured data, the data to be classified is processed to determine the text data and resource directory data of the data to be classified, including:

[0012] The database table fields containing the data to be classified already have comments, confirming that they are resource catalog data.

[0013] Obtain the field values ​​of the data to be classified and determine them as data text.

[0014] Optionally, if the data to be classified is unstructured data, the data to be classified is processed to determine the text data and resource directory data of the data to be classified, including:

[0015] The file storage directory and file name of the data to be classified are determined as resource directory data;

[0016] The text content of the data to be classified is segmented to obtain fragmented text, and the fragmented text is identified as text data.

[0017] Optionally, the method further includes:

[0018] Obtain the classification and grading standard table of the data to be classified;

[0019] Obtain each subclass and its corresponding subclass description text from the classification and grading determination specification table;

[0020] The subclass description text is segmented into words to obtain the word segmentation set of description text corresponding to each subclass.

[0021] The word segmentation set of the description text corresponding to each subclass is used as the description text data of each subclass.

[0022] Optionally, based on the text data of the data to be classified and a pre-trained text classification model, a first classification result for the data to be classified is obtained, including:

[0023] Convert the text data of the text data to be classified into sentence vectors;

[0024] The sentence vector is input into a pre-trained text classification model to obtain the first classification result of the data to be classified;

[0025] The text classification model is trained on a semantic understanding model based on sample text data with classification labels.

[0026] Optionally, based on the resource catalog data and subclass description text data of the data to be classified, a second classification result is obtained for the data to be classified, including:

[0027] Convert the resource catalog data of the data to be classified into a first text feature vector;

[0028] Convert the text data describing each subclass into a second text feature vector;

[0029] The bag-of-words model is used to represent the first text feature vector and the second text feature vector as numerical vectors, resulting in the first numerical vector and the second numerical vector.

[0030] Based on the first and second numerical vectors, the cosine similarity comparison method is used to determine the similarity value between the resource catalog data and the description text data of each subclass of the data to be classified.

[0031] The second classification result is determined based on the similarity value between the resource catalog data of the data to be classified and the description text data of each subclass.

[0032] A second aspect of the present invention provides a data classification apparatus, the apparatus comprising:

[0033] The data processing module is used to process the data to be classified and to determine the text data and resource directory data of the data to be classified.

[0034] The first classification module is used to obtain the first classification result of the data to be classified based on the text data of the data to be classified and the pre-trained text classification model;

[0035] The second classification module is used to obtain the second classification result of the data to be classified based on the resource catalog data and the description text data of each subclass of the data to be classified; the description text data of each subclass is obtained based on the subclass description text in the classification and grading determination specification table corresponding to the data to be classified.

[0036] The result determination module is used to determine the subclass to which the data to be classified belongs based on the first classification result and the second classification result.

[0037] Optionally, when the data to be classified is structured data, the data processing module is specifically used for:

[0038] The database table fields containing the data to be classified already have comments, confirming that they are resource catalog data.

[0039] Obtain the field values ​​of the data to be classified and determine them as data text.

[0040] Optionally, when the data to be classified is unstructured data, the data processing module is specifically used for:

[0041] The file storage directory and file name of the data to be classified are determined as resource directory data;

[0042] The text content of the data to be classified is segmented to obtain fragmented text, and the fragmented text is identified as text data.

[0043] Optionally, the device further includes:

[0044] The first acquisition module is used to acquire the classification and grading determination standard table of the data to be classified.

[0045] The second acquisition module is used to acquire each subclass and the subclass description text corresponding to each subclass in the classification and grading judgment specification table;

[0046] The word segmentation module is used to perform word segmentation on the subclass description text to obtain the word segmentation set of description text corresponding to each subclass.

[0047] The determination module is used to use the word segmentation set of the description text corresponding to each subclass as the description text data of each subclass.

[0048] Optionally, the first classification module is specifically used for:

[0049] Convert the text data of the text data to be classified into sentence vectors;

[0050] The sentence vector is input into a pre-trained text classification model to obtain the first classification result of the data to be classified;

[0051] The text classification model is trained on a semantic understanding model based on sample text data with classification labels.

[0052] Optionally, the second classification module is specifically used for:

[0053] Convert the resource catalog data of the data to be classified into a first text feature vector;

[0054] Convert the text data describing each subclass into a second text feature vector;

[0055] The bag-of-words model is used to represent the first text feature vector and the second text feature vector as numerical vectors, resulting in the first numerical vector and the second numerical vector.

[0056] Based on the first and second numerical vectors, the cosine similarity comparison method is used to determine the similarity value between the resource catalog data and the description text data of each subclass of the data to be classified.

[0057] The second classification result is determined based on the similarity value between the resource catalog data of the data to be classified and the description text data of each subclass.

[0058] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the data classification method as described in the first aspect of the present invention.

[0059] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the data classification method as described in the first aspect of the present invention.

[0060] A fifth aspect of the present invention provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the data classification method described in the first aspect of the present invention.

[0061] In this embodiment of the invention, a first classification result is obtained based on the text data of the data to be classified, and a second classification result is obtained based on the resource directory data of the data to be classified. The subclass to which the data to be classified belongs is then determined by combining the first classification result and the second classification result.

[0062] The technical solution provided by this invention does not rely on regular expressions. It can classify not only structured data but also unstructured data. Furthermore, in the technical solution provided by this invention, the description text data of each subclass is obtained based on the subclass description text in the classification and grading determination specification table corresponding to the data to be classified. Based on the resource directory data of the data to be classified and the description text data of each subclass, the second classification result of the data to be classified can be obtained by scanning all subclasses included in the classification and grading determination specification table to accurately determine the second classification result of the data to be classified, and thus accurately determine the subclass to which the data to be classified belongs. Attached Figure Description

[0063] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a flowchart of a data classification method according to an embodiment of the present invention;

[0065] Figure 2 This is a flowchart of another data classification method according to an embodiment of the present invention;

[0066] Figure 3 This is a structural block diagram of a data classification device according to an embodiment of the present invention. Detailed Implementation

[0067] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0068] Specifically, this invention proposes a data classification method, referring to... Figure 1 The flowchart illustrates a data classification method according to an embodiment of the present invention, the method comprising the following steps:

[0069] S101, Process the data to be classified, and determine the text data and resource directory data of the data to be classified.

[0070] In this embodiment of the invention, the data to be classified can be data from a database. Typically, the data to be classified is industry-related data, such as a book database or a human resource management database.

[0071] In this embodiment of the invention, the text data of the data to be classified refers to the text body included in the data to be classified. For example, when the data to be classified is a data table in a database, the text data refers to the specific text content corresponding to each field.

[0072] In this embodiment of the invention, resource catalog data refers to the existing annotations of various databases, data tables, or fields included in the data to be classified, or the storage directory and file name of the files.

[0073] In this embodiment of the invention, the resource catalog data reflects, to some extent, the descriptive and annotation information corresponding to the text data of the data to be classified.

[0074] S102, based on the text data of the data to be classified and the pre-trained text classification model, obtain the first classification result of the data to be classified.

[0075] In this embodiment of the invention, the text classification model can be a semantic understanding model, which can learn in advance the correspondence between sample text data and various data categories.

[0076] In this embodiment of the invention, the first classification result represents the classification result obtained by classifying the text content of the data to be classified.

[0077] S103, based on the resource catalog data and the description text data of each subclass of the data to be classified, the second classification result of the data to be classified is obtained.

[0078] The subclass description text data is obtained based on the subclass description text in the classification and grading determination specification table corresponding to the data to be classified.

[0079] In this embodiment of the invention, a classification and grading determination standard table corresponding to the data to be classified can be obtained first, and then the description text of each subclass can be obtained from the classification and grading determination standard table.

[0080] Preferably, in this embodiment of the invention, the description text of the smallest subclass in the classification and grading determination specification table can be obtained to obtain a classification result with a smaller granularity.

[0081] In this embodiment of the invention, the classification and grading standard table corresponding to the data to be classified can be obtained according to the industry to which the data to be classified belongs, based on common industry classification and grading standards.

[0082] In this embodiment of the invention, considering one of the management characteristics of databases and file repositories, the descriptive and annotation information of the corresponding text content can be reflected in the fields, data tables, or existing comments of each text content in the database. Similarly, in the file repository, the filenames and storage directories of each file can also reflect the descriptive and annotation information of the file content. Therefore, this embodiment of the invention proposes to obtain a second classification result by comparing the resource directory data of the data to be classified (which can reflect the descriptive and annotation information of the data text) with the descriptive text data of each subclass.

[0083] The second classification result of this invention represents the correlation between the descriptive information and annotation information of the data to be classified and the descriptive text of each subclass.

[0084] S104, determine the subclass to which the data to be classified belongs based on the first classification result and the second classification result.

[0085] In this embodiment of the invention, the first classification result and the second classification result can be calculated based on weight calculation and threshold setting to obtain the final classification result, thereby determining the subclass to which the data to be classified belongs.

[0086] Specifically, technicians can set the weights of the first classification result and the second classification result based on their experience and needs. They can also set the confidence thresholds for the first classification result and the second classification result based on their experience and needs.

[0087] Specifically, in one optional embodiment of the present invention, a flowchart of another data classification method is provided, such as... Figure 2 As shown, the method includes the following steps:

[0088] S201, Process the data to be classified, and determine the text data and resource directory data of the data to be classified.

[0089] In this embodiment of the invention, taking into account the characteristics of both structured and unstructured data, the following specific points are proposed:

[0090] When the data to be classified is structured data, step S201 includes the following sub-steps:

[0091] S2011A, the database table field of the data to be classified already has annotations, and it is determined to be resource catalog data.

[0092] In this embodiment of the invention, structured data refers to data in the form of a database.

[0093] In this embodiment of the invention, for structured data collected by the database access method, the existing comments of the database / table / field can be obtained, and the existing comments of the database / table / field can be concatenated to obtain resource catalog data.

[0094] S2012A, Obtain the field values ​​of the data to be classified and determine them as data text.

[0095] In this embodiment of the invention, when the data to be classified is a data table, dataset, or database, the data text can be obtained based on the set of field values ​​of the data to be classified.

[0096] In this embodiment of the invention, considering that text data recorded in the same data table or dataset generally correspond to the same topic, the data can be classified by the data table or dataset.

[0097] Preferably, in this embodiment of the invention, considering that the text data recorded in the same data table or dataset generally correspond to the same topic, the data corresponding to each field has a certain similarity. Therefore, in this embodiment of the invention, a preset number of data rows can be randomly collected for each data table or dataset to obtain a set of field values ​​of the data to be classified.

[0098] Preferably, in this embodiment of the invention, the obtained set of field values ​​can be aggregated to obtain the field values ​​that appear most frequently in the set of field values ​​as the central field value, and the central field value is used as the data text.

[0099] For example, for a certain data table, the table comments of the data table can be obtained as resource directory data, and 100 rows of data in the data table can be randomly obtained as data text.

[0100] When the data to be classified is unstructured data, step S201 includes the following sub-steps:

[0101] S2011B, Obtain the file storage directory and file name of the data to be classified and determine them as resource directory data.

[0102] S2012B, the text content of the data to be classified is segmented to obtain segmented text, and the segmented text is identified as text data.

[0103] In this embodiment of the invention, unstructured data refers to data in the form of a file library.

[0104] In this embodiment of the invention, for unstructured data collected through the file library access method, the storage directory and file name of the file can be used as resource directory data.

[0105] In this embodiment of the invention, when the data to be classified is in the form of a text file, the content of the text file can be segmented according to a set segmentation threshold to obtain multiple text segments. Then, each text segment is formatted to handle issues such as garbled characters, punctuation, and special symbols. Finally, each text segment is determined as text data.

[0106] Preferably, in this embodiment of the invention, each segment of text can be input into a reading comprehension model to obtain summary information or keyword information of the text content of the data to be classified, and the summary information or keyword information can be determined as text data.

[0107] Preferably, in this embodiment of the invention, after obtaining the text data to be classified, regular expressions can be used to determine its classification label. Therefore, for some simple and standardized text data, regular expressions can be used directly to determine its classification without subsequent steps. For text data that cannot be classified using regular expressions, subsequent steps are required for classification.

[0108] S202, based on the text data of the data to be classified and the pre-trained text classification model, obtain the first classification result of the data to be classified.

[0109] Specifically, a text classification model can calculate the probability that the text data to be classified belongs to each subclass for each subclass, thereby obtaining a ranking of the probability that the text data to be classified belongs to any subclass.

[0110] Specifically, in this embodiment of the invention, the text classification model can be set according to the probability that the text data to be classified belongs to the first N subclasses. Thus, the first classification result can be expressed as the probability P1, P2, P3, etc., that the text data belongs to the first N subclasses.

[0111] Specifically, in this embodiment of the invention, step S202 may include the following sub-steps:

[0112] S2021, convert the text data of the text data to be classified into sentence vectors.

[0113] S2022, The sentence vector is input into a pre-trained text classification model to obtain the first classification result of the data to be classified.

[0114] The text classification model is trained on a semantic understanding model based on sample text data with classification labels.

[0115] Specifically, in this embodiment of the invention, text data with a determined classification is used as sample text data, and the classification of the sample text data is used as sample label.

[0116] In this embodiment of the invention, the text classification model is a neural network model based on Natural Language Processing (NLP).

[0117] In this embodiment of the invention, sample data can be obtained from a data resource library with a defined classification and grading through data or file library access. The obtained data is then processed into sample data in row format, where the first column of each row is the classification code and the second column is the sample data text.

[0118] In this embodiment of the invention, the semantic understanding model can be the ErnieFor Sequence Classification pre-trained text classification model based on Ernie. In this embodiment of the invention, a few-shot training method based on the entailment task can be used to reformulate the NLP task as an entailment task.

[0119] In this embodiment of the invention, during model training, sample text data can be represented as sentence vectors, and the probability of classifying the sample text data by sample labels can be used as the maximum expected value to train the semantic understanding model.

[0120] Specifically, in this embodiment of the invention, the pre-trained text classification model Ernie For Sequence Classification and the tokenizer based on Ernie can be loaded, and the word embedding model can be loaded. The tokenizer is used to convert the sample text data into numbers, and the batch size and the maximum truncation length of the text, max_seq_length, are adjusted for data loading. The initial learning rate of Adam and the total number of training epochs to be performed are set. Based on the Entailment task, a fully connected layer is superimposed on the last layer of Ernie For Sequence Classification, and num_classes is set to 2. The NLP fine-tune task is uniformly converted into the Entailment 2 classification task. The optimizer, loss function, evaluation metric, etc. required for training are defined, and the model is trained.

[0121] The text classification model proposed in this embodiment of the invention adopts a few-sample training method based on entailment, which enables the model to have good judgment accuracy even in scenarios with small sample sizes.

[0122] In this embodiment of the invention, the text classification model analyzes and predicts the probability of text data belonging to each category, obtains the probability of the text classification model belonging to each category, and can output the top N categories with the highest probabilities as the first classification result.

[0123] In this embodiment of the invention, the value of N can be set by technicians based on experience or requirements.

[0124] In this embodiment of the invention, artificial intelligence technology is used to assist in the classification judgment of the data to be classified through model training, thereby improving the intelligence level of the classification and grading task and significantly reducing the workload of manual classification and grading. In this embodiment of the invention, by training an NLP-based text classification model and using a few-sample training method, good judgment accuracy can be achieved even in scenarios with small sample sizes.

[0125] S203, Obtain the classification and grading determination standard table of the data to be classified.

[0126] S204, obtain each subclass and its corresponding subclass description text from the classification and grading determination specification table.

[0127] S205, perform word segmentation on the subclass description text to obtain word segmentation sets of description text corresponding to each subclass.

[0128] S206, use the word segmentation set of the description text corresponding to each subclass as the description text data of each subclass.

[0129] In this embodiment of the invention, steps S203 to S206 can be performed before step S201 or after step S201.

[0130] In this embodiment of the invention, the code of the smallest subclass and the subclass description text corresponding to each subclass in the classification and grading determination specification text of the data to be classified can be obtained and processed into a classification and grading determination specification table.

[0131] Specifically, in this embodiment of the invention, the descriptive text of each subclass can be segmented into words to obtain a set of segmented descriptive text corresponding to each subclass. Then, the classification and grading determination table of the data to be classified can be preprocessed into a classification and grading table in the format of {classification code, segmentation set}, where the first column of each row is the classification code, and the second column is the set of segmented descriptive text corresponding to that subclass using a word segmentation tool.

[0132] Preferably, in this embodiment of the invention, some word segments that have no actual meaning in each descriptive text word segmentation set can be filtered out, such as function words like "etc." and "and", and only word segments with entity meaning can be retained, so as to improve the efficiency of subsequent calculations.

[0133] S207, based on the resource catalog data and the description text data of each subclass of the data to be classified, the second classification result of the data to be classified is obtained.

[0134] Specifically, in this embodiment of the invention, step S207 may include the following sub-steps:

[0135] S2071, convert the resource catalog data of the data to be classified into a first text feature vector.

[0136] S2072, convert the description text data of each subclass into a second text feature vector.

[0137] S2073, The bag-of-words model is used to represent the first text feature vector and the second text feature vector as numerical vectors to obtain the first numerical vector and the second numerical vector.

[0138] S2074, based on the first numerical vector and the second numerical vector, the similarity value between the resource catalog data and the description text data of each subclass of the data to be classified is determined by using the cosine similarity comparison method.

[0139] S2075, determine the second classification result based on the similarity value between the resource catalog data of the data to be classified and the description text data of each subclass.

[0140] In this embodiment of the invention, the resource catalog data and the description text data of each subclass of the data to be classified can be judged for similarity to obtain the similarity between the resource catalog data and the description text data of each subclass of the data to be classified. Then, based on the similarity, they are sorted. Therefore, in this embodiment of the invention, the second classification result can output the classification corresponding to the top M subclass description text data with the highest similarity. The second classification result can represent the classification corresponding to the top M subclass description text data with the highest similarity and the similarity scores Q1, Q2, Q3…

[0141] In this embodiment of the invention, the value of M is set by a technician based on experience or requirements. The values ​​of M and N can be the same or different.

[0142] S208, Based on the first classification result and the second classification result, determine the subclass to which the data to be classified belongs.

[0143] Specifically, in this embodiment of the invention, the weights of the first classification result and the second classification result, as well as the confidence thresholds of the first classification result and the second classification result, can be set by technicians based on experience or needs.

[0144] The classification result with a probability greater than the confidence threshold corresponding to the first classification result among the multiple possible categories included in the first classification result is taken as the first credible category, and the classification result with a similarity greater than the confidence threshold corresponding to the second classification result among the multiple possible categories included in the second classification result is taken as the second credible category.

[0145] Preferably, in this embodiment of the invention, the probability or similarity of unreliable results included in the first classification result and the second classification result can be set to zero.

[0146] Furthermore, in this embodiment of the invention, the final result can be obtained by comprehensively calculating the probability corresponding to the first trust category and the similarity corresponding to the same category in the second trust category.

[0147] In this embodiment of the invention, by preprocessing the data to be classified and collecting and preprocessing the classification and grading determination standard text, text data of the data to be classified, resource catalog data, and a classification and grading determination standard table are obtained. Thus, a first classification result can be obtained based on the text classification model and text data determination. In this embodiment of the invention, considering the management characteristics of databases and file repositories, a second classification result is obtained by comparing the word vector similarity between the resource catalog data of the data to be classified and the description text data of each subclass in the classification and grading determination standard table. Finally, based on the weight calculation and threshold setting of the first and second classification results, the final classification result is obtained, determining the subclass to which the data to be classified belongs.

[0148] Therefore, this invention overcomes the limitations of current regular expressions in terms of applicability, enabling the determination of all possible subcategories to which the data to be classified may belong. Furthermore, this invention also achieves intelligent determination of unstructured data, especially application files. Thus, this invention improves the overall intelligence of classification tasks. In addition, this invention utilizes model training to assist in classification, significantly reducing the workload of manual classification and grading, and is applicable to data classification across various industries.

[0149] The technical solution provided by the embodiments of the present invention can be applied to the field of data security management technology. Specifically, in the embodiments of the present invention, the security level of each subclass can be classified based on the classification code. Therefore, while obtaining the subclass to which the data to be classified belongs, the security level of the data to be classified can also be determined, so that the data can be managed for security based on the security level of the data to be classified.

[0150] The technical solutions provided in this invention can also be applied to: data asset management in data middleware platforms, data classification and grading management in business systems, and support for data security tools in business systems. They are applicable to common big data management platforms and can be used in scenarios requiring comprehensive data security control through data classification and grading, making classification and grading determination tools intelligent and automated.

[0151] Based on the same inventive concept, embodiments of the present invention provide a data classification device. (See reference...) Figure 3 , Figure 3 This is a schematic diagram of the data classification device provided in an embodiment of the present invention. Figure 3 As shown, the device includes:

[0152] Data processing module 301 is used to process the data to be classified and determine the text data and resource directory data of the data to be classified.

[0153] The first classification module 302 is used to obtain the first classification result of the data to be classified based on the text data of the data to be classified and the pre-trained text classification model;

[0154] The second classification module 303 is used to obtain the second classification result of the data to be classified based on the resource catalog data and the description text data of each subclass of the data to be classified; the description text data of each subclass is obtained based on the subclass description text in the classification and grading determination specification table corresponding to the data to be classified.

[0155] The result determination module 304 is used to determine the subclass to which the data to be classified belongs based on the first classification result and the second classification result.

[0156] Optionally, when the data to be classified is structured data, the data processing module 301 is specifically used for:

[0157] The database table fields containing the data to be classified already have comments, confirming that they are resource catalog data.

[0158] Obtain the field values ​​of the data to be classified and determine them as data text.

[0159] Optionally, when the data to be classified is unstructured data, the data processing module 301 is specifically used for:

[0160] The file storage directory and file name of the data to be classified are determined as resource directory data;

[0161] The text content of the data to be classified is segmented to obtain fragmented text, and the fragmented text is identified as text data.

[0162] Optionally, the device further includes:

[0163] The first acquisition module is used to acquire the classification and grading determination standard table of the data to be classified.

[0164] The second acquisition module is used to acquire each subclass and the subclass description text corresponding to each subclass in the classification and grading judgment specification table;

[0165] The word segmentation module is used to perform word segmentation on the subclass description text to obtain the word segmentation set of description text corresponding to each subclass.

[0166] The determination module is used to use the word segmentation set of the description text corresponding to each subclass as the description text data of each subclass.

[0167] Optionally, the first classification module 302 is specifically used for:

[0168] Convert the text data of the text data to be classified into sentence vectors;

[0169] The sentence vector is input into a pre-trained text classification model to obtain the first classification result of the data to be classified;

[0170] The text classification model is trained on a semantic understanding model based on sample text data with classification labels.

[0171] Optionally, the second classification module 303 is specifically used for:

[0172] Convert the resource catalog data of the data to be classified into a first text feature vector;

[0173] Convert the text data describing each subclass into a second text feature vector;

[0174] The bag-of-words model is used to represent the first text feature vector and the second text feature vector as numerical vectors, resulting in the first numerical vector and the second numerical vector.

[0175] Based on the first and second numerical vectors, the cosine similarity comparison method is used to determine the similarity value between the resource catalog data and the description text data of each subclass of the data to be classified.

[0176] The second classification result is determined based on the similarity value between the resource catalog data of the data to be classified and the description text data of each subclass.

[0177] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0178] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the steps of the data classification method described in any of the above embodiments.

[0179] This invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps in the data classification method described in any of the above embodiments.

[0180] This invention also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the data classification method described in any of the above embodiments.

[0181] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0182] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0183] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0184] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0185] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0186] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0187] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0188] The above provides a detailed description of the data classification method, electronic device, storage medium, and computer program product provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data classification method, characterized in that, The method includes: Process the data to be classified to determine the text data and resource directory data of the data to be classified; Based on the text data of the data to be classified and the pre-trained text classification model, the first classification result of the data to be classified is obtained; Based on the resource catalog data and subclass description text data of the data to be classified, the second classification result of the data to be classified is obtained; the subclass description text data is obtained based on the subclass description text in the classification and grading determination specification table corresponding to the data to be classified. Based on the first classification result and the second classification result, determine the subclass to which the data to be classified belongs; When the data to be classified is structured data, the data to be classified is processed to determine the text data and resource directory data of the data to be classified, including: The database table fields containing the data to be classified already have comments, confirming that they are resource catalog data. Obtain the field values ​​of the data to be classified and determine them as data text; When the data to be classified is unstructured data, the data to be classified is processed to determine the text data and resource directory data of the data to be classified, including: The file storage directory and file name of the data to be classified are determined as resource directory data; The text content of the data to be classified is segmented to obtain fragmented text, and the fragmented text is identified as text data.

2. The data classification method according to claim 1, characterized in that, The method further includes: Obtain the classification and grading standard table of the data to be classified; Obtain each subclass and its corresponding subclass description text from the classification and grading determination specification table; The subclass description text is segmented into words to obtain the word segmentation set of description text corresponding to each subclass. The word segmentation set of the description text corresponding to each subclass is used as the description text data of each subclass.

3. The data classification method according to claim 1, characterized in that, Based on the text data of the data to be classified and a pre-trained text classification model, a first classification result for the data to be classified is obtained, including: Convert the text data of the data to be classified into sentence vectors; The sentence vector is input into a pre-trained text classification model to obtain the first classification result of the data to be classified; The text classification model is trained on a semantic understanding model based on sample text data with classification labels.

4. The data classification method according to claim 1, characterized in that, Based on the resource catalog data and subclass description text data of the data to be classified, a second classification result is obtained for the data to be classified, including: Convert the resource catalog data of the data to be classified into a first text feature vector; Convert the text data describing each subclass into a second text feature vector; The bag-of-words model is used to represent the first text feature vector and the second text feature vector as numerical vectors, resulting in the first numerical vector and the second numerical vector. Based on the first and second numerical vectors, the cosine similarity comparison method is used to determine the similarity value between the resource catalog data and the description text data of each subclass of the data to be classified. The second classification result is determined based on the similarity value between the resource catalog data of the data to be classified and the description text data of each subclass.

5. A data classification device, characterized in that, The device includes: The data processing module is used to process the data to be classified and to determine the text data and resource directory data of the data to be classified. The first classification module is used to obtain the first classification result of the data to be classified based on the text data of the data to be classified and the pre-trained text classification model; The second classification module is used to obtain the second classification result of the data to be classified based on the resource catalog data and the description text data of each subclass of the data to be classified; the description text data of each subclass is obtained based on the subclass description text in the classification and grading determination specification table corresponding to the data to be classified. The result determination module is used to determine the subclass to which the data to be classified belongs based on the first classification result and the second classification result; When the data to be classified is structured data, the data processing module is specifically used for: The database table fields containing the data to be classified already have comments, confirming that they are resource catalog data. Obtain the field values ​​of the data to be classified and determine them as data text; When the data to be classified is unstructured data, the data processing module is specifically used for: The file storage directory and file name of the data to be classified are determined as resource directory data; The text content of the data to be classified is segmented to obtain fragmented text, and the fragmented text is identified as text data.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the data classification method according to any one of claims 1-4.

7. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the data classification method according to any one of claims 1-4.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the data classification method according to any one of claims 1-4.

Images