Data classification and grading method and system, electronic equipment and storage medium

Abstract

The invention provides a data classification and grading method and system, electronic equipment and a storage medium, and the method comprises the steps: connecting a data source to obtain target data, and extracting a field name and a field meaning of the target data; a process engine is utilized to acquire a plurality of classification and grading methods matched with the target data from a classification and grading method set, and the classification and grading methods comprise a target classification and grading method; performing classification and grading processing on the field names and the field meanings of the target data by adopting the classification and grading methods in sequence, and obtaining classification and grading results of the target data processed by each classification and grading method; optimizing the target classification and grading method according to classification and grading results corresponding to all classification and grading methods; and performing classification and grading processing on the target data by using the optimized target classification and grading method to obtain a target classification and grading result.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data classification and grading method, system, electronic device and storage medium. Background Art

[0002] In today's rapidly developing digital society, large amounts of data are constantly generated and accumulated, and classification and grading systems are essential for effectively organizing, managing, and protecting data. There are three main methods for classifying and grading structured data: one is to formulate rules from data; the second is to train classification models based on feature vectors and use machine learning or natural language processing methods to predict the category of new data; the third is to extract features from metadata and represent them as feature vectors. Commonly used technologies include word segmentation, frequent item sets, word2vec, etc.

[0003] However, these methods generally have the problem of being difficult to quickly adapt to the actual project environment when applied to new scenarios and being unable to accurately select appropriate methods to handle data classification and grading. This not only affects efficiency and accuracy, but also requires continuous human intervention, resulting in a waste of resources. Summary of the invention

[0004] The present disclosure provides a data classification and grading method, system, electronic device and storage medium to at least solve the above technical problems existing in the prior art.

[0005] According to a first aspect of the present disclosure, a data classification and grading method is provided, the method comprising:

[0006] Connecting to a data source to obtain target data, and extracting field names and field meanings of the target data;

[0007] Using a process engine, obtaining multiple classification and grading methods matching the target data from a classification and grading method set, including a target classification and grading method;

[0008] The classification and grading methods are sequentially used to classify and grade the field names and field meanings of the target data, and the classification and grading results of the target data processed by each classification and grading method are obtained;

[0009] Optimize the target classification and grading method according to the classification and grading results corresponding to all classification and grading methods;

[0010] The target data is classified and graded using the optimized target classification and grading method to obtain a target classification and grading result.

[0011] In one possible implementation, before connecting to the data source, the method further includes:

[0012] Establishing a classification and grading method set, wherein the classification and grading method set includes a plurality of classification and grading methods;

[0013] By using a process engine, an execution process and a preset weight are arranged for each classification and grading method in the classification and grading method set.

[0014] In one possible implementation, when the target data has complete metadata, extracting the field name and field meaning of the target data includes:

[0015] When the target data is structured data, query the metadata table corresponding to the relational database to which the target data belongs to obtain the field name and field meaning of the target data;

[0016] When the target data is in an unstructured format, the file name and header name of the target data are parsed to obtain the field names and field meanings of the target data.

[0017] In one possible implementation, when the target data does not have complete metadata, extracting the field name and field meaning of the target data includes:

[0018] Acquire sample data of the target data, analyze the sample data, and infer the field names and field meanings of the target data;

[0019] Alternatively, a regular expression is used to obtain the field name of the target data and the Chinese name matching the field name to obtain the field meaning of the target data;

[0020] Alternatively, the field name of the target data is obtained, and the field meaning corresponding to the field name is obtained according to a mapping dictionary established in advance.

[0021] In one possible implementation, the classification and grading methods are sequentially used to classify and grade the field names and field meanings of the target data, and the classification and grading results of the target data processed by each classification and grading method are obtained, including:

[0022] Using the process engine, the processing order is arranged according to the preset weight of each classification and grading method;

[0023] The field names and field meanings of the target data are classified and graded in turn according to the processing order to obtain the classification and grading results of processing the target data by each classification and grading method.

[0024] In one possible implementation, the optimizing the target classification and grading method according to the classification and grading results corresponding to all the classification and grading methods includes:

[0025] If the target classification and grading method correctly identifies the level and category of the target data, the model of the target classification and grading method is optimized using the classification and grading results corresponding to all other classification and grading methods to avoid overfitting;

[0026] If the target classification and grading method cannot correctly identify the level and category of the target data, the model parameters of the target classification and grading method are adjusted according to the classification and grading results corresponding to all other classification and grading methods.

[0027] In one embodiment, the method further comprises:

[0028] The preset weights of the classification and grading method are adjusted according to the alarm information and the classification and grading result report output in real time.

[0029] According to a second aspect of the present disclosure, a data classification and grading system is provided, the system comprising:

[0030] An acquisition module connects to a data source to acquire target data and extracts field names and field meanings of the target data;

[0031] A process engine module, used for acquiring a plurality of classification and grading methods matching the target data from a classification and grading method set, including a target classification and grading method;

[0032] The process engine module is further used to sequentially use the classification and grading methods to classify and grade the field names and field meanings of the target data, and obtain the classification and grading results of each classification and grading method for processing the target data;

[0033] An optimization module, used for optimizing the target classification and grading method according to the classification and grading results corresponding to all classification and grading methods;

[0034] The process engine module is also used to classify and classify the target data using the optimized target classification and grading method to obtain a target classification and grading result.

[0035] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0036] at least one processor; and

[0037] a memory communicatively connected to the at least one processor; wherein,

[0038] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.

[0039] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure.

[0040] The data classification and grading method, system, electronic device and storage medium disclosed in the present invention first connect and obtain the data source, extract the field name and field meaning from the obtained target data; use the process engine to search and match multiple classification and grading methods that best match the characteristics of the target data in the classification and grading method set, including an optimal target classification and grading method. These classification and grading methods are used in turn to classify and grade the field names and field meanings of the target data, and the target classification and grading method is optimized according to the results to improve its adaptability and accuracy. Finally, the optimized target classification and grading method is used to perform final classification and grading on the target data to obtain stable and accurate classification and grading results. In this way, by intelligently matching and optimizing the classification and grading methods, the system can quickly adapt to new scenarios to obtain stable and accurate classification and grading results, while reducing human intervention and improving processing efficiency and accuracy.

[0041] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0043] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0044] Figure 1 A schematic diagram of the implementation process of the data classification and grading method according to an embodiment of the present disclosure is shown;

[0045] Figure 2 A schematic diagram of the implementation process of establishing a classification and grading method set and arranging the execution process and preset weights of each classification and grading method in the classification and grading method set is shown in an embodiment of the present disclosure;

[0046] Figure 3 The schematic diagram shows the implementation process of extracting the field name and field meaning of the target data in the embodiment of the present disclosure. Figure 1 ;

[0047] Figure 4 The schematic diagram shows the implementation process of extracting the field name and field meaning of the target data in the embodiment of the present disclosure. Figure 2 ;

[0048] Figure 5 A schematic diagram of the implementation process of obtaining the classification and grading results of the target data by each classification and grading method according to an embodiment of the present disclosure is shown;

[0049] Figure 6 A schematic diagram of the implementation process of optimizing the target classification and grading method according to an embodiment of the present disclosure is shown;

[0050] Figure 7 A schematic diagram of the structure of a data classification and grading system according to an embodiment of the present disclosure is shown;

[0051] Figure 8 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0052] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0053] The present disclosure provides a data classification and grading method, such as Figure 1 As shown, the method includes:

[0054] Step 101: Connect to a data source to obtain target data, and extract field names and field meanings of the target data.

[0055] In this example, connecting to a data source includes: 1. Obtaining a data source, including obtaining the data source by actively scanning the data source in the host, analyzing the data source information in the log, or manually reporting the data source information by the user. 2. After obtaining the data source, collect basic information about the data source and determine the type of data source. For example, a relational database, a CSV file, an Excel spreadsheet, or another type of data source. 3. Determine the connection driver of the data source based on the type of data source, and connect to the data source based on the pre-adapted connection driver. 4. Establish a connection or obtain access rights to the data source to obtain data and perform subsequent operations.

[0056] After connecting to the data source, the acquired data is cleaned, which mainly includes the following processing:

[0057] Missing value processing: Detect and process missing values ​​in the data, which can be done by filling, deleting or interpolating. Outlier processing: Detect and process outliers, using statistical methods or business rule-based methods. Duplicate value processing: Detect and process duplicate values ​​in the data to ensure the uniqueness of the data. Data type conversion: Ensure that each field has the correct data type and perform necessary data type conversion. Store the cleaned target data in the target location specified by the system, such as a database, file system or other data storage medium.

[0058] For example: Terminal number: "12" is a character type, which needs to be converted to the numeric type 12; Online or not: No, it needs to be converted to a Boolean value true or false; Terminal type list: linux-centos, windows-server-2016, linux-centos, after filtering duplicate values, it needs to be converted to linux-centos, windows-server-2016.

[0059] After data cleaning, the target data is obtained, and then the field name and field meaning of the target data are extracted. The field name is usually in English, and the field meaning is the Chinese name or comment corresponding to the field name. For example: the field name is "user_name", the field meaning is "user's name", and its corresponding label is "name"; the field name is "voice_dura", the field meaning is "the duration of making or receiving calls this month", and its corresponding label should be "call duration". There are different methods for obtaining the field name and field meaning of target data for different types of target data, which will be described in detail in subsequent embodiments.

[0060] Step 102: Utilizing a process engine, a plurality of classification and grading methods matching the target data are obtained from a classification and grading method set, including a target classification and grading method.

[0061] In this example, the process engine is a key component set in the data classification and grading system, which is used to define, monitor and optimize the processing flow of data classification and grading, and can automatically execute each step in the processing flow. The classification and grading method set covers a variety of classification and grading methods. The process engine intelligently searches and matches the most appropriate multiple classification and grading methods in the classification and grading method set according to the characteristics of the target data (such as field name, field meaning, data type, security level requirements, etc.), including the target classification and grading method with the highest matching degree with the target data.

[0062] Step 103: Use the classification and grading methods in turn to classify and grade the field names and field meanings of the target data, and obtain the classification and grading results of each classification and grading method for processing the target data.

[0063] In this example, classification and grading are performed by inputting the field names and field meanings of the target data into the models corresponding to different classification and grading methods in turn, thereby obtaining the classification and grading results of the target data for each classification and grading method, including the type and level of the target data.

[0064] Step 104: Optimize the target classification and grading method according to the classification and grading results corresponding to all classification and grading methods.

[0065] In this example, the classification and grading results of the target classification and grading method may not meet expectations in many ways. Therefore, it is necessary to use the classification and grading results corresponding to other classification and grading methods to adjust the model parameters of the target classification and grading method, aiming to improve the adaptability of the target classification and grading method and ensure that it can output stable classification and grading results in various situations. The specific adjustment and optimization process will be described in detail in the subsequent embodiments.

[0066] Step 105: classify and classify the target data using the optimized target classification and grading method to obtain a target classification and grading result.

[0067] In this example, the optimized target classification and grading method has the ability to accurately output the classification and grading results of the target data, and obtain the target classification and grading results.

[0068] The present disclosure provides a data classification and grading method, which first connects and obtains the data source, extracts the field name and field meaning from the obtained target data; uses the process engine to search and match multiple classification and grading methods that best match the target data characteristics in the classification and grading method set, including an optimal target classification and grading method. These classification and grading methods are used in turn to classify and grade the field names and field meanings of the target data, and the target classification and grading method is optimized according to the results to improve its adaptability and accuracy. Finally, the optimized target classification and grading method is used to perform final classification and grading on the target data to obtain stable and accurate classification and grading results. In this way, by intelligently matching and optimizing the classification and grading methods, the system can quickly adapt to new scenarios to obtain stable and accurate classification and grading results, while reducing human intervention and improving processing efficiency and accuracy.

[0069] In one example, before connecting to a data source, such as Figure 2 As shown, the method also includes:

[0070] Step 201: Establish a classification and grading method set, wherein the classification and grading method set includes multiple classification and grading methods.

[0071] In one example, the classification and grading method set includes at least: Logistic Regression, Decision Trees, Support Vector Machines (SVM), Naive Bayes, K-Nearest Neighbors (KNN), Random Forest, Gradient Boosting Trees, Neural Networks, XGBoost and other classification and grading methods. The number of classification and grading methods in the classification and grading method set is greater than or equal to the number of classification and grading methods matched by the process engine for the target data.

[0072] Step 202: Utilize a process engine to arrange execution processes and preset weights for each classification and grading method in the classification and grading method set.

[0073] In this example, before the process engine processes data, you first need to write a Business Process Model and Notation (BPMN) file. A BPMN file is a file that uses standard symbols and methods to draw business process diagrams, which records the steps, decision points, participants, and information flows of a business process.

[0074] By writing a BPMN file, specifying the execution process and initial weight of each classification and grading method in the file, and importing the file into the data classification and grading system, when the data flow arrives, the process engine selects a classification and grading method from the classification and grading method set based on the BPMN file, and automatically executes the processing flow of each classification and grading method.

[0075] In one example, when the target data has complete metadata, the field name and field meaning of the target data are extracted, such as Figure 3 As shown, including:

[0076] Step 301: When the target data is structured data, query the metadata table corresponding to the relational database to which the target data belongs to obtain the field name and field meaning of the target data.

[0077] In this example, structured data is usually stored in a relational database, such as MySQL, PostgreSQL, Oracle, etc. The data is organized in the form of tables, each table consists of multiple fields (columns), and each field stores a specific type of information.

[0078] When the target data is structured data, first, you need to determine the relational database where the target data is located. A relational database usually contains one or more metadata tables (also called system tables, information model tables), which store the structural information of all tables in the database, including table name, field name, field data type, field length, field meaning, etc. By querying these metadata tables, you can accurately obtain the field name and field meaning of the target data table (that is, the table to which the target data belongs).

[0079] For example, for MySQL, information_schema is a special database that contains information about all other databases, including table structure and field information. Therefore, you can obtain this information by querying related tables in information_schema (such as the COLUMNS table).

[0080] Step 302: When the target data is in an unstructured format, parse the file name and header name of the target data to obtain the field name and field meaning of the target data.

[0081] In this example, unstructured data usually refers to data that is not stored in the form of relational database tables, such as CSV files, Excel files, etc. These files usually contain a header (the first line), which is the field name. By reading and parsing the file names and header names of these unstructured data, the field meanings of the target data can be inferred.

[0082] In one example, when the target data does not have complete metadata, the field name and field meaning of the target data are extracted, such as Figure 4 As shown, including:

[0083] Step 401: Acquire sample data of the target data, analyze the sample data, and infer the field names and field meanings of the target data.

[0084] In this example, the sample data of the target data is part of the target data provided by the data source provider. The value of this part of the data can be anonymized or completely fake data, but the data format must be the same as the real data. Observe the data format and characteristics of each field in the target data, such as phone number, ID number, name, user name, and IP address. Based on the data format and characteristics, infer the meaning of each field.

[0085] For example, the following field is extracted from the sample data provided by the data source provider: "id":"110111**********2X". Based on the length and format of this field, it is inferred that the field is the ID card number, so id->ID card number is used as the field name and field meaning of the target data.

[0086] For another example, the following fields are extracted from the sample data: {"tel":"131****1111","id":"110111**********2X","name":"张三","username":"zhangs121","adress":"172.12.21.33"}. The format of the tel field is similar to a phone number, so it can be inferred that it is a phone number; the format of the id field is similar to an ID card number, so it can be inferred that it is an ID card number; the content of the adress field looks like an IP address, so it can be inferred that it is an IP address. In this way, the field names and field meanings of most of the target data can be obtained.

[0087] Step 402: Obtain the field names of the target data and the Chinese names matching the field names in the form of regular expressions to obtain the field meanings of the target data.

[0088] In this example, regular expressions are written according to the naming rules of field names and field meanings. Obtain the field names of the target data, and use regular expressions to match the field names of the target data to identify the field names that conform to specific naming rules. For example, using the regular expression \b(user[._]?name)\b can match field names in the forms of userName, username, user.name, user_name, etc. Similarly, according to the field meanings, they can also be obtained by matching with corresponding regular expressions.

[0089] Step 403: Obtain the field names of the target data, and according to the pre-established mapping dictionary, obtain the field meanings corresponding to the field names.

[0090] In this example, through the pre-established mapping dictionary, the field names are mapped to their corresponding field meanings. For example, according to business requirements and data characteristics, a mapping dictionary from field names to field meanings is established, such as {user.info1: "User information of the First Research Center"}. When the field names of the target data are obtained, the corresponding field meanings can be obtained by querying the mapping dictionary.

[0091] Export the extracted target data field names and field meanings into a table, JSON, or other formats for subsequent analysis and use.

[0092] In an example, the classification and grading method is sequentially used to perform classification and grading processing on the field names and field meanings of the target data, and the classification and grading results of the target data by each classification and grading method are obtained, as Figure 5 shown, including:

[0093] Step 501: Using the process engine, arrange the processing order according to the preset weight of each classification and grading method.

[0094] In this example, the compiled BPMN file sets a preset weight for each classification and grading method in the classification and grading set. After using the process engine to obtain multiple classification and grading methods matching the target data from the classification and grading set, the field names and field meanings of the target data are classified and graded in turn according to the preset weight order of each matching classification and grading method to obtain the classification and grading results corresponding to each classification and grading method.

[0095] Step 502: Classify and grade the field names and field meanings of the target data in sequence according to the processing order to obtain classification and grading results of processing the target data by each classification and grading method.

[0096] In this example, the process engine first executes the target classification and grading method to classify and grade the field names and field meanings of the target data, and then executes other classification and grading methods in sequence. For example, according to the preset weights, different classification and grading methods are executed on the field names and field meanings of the target data. The preset weights are in the order of method A, method B, method C, etc. Among them, method A focuses on identifying keywords in the field names and field meanings of the target data; method B focuses on extracting the specific values ​​of the field names and field meanings of the target data for classification and grading; method C uses a large number of template rules preset in advance to classify and grade the target data.

[0097] Each method analyzes the field name and meaning of the target data. For example, method A searches for sensitive words such as "user information" and "password" in the field name to determine whether the field belongs to the category that requires special protection. Method B classifies and grades the target data by identifying the phone number, ID number, etc. in the field name and field meaning.

[0098] After each classification and grading method is executed, a classification and grading result will be generated according to the logic of the method, including the category and level of the target data. If a classification and grading method successfully assigns a category and level to the target data item (i.e., "hit label"), such as "confidential data" or "level 1", the result can be used directly and the data item can be processed accordingly, or further optimized. If it is unsuccessful (i.e., the label is not hit), continue to the next method, or use the preset default label and classification and grading.

[0099] If all preset classification and grading methods cannot assign a valid category and level to the data item, a default label will be used, which is associated with the default classification and level (such as "General Information, Level 3") to ensure that all data items can be processed.

[0100] In one example, the target classification and grading method is optimized according to the classification and grading results corresponding to all classification and grading methods, such as Figure 6 As shown, including:

[0101] Step 601: If the target classification and grading method correctly identifies the level and category of the target data, the model of the target classification and grading method is optimized using the classification and grading results corresponding to all other classification and grading methods to avoid overfitting.

[0102] In this example, if the target classification and grading method (method A) can correctly identify the level and category of the target data. However, even in this case, the potential value of other classification and grading methods (such as methods B, C, etc.) should not be ignored. In order to further improve the generalization ability of the model of the target classification and grading method and avoid overfitting (i.e., the model performs well on the training data but performs poorly on unseen data), the classification and grading results of all other classification and grading methods can be used as additional information or "regularization terms" to optimize the model.

[0103] Specifically, the outputs of other classification and grading methods can be fused with the output of the target classification and grading method (such as weighted average, voting, etc.) to produce more robust prediction results. It is also possible to introduce constraints on the output results of other classification and grading methods during the training process of the model of the target classification and grading method to limit the complexity of the model of the target classification and grading method and prevent it from overfitting the training data. It is also possible to use the different interpretations of the training data by the models of other classification and grading methods to generate more diverse training samples, thereby enhancing the generalization ability of the model of the target classification and grading method.

[0104] Step 602: If the target classification and grading method cannot correctly identify the level and category of the target data, the model parameters of the target classification and grading method are adjusted according to the classification and grading results corresponding to all other classification and grading methods.

[0105] In this example, the target classification method failed to correctly identify the levels and categories of the target data. At this point, the model parameters of the target classification method model need to be adjusted based on the correct results of all other classification methods.

[0106] Specifically, according to the classification and grading results of other classification and grading methods, re-evaluate and adjust the key parameters (such as learning rate, weight, bias, etc.) of the model of the target classification and grading method in order to improve its recognition performance. It is also possible to analyze the features used by the models of other classification and grading methods to identify the target data, and consider introducing these features or feature transformation methods into the model of the target classification and grading method to enhance its recognition ability. If the classification and grading results of other classification and grading methods are very reliable, you can consider using their classification and grading results as one of the inputs of the model of the target classification and grading method, or adopt more complex model fusion strategies (such as stacking generalization Stacking, Mixture of Experts, etc.) to combine the classification and grading results of multiple models to form the final prediction output.

[0107] In one example, the method further includes: adjusting preset weights of the classification and grading method according to the alarm information and the classification and grading result report output in real time.

[0108] In this example, when anomalies, errors, or performance indicators fall below preset thresholds during the classification and grading process, the classification and grading system triggers an alarm. These alarms include, but are not limited to, a sudden drop in accuracy, excessive time consumption, data source connection problems, etc. At the same time, the system will continue to generate and export real-time reports containing classification and grading results, which list in detail the data of each operation, such as each model's recognition of different data sources, the accuracy of recognition, and the time required to complete these operations.

[0109] Users or system administrators will analyze various indicators (especially accuracy and time consumption) based on alarm information or regular review of real-time classification and grading result reports. If the model performance of the classification and grading method is found to be poor (such as accuracy lower than expected, too long), it is necessary to consider adjusting the weight of the method to optimize the overall efficiency and accuracy of the classification and grading method model.

[0110] Users or system administrators can modify the BPMN file in the system and then re-import it into the process engine. In this way, the system will perform subsequent classification and grading tasks according to the new weight configuration. By reducing the weight of inefficient or low-accuracy methods and increasing the weight of efficient or high-accuracy methods, the system can allocate resources more intelligently and improve the overall classification and grading performance.

[0111] In addition, the optimization of the system is an ongoing process. As data continues to accumulate and classification and grading tasks continue to proceed, the system will continue to adjust and optimize the weights of the classification and grading methods based on new feedback. In this way, the system can gradually adapt to changes in data and the evolution of needs, maintaining efficient and accurate classification and grading capabilities.

[0112] The present disclosure also provides a data classification and grading system, such as Figure 7 As shown, the system comprises:

[0113] The acquisition module 701 connects to the data source to acquire the target data, and extracts the field name and field meaning of the target data;

[0114] The process engine module 702 is used to obtain a plurality of classification and grading methods matching the target data from the classification and grading method set, including the target classification and grading method;

[0115] The process engine module 702 is further used to sequentially use the classification and grading methods to classify and grade the field names and field meanings of the target data, and obtain the classification and grading results of each classification and grading method for processing the target data;

[0116] An optimization module 703 is used to optimize the target classification and grading method according to the classification and grading results corresponding to all classification and grading methods;

[0117] The process engine module 702 is also used to classify and classify the target data using the optimized target classification and grading method to obtain a target classification and grading result.

[0118] In one example, the system also includes: an establishment module, which is used to establish a classification and grading method set before connecting to a data source, and the classification and grading method set includes multiple classification and grading methods; a process engine, which is also used to orchestrate execution processes and preset weights for each classification and grading method in the classification and grading method set before connecting to a data source.

[0119] In one example, when the target data has complete metadata, the acquisition module 701 is specifically used to:

[0120] When the target data is structured data, query the metadata table corresponding to the relational database to which the target data belongs to obtain the field name and field meaning of the target data;

[0121] When the target data is in an unstructured format, the file name and header name of the target data are parsed to obtain the field names and field meanings of the target data.

[0122] In an example, when the target data does not have complete metadata, the acquisition module 701 is specifically used to:

[0123] Acquire sample data of the target data, analyze the sample data, and infer the field names and field meanings of the target data;

[0124] Alternatively, a regular expression is used to obtain the field name of the target data and the Chinese name matching the field name to obtain the field meaning of the target data;

[0125] Alternatively, the field name of the target data is obtained, and the field meaning corresponding to the field name is obtained according to a mapping dictionary established in advance.

[0126] In one example, when the process engine module 702 obtains the classification and grading results of processing the target data by each classification and grading method, it is specifically used to:

[0127] Arrange the processing order according to the preset weight of each classification and grading method; classify and grade the field names and field meanings of the target data in turn according to the processing order to obtain the classification and grading results of each classification and grading method for processing the target data.

[0128] In one example, the optimization module 703 is specifically used to:

[0129] If the target classification and grading method correctly identifies the level and category of the target data, the model of the target classification and grading method is optimized using the classification and grading results corresponding to all other classification and grading methods to avoid overfitting;

[0130] If the target classification and grading method cannot correctly identify the level and category of the target data, the model parameters of the target classification and grading method are adjusted according to the classification and grading results corresponding to all other classification and grading methods.

[0131] In one example, the system further includes: an adjustment module for adjusting the preset weights of the classification and grading method according to the alarm information and the classification and grading result report output in real time.

[0132] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0133] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0134] like Figure 8As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0135] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0136] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as data classification and grading methods. For example, in some embodiments, the data classification and grading method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the data classification and grading method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the data classification and grading method in any other appropriate manner (e.g., by means of firmware).

[0137] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0139] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0140] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0141] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0142] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0143] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0144] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0145] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A data classification and grading method, characterized in that: The method comprises: Connecting to a data source to obtain target data, and extracting field names and field meanings of the target data; Using a process engine, obtaining multiple classification and grading methods matching the target data from a classification and grading method set, including a target classification and grading method; Using the classification and grading methods in sequence to classify and grade the field names and field meanings of the target data, and obtaining classification and grading results of the target data processed by each classification and grading method; Optimize the target classification and grading method according to the classification and grading results corresponding to all classification and grading methods; The target data is classified and graded using the optimized target classification and grading method to obtain a target classification and grading result.

2. The method according to claim 1, characterized in that Before connecting to the data source, the method further includes: Establishing a classification and grading method set, wherein the classification and grading method set includes a plurality of classification and grading methods; By using a process engine, an execution process and a preset weight are arranged for each classification and grading method in the classification and grading method set.

3. The method according to claim 1, characterized in that When the target data has complete metadata, the step of extracting the field names and field meanings of the target data includes: When the target data is structured data, query the metadata table corresponding to the relational database to which the target data belongs to obtain the field name and field meaning of the target data; When the target data is in an unstructured format, the file name and header name of the target data are parsed to obtain the field names and field meanings of the target data.

4. The method according to claim 1, characterized in that When the target data does not have complete metadata, the step of extracting the field name and field meaning of the target data includes: Acquire sample data of the target data, analyze the sample data, and infer the field names and field meanings of the target data; Alternatively, a regular expression is used to obtain the field name of the target data and the Chinese name matching the field name to obtain the field meaning of the target data; Alternatively, the field name of the target data is obtained, and the field meaning corresponding to the field name is obtained according to a mapping dictionary established in advance.

5. The method according to claim 2, characterized in that: The step of sequentially using the classification and grading methods to classify and grade the field names and field meanings of the target data, and obtaining the classification and grading results of each classification and grading method for processing the target data, includes: Using the process engine, the processing order is arranged according to the preset weight of each classification and grading method; The field names and field meanings of the target data are classified and graded in turn according to the processing order to obtain the classification and grading results of processing the target data by each classification and grading method.

6. The method according to claim 1, characterized in that The optimizing the target classification and grading method according to the classification and grading results corresponding to all the classification and grading methods comprises: If the target classification and grading method correctly identifies the level and category of the target data, the model of the target classification and grading method is optimized using the classification and grading results corresponding to all other classification and grading methods to avoid overfitting; If the target classification and grading method cannot correctly identify the level and category of the target data, the model parameters of the target classification and grading method are adjusted according to the classification and grading results corresponding to all other classification and grading methods.

7. The method according to claim 2, characterized in that: The method further comprises: The preset weights of the classification and grading method are adjusted according to the alarm information and the classification and grading result report output in real time.

8. A data classification and grading system, characterized in that: The system comprises: An acquisition module connects to a data source to acquire target data and extracts field names and field meanings of the target data; A process engine module, used for acquiring a plurality of classification and grading methods matching the target data from a classification and grading method set, including a target classification and grading method; The process engine module is further used to sequentially use the classification and grading methods to classify and grade the field names and field meanings of the target data, and obtain the classification and grading results of each classification and grading method for processing the target data; An optimization module, used for optimizing the target classification and grading method according to the classification and grading results corresponding to all classification and grading methods; The process engine module is also used to classify and classify the target data using the optimized target classification and grading method to obtain a target classification and grading result.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images