Data classification method, device, computer equipment and storage medium

Abstract

The present application relates to a data level classification method, device, computer equipment and storage medium. The method includes: cleaning and standardizing attribute information to obtain standard attribute information; calculating the correlation matrix of the standard attribute information; calculating the characteristic root and standardized eigenvector of the correlation matrix; calculating the cumulative contribution according to the characteristic root, and determining the target dimension according to the cumulative contribution; obtaining the target dimension load matrix according to the eigenvalue and the standardized eigenvector, calculating the quality value corresponding to the target dimension using a regression algorithm according to the target dimension load matrix, and determining the level of the source data according to the quality value of the source data. The method can effectively monitor and maintain the data quality according to the level of the data.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data level classification method, device, computer equipment and storage medium. Background Art

[0002] With the development of big data technology, the amount of data has begun to show an exponential growth trend, and data has become one of the most important assets of enterprises. In business scenarios involving big data processing, the amount of data will increase, which will lead to the continuous expansion of data storage warehouses. This situation makes a large amount of data redundant, and users cannot perceive invalid data or low-value data in time. High-value data cannot be monitored in a focused manner, so that data quality can be effectively monitored and maintained. Summary of the invention

[0003] Based on this, it is necessary to provide a data classification method, device, computer equipment and storage medium that can effectively monitor and maintain data quality in response to the above technical problems.

[0004] A data classification method, the method comprising:

[0005] Get the attribute information corresponding to the source data;

[0006] Reduce the dimension of attribute information to obtain the target dimension and the quality value corresponding to the target dimension;

[0007] Obtain target attribute information corresponding to the target dimension from the attribute information;

[0008] Calculate the weight corresponding to the target dimension according to the target attribute information;

[0009] The quality value of the source data is calculated based on the weight and quality value corresponding to the target dimension, and the level of the source data is determined based on the quality value of the source data.

[0010] In one embodiment, obtaining attribute information corresponding to source data includes:

[0011] Determine each data table in the source data, and obtain attribute information corresponding to each data table, where the attribute information is used to describe the attributes of the data table.

[0012] In one embodiment, reducing the dimension of the attribute information to obtain the target dimension and the quality value corresponding to the target dimension includes:

[0013] Clean and standardize the attribute information to obtain standard attribute information;

[0014] Calculate the correlation matrix of standard attribute information;

[0015] Compute the eigenvalues ​​and normalized eigenvectors of the correlation matrix;

[0016] Calculate the cumulative contribution based on the characteristic roots, and determine the target dimension based on the cumulative contribution;

[0017] The target dimension load matrix is ​​obtained according to the eigenvalues ​​and the standardized eigenvectors, and the mass value corresponding to the target dimension is calculated using a regression algorithm according to the target dimension load matrix.

[0018] In one embodiment, after reducing the dimension of the attribute information to obtain the target dimension and the quality value corresponding to the target dimension, the method further includes:

[0019] Determine the special outliers corresponding to the attribute information;

[0020] A level configuration instruction for a special outlier is received, and a level of source data corresponding to the special outlier is obtained according to the level configuration instruction.

[0021] In one embodiment, calculating the weight corresponding to the target dimension according to the target attribute information includes:

[0022] Normalize the target attribute information, and use the normalized target attribute information to calculate the entropy value corresponding to the target dimension;

[0023] The entropy value corresponding to the target dimension is used to calculate the difference coefficient corresponding to the target dimension, and the weight corresponding to the target dimension is calculated using the difference coefficient.

[0024] In one embodiment, determining the level of the source data according to the quality value of the source data includes:

[0025] Determine the quality value of each data table in the source data and obtain the database identifier corresponding to each data table;

[0026] Obtain the business scenario identifier corresponding to each data table according to the correspondence between the database identifier and the business scenario identifier;

[0027] The corresponding grading rules are obtained according to the business scenario identifier, and the grade of each data table in the source data is determined according to the grading rules and the quality value of each data table.

[0028] In one embodiment, after obtaining the corresponding level classification rule according to the business scenario identifier and determining the level of each data table according to the level classification rule and the quality value of each data table, the method further includes:

[0029] The identifier of each data table, the identifier of the database corresponding to each data table and the level of each data table are associated and stored in a data level database;

[0030] Receiving a data processing instruction, parsing the data processing instruction to obtain a target database identifier and a corresponding target data table identifier;

[0031] According to the target database identifier and the target data table identifier, the level of the target data table corresponding to the target data table identifier is searched in the data level database;

[0032] Execute data processing instructions in the target data table according to the level of the target data table.

[0033] A data level classification device, the device comprising:

[0034] An information acquisition module is used to obtain attribute information corresponding to source data;

[0035] Dimensionality reduction module, used to reduce the dimension of attribute information to obtain the target dimension and the quality value corresponding to the target dimension;

[0036] An information obtaining module is used to obtain target attribute information corresponding to the target dimension from the attribute information;

[0037] A weight calculation module is used to calculate the weight corresponding to the target dimension according to the target attribute information;

[0038] The grade determination module is used to calculate the quality value of the source data according to the weight and quality value corresponding to the target dimension, and determine the grade of the source data according to the quality value of the source data.

[0039] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0040] Get the attribute information corresponding to the source data;

[0041] Reduce the dimension of attribute information to obtain the target dimension and the quality value corresponding to the target dimension;

[0042] Obtain target attribute information corresponding to the target dimension from the attribute information;

[0043] Calculate the weight corresponding to the target dimension according to the target attribute information;

[0044] The quality value of the source data is calculated based on the weight and quality value corresponding to the target dimension, and the level of the source data is determined based on the quality value of the source data.

[0045] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0046] Get the attribute information corresponding to the source data;

[0047] Reduce the dimension of attribute information to obtain the target dimension and the quality value corresponding to the target dimension;

[0048] Obtain target attribute information corresponding to the target dimension from the attribute information;

[0049] Calculate the weight corresponding to the target dimension according to the target attribute information;

[0050] The quality value of the source data is calculated based on the weight and quality value corresponding to the target dimension, and the level of the source data is determined based on the quality value of the source data.

[0051] The above-mentioned data level classification method, device, computer equipment and storage medium obtain the attribute information of the source data, reduce the dimension of the attribute information to obtain the target dimension and the quality value corresponding to the target dimension, obtain the target attribute information according to the target dimension and use the target attribute information to calculate the weight of the target dimension, calculate the quality value of the source data according to the weight of the target dimension and the quality value corresponding to the target dimension, and determine the level of the source data according to the quality value of the source data. Different monitoring and maintenance methods are adopted according to the different levels of the data source, for example, high-level data is monitored and maintained and updated in real time, and low-level data is deleted and maintained, etc., so that the source data can be effectively monitored and maintained according to the level of the source data. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 An application environment diagram of a data level classification method in an embodiment;

[0053] Figure 2 A schematic diagram of a flow chart of a data level classification method in one embodiment;

[0054] Figure 3 A schematic diagram of a process for reducing the dimension of attribute information in one embodiment;

[0055] Figure 4 A schematic diagram of a process for configuring a special outlier level in one embodiment;

[0056] Figure 5 A schematic diagram of a process for calculating a target dimension weight in one embodiment;

[0057] Figure 6 A schematic diagram of a process for determining a data table level in one embodiment;

[0058] Figure 7 A schematic diagram of a data processing flow in one embodiment;

[0059] Figure 8 It is a structural block diagram of a data level classification device in one embodiment;

[0060] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] The data classification method provided in this application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The server 104 obtains the attribute information corresponding to the source data; reduces the dimension of the attribute information to obtain the target dimension and the quality value corresponding to the target dimension; obtains the target attribute information corresponding to the target dimension from the attribute information; calculates the weight corresponding to the target dimension according to the target attribute information; calculates the quality value of the source data according to the weight corresponding to the target dimension and the quality value, and the server 104 determines the level of the source data according to the quality value of the source data. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0063] In one embodiment, Figure 2 As shown, a data classification method is provided, which is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0064] S202: Acquire attribute information corresponding to source data.

[0065] The source data refers to the data to be graded. The source data can be data from a single data table or a single database. The database can have multiple data tables or multiple databases. Attribute information refers to information used to describe the attributes of the source data. The attribute information can include multiple dimensions, such as creation time, update frequency, access frequency, daily activity, monthly activity, number of scheduled data, percentage of sensitive data, whether updates are intermittent, percentage of empty data, and number of service applications.

[0066] Specifically, the server obtains the attribute information corresponding to the source data from various channels, and the data corresponding to each dimension in the attribute information is obtained from different channels. You can search from the source data, for example, the table creation time, update frequency, etc. can be directly searched from the source data. You can obtain it from the access log, for example, the access frequency, daily activity, monthly activity, number of scheduled and service applications, etc. can be searched from the access log. You can obtain it from the update log, for example, the update frequency, whether the update is intermittent, etc. can be searched from the update log. You can also obtain the data from different channels and calculate the attribute information, for example, the number of empty data obtained from the source data, and calculate the proportion of empty data. You can also obtain the number of sensitive data from the source data and calculate the proportion of sensitive data.

[0067] S204, reducing the dimension of the attribute information to obtain a target dimension and a quality value corresponding to the target dimension.

[0068] The target dimension refers to the attribute information dimension obtained after dimensionality reduction. The quality value refers to the evaluation score of the target dimension after dimensionality reduction.

[0069] Specifically, the server uses a dimensionality reduction algorithm to reduce the dimension of the attribute information, and obtains the target dimension after dimensionality reduction and the quality value corresponding to the target dimension. Among them, the dimensionality reduction algorithm can use factor analysis. Factor analysis is a multivariate statistical analysis method that starts from studying the internal dependencies of variables and reduces some variables with intricate relationships to a few comprehensive factors. Dimensionality reduction calculation can be performed using x=AF+e. Among them, x refers to the standardized attribute information to be, A is the factor loading matrix, F is the dimension to be reduced, and e is a special discrete dimension. For example, after reducing the dimension of the attribute information, five target dimensions can be obtained, namely basic properties, activity, contribution, sensitivity and stability. Among them, the basic properties are used to describe the basic properties of the source data, and the activity is the activity of the source data since its establishment. Contribution is used to describe the degree of scheduling of the source data. Sensitivity is the proportion of sensitive data in the source data stipulated by the state and enterprise information security regulations. Stability refers to the stability of the source data, which is reflected in the update interruption, the proportion of empty data, and whether the size of the data partition is 0.

[0070] S206, obtaining target attribute information corresponding to the target dimension from the attribute information.

[0071] Among them, the target attribute information is the data corresponding to the target dimension.

[0072] Specifically, when the server obtains the target dimension, it obtains the target attribute information corresponding to the target dimension from the attribute information. For example, stability is obtained based on update intervals, data null ratio, and whether the data partition size is 0. Basic properties are obtained based on creation time, update frequency, etc.

[0073] S208, calculating the weight corresponding to the target dimension according to the target attribute information.

[0074] Specifically, the server uses the entropy method to calculate the weight corresponding to the target dimension according to the target attribute information. The entropy method refers to a mathematical method used to determine the degree of dispersion of an indicator. The greater the degree of dispersion, the greater the impact of the indicator on the comprehensive evaluation. The entropy value can be used to determine the degree of dispersion of an indicator.

[0075] S210, calculating the quality value of the source data according to the weight and quality value corresponding to the target dimension, and determining the level of the source data according to the quality value of the source data.

[0076] Specifically, the server calculates the quality value by weighted average according to the weight corresponding to the target dimension and the quality value corresponding to the target dimension, and the quality value is the quality value of the source data. The server obtains the level of the source data according to the preset level classification rules according to the quality value of the source data.

[0077] In the above data level classification method, by obtaining the attribute information of the source data, the attribute information is reduced in dimension to obtain the target dimension and the quality value corresponding to the target dimension, the target attribute information is obtained according to the target dimension and the weight of the target dimension is calculated using the target attribute information, the quality value of the source data is calculated according to the weight of the target dimension and the quality value corresponding to the target dimension, the level of the source data is determined according to the quality value of the source data, and different monitoring and maintenance methods are adopted according to the different levels of the data source, for example, high-level data is monitored and maintained in real time, low-level data is deleted and maintained, etc. Therefore, the source data can be effectively monitored and maintained according to the level of the source data.

[0078] In one embodiment, step S202, i.e., obtaining attribute information corresponding to the source data, includes the steps of:

[0079] Determine each data table in the source data, and obtain attribute information corresponding to each data table, where the attribute information is used to describe the attributes of the data table.

[0080] Specifically, the server determines each data table included in the source data, and obtains the attribute information corresponding to each data table, and the attribute information is used to describe the attributes of the corresponding data table. The data tables included in the source data can be data tables in the same database or in different databases. The level of each data table can be calculated based on the attribute information of each data table. That is, when calculating the level of the source data, the level of each data table can be calculated in units of data tables, and the level of each data table can be determined, which is convenient for subsequent monitoring and maintenance of each data table.

[0081] In one embodiment, Figure 3As shown, step S204, i.e., reducing the dimension of the attribute information to obtain the target dimension and the quality value corresponding to the target dimension, includes the steps of:

[0082] S302, cleaning and standardizing the attribute information to obtain standard attribute information.

[0083] Specifically, the server cleans the attribute information, including checking data consistency and processing invalid and missing values. For example, invalid values ​​can be deleted and missing values ​​can be filled with 0. The cleaned attribute information is then standardized. For example, if the source data has m data tables and the attribute information corresponding to each data table has n dimensions, the cleaned attribute information is formed into a data set and used Use to standardize and get the standardized data set. Among them, z ij Refers to the standard attribute information of the jth dimension in the i-th data table after standardization, x ij It is the unstandardized attribute information of the jth dimension in the i-th data table. is the mean of the jth dimension. is the standard deviation of the jth dimension,

[0084] S304, calculating the correlation matrix of the standard attribute information.

[0085] The correlation matrix is ​​used to represent the linear correlation between the dimensions in the standard attribute information, which can be calculated using the sample correlation coefficient.

[0086] Specifically, the server uses the sample correlation coefficient calculation method to calculate the correlation matrix of the standard attribute information. For example, for the standardized data set, use in, The correlation matrix representing the standard attribute information. cov(y i *y j ) refers to the covariance between the i-th dimension and the j-th dimension in the standard attribute information. i ) is the variance of the i-th dimension, is the standard deviation of the j-th dimension.

[0087] S306, calculating the eigenvalues ​​and standardized eigenvectors of the correlation matrix.

[0088] Specifically, the server calculates the characteristic roots and standardized eigenvectors of the correlation matrix based on the calculated correlation matrix, for example, using For example, the characteristic roots of the correlation matrix are calculated. Where λ is the characteristic root to be calculated, and E is the unit matrix. If there are n dimensions in the attribute information, n characteristic roots will be calculated. Then according to The standardized eigenvector of the correlation matrix is ​​calculated according to . Where l is the standardized eigenvector of the correlation matrix.

[0089] S308, calculating the cumulative contribution according to the characteristic roots, and determining the target dimension according to the cumulative contribution.

[0090] Specifically, the server calculates the cumulative contribution of each dimension in the attribute information according to the characteristic root, and determines the dimension corresponding to the cumulative contribution exceeding the preset threshold of the cumulative contribution as the target dimension. The preset threshold of the cumulative contribution can be 0.8. The cumulative contribution of each dimension in the attribute information is calculated, where j refers to the jth dimension in the attribute information. It refers to calculating the sum of the characteristic roots of the first k dimensions in the attribute information. is the sum of all characteristic roots in the attribute information. Then To calculate the cumulative contribution of the kth dimension in the attribute information, start calculating from the first dimension in sequence. When k is equal to n, the cumulative contribution of each dimension in the attribute information is obtained. The dimension corresponding to the cumulative contribution exceeding the preset threshold is selected as the target dimension. For example, the target dimension can be Q.

[0091] S310, obtaining a target dimension load matrix according to the eigenvalues ​​and the standardized eigenvectors, and calculating a quality value corresponding to the target dimension using a regression algorithm according to the target dimension load matrix.

[0092] Specifically, the server calculates the target dimension load matrix using the principal component solution method based on the eigenvalues ​​and standardized eigenvectors. Where A is the target dimension loading matrix, Q refers to the number of target dimensions, and λ Q is the characteristic root of the Qth target dimension, l Q is the standardized feature vector of the Qth target dimension. Then, the quality value corresponding to the target dimension is calculated using the regression algorithm according to the target dimension load matrix. Among them, the regression algorithm refers to the use of F j =x j R -1 A' is calculated to obtain the score of the j-th target dimension, where R is the correlation matrix and A' is the matrix obtained by performing the maximum orthogonal rotation on the A target dimension loading matrix.

[0093] In the above embodiment, the attribute information is dimensioned by factor analysis to obtain the target dimension and the quality value corresponding to the target dimension, thereby improving the accuracy of the obtained target dimension and facilitating subsequent use.

[0094] In one embodiment, Figure 4 As shown, after step S202, that is, after reducing the dimension of the attribute information to obtain the target dimension and the quality value corresponding to the target dimension, the following steps are also included:

[0095] S402, determining special outliers corresponding to attribute information.

[0096] Among them, the characteristic outlier refers to the special outlier attribute information in the attribute information obtained after the dimension reduction is performed in the attribute information.

[0097] Specifically, after performing dimensionality reduction, the server obtains special outlier attribute information in the attribute information, that is, obtains special outlier attribute information in the attribute information.

[0098] S404, receiving a level configuration instruction for a special outlier, and obtaining the level of the source data corresponding to the special outlier according to the level configuration instruction.

[0099] Specifically, the server receives a level configuration instruction for the special outlier point, and according to the level configuration instruction, obtains the source data corresponding to the special outlier point, ie, the special attribute information, and obtains the level of the configuration of the source data, thereby obtaining the level of the source data corresponding to the special attribute information.

[0100] In the above embodiment, by determining the special outlier corresponding to the attribute information, receiving the level configuration instruction for the special outlier, and obtaining the level of the source data corresponding to the special outlier according to the level configuration instruction, the source data corresponding to the special attribute information can be configured with the level, that is, all the source data in the server can be configured with the corresponding level to avoid omissions and facilitate subsequent monitoring and maintenance.

[0101] In one embodiment, Figure 5 As shown, step S208, calculating the weight corresponding to the target dimension according to the target attribute information, includes the steps of:

[0102] S502, normalize the target attribute information, and use the normalized target attribute information to calculate the entropy value corresponding to the target dimension.

[0103] Specifically, the server normalizes the target attribute information obtained, and calculates the entropy value corresponding to the target dimension based on the normalized target attribute information. For example, if the source data has P data tables, and the target attribute information corresponding to each data table has Q target dimensions, you can use Perform normalized calculation, where X ij is the value of the jth target dimension in the target attribute information of the i-th data table after normalization, x ij is the value of the jth target dimension in the target attribute information of the i-th data table, that is, the unnormalized original value, max(x j ) is the maximum value of the jth target dimension in the target attribute information of P data tables, min(x j) is the minimum value of the jth target dimension in the target attribute information of P data tables. Then, according to the normalized target attribute information, the proportion of the value of the jth target dimension in the ith data table to the sum of the values ​​of the jth target dimension in the m data tables can be calculated using Calculate the proportion of the values ​​of each target dimension in all data. ij Indicates the ratio of the value of the jth target dimension in the i-th data table to the sum of the values ​​of the jth target dimension in P data tables. ij Represents the value of the j-th target dimension in the i-th data table. represents the sum of the values ​​of the jth target dimension in P data tables. Then use P ij To calculate the entropy of the target dimension, you can use Calculate the entropy value of the jth target dimension. Where k is a constant, use Calculated, Q is the number of dimensions of the target dimension.

[0104] S504, using the entropy value corresponding to the target dimension to calculate the difference coefficient corresponding to the target dimension, and using the difference coefficient to calculate the weight corresponding to the target dimension.

[0105] Specifically, the server uses the entropy value corresponding to each target dimension to calculate the difference coefficient corresponding to each target dimension, also known as information entropy redundancy. j =1-e j Calculate the difference coefficient of the jth target dimension, and then use the obtained difference coefficients to calculate the weights corresponding to each target dimension. You can use in, represents the sum of the difference coefficients of Q target dimensions. j It refers to the calculated weight of the j-th target dimension.

[0106] In a specific embodiment, when the weights w of each target dimension are calculated j When, according to the weight w of the target dimension j and the quality value F of the target dimension j use Calculate and obtain the quality value of each data table in the source data.

[0107] In the above embodiment, by normalizing the target attribute information, using the normalized target attribute information to calculate the entropy value corresponding to the target dimension, using the entropy value corresponding to the target dimension to calculate the difference coefficient corresponding to the target dimension, and using the difference coefficient to calculate the weight corresponding to the target dimension, the weight corresponding to each target dimension can be accurately calculated.

[0108] In one embodiment, Figure 6As shown, step S210, i.e. determining the level of the source data according to the quality value of the source data, comprises the steps of:

[0109] S602, determining the quality value of each data table in the source data and obtaining the database identifier corresponding to each data table.

[0110] The database identifier is used to uniquely identify the corresponding database, and each database stores data in multiple data tables.

[0111] Specifically, the server determines the quality value of each data table in the source data, and then obtains the database identifier storing each data table.

[0112] S604, obtaining the business scenario identifier corresponding to each data table according to the correspondence between the database identifier and the business scenario identifier.

[0113] The business scenario identifier is used to uniquely identify the business scenario in which the data in the database is used. Data for different business scenarios is stored in different databases, and each different database can be stored in a relational data server.

[0114] Specifically, the server obtains the business scenario identifiers corresponding to each data table in the source data according to the relationship between the databases storing the data of the pre-set business scenarios, that is, the corresponding relationship between the database identifiers and the business scenario identifiers.

[0115] S606, obtaining a corresponding grading rule according to the business scenario identifier, and determining the grade of each data table in the source data according to the grading rule and the quality value of each data table.

[0116] Specifically, the corresponding grading rules are obtained according to the business scenario identifier, and the grade of each data table in the source data is determined according to the grading rules and the quality values ​​of each data table. Different grading rules can be set for different business scenarios. For example, if there are multiple data tables in a database under a business scenario, the quality values ​​of these multiple data tables are obtained, the maximum value of the quality values ​​of these multiple data tables is obtained, the maximum value is divided into five equal parts, and the grading rules are obtained. The data tables in the database are graded according to the quality value intervals after being divided into five equal parts. In a specific application scenario, the quintile algorithm can be used to divide the quality values ​​of each data table, and the grade of each data table is determined according to the division results.

[0117] In the above embodiment, the quality value of each data table in the source data is determined and the database identifier corresponding to each data table is obtained. The business scenario identifier corresponding to each data table is obtained based on the correspondence between the database identifier and the business scenario identifier. The corresponding grade classification rule is obtained based on the business scenario identifier. The grade of each data table in the source data is determined according to the grade classification rule and the quality value of each data table. Data under different business scenarios can be graded according to different grade classification rules, thereby improving the accuracy of graded data.

[0118] In one embodiment, Figure 7 As shown, after step S606, after obtaining the corresponding level classification rule according to the business scenario identifier, and determining the level of each data table according to the level classification rule and the quality value of each data table, the following steps are also included:

[0119] S702: Associate and store the identifier of each data table, the identifier of the database corresponding to each data table, and the level of each data table in a data level database.

[0120] The data level database is a database used to store the level information of each data table. This database is independent of the business data database.

[0121] Specifically, the server stores the identifier of each data table, the identifier of the database corresponding to each data table, and the level of each data table in a data level database, which may be a hive database.

[0122] S704, receiving a data processing instruction, parsing the data processing instruction to obtain a target database identifier and a corresponding target data table identifier.

[0123] The target database identifier refers to the identifier of the database storing the data to be processed. The data is stored in a relational database. The target data table identifier refers to the identifier of the data table storing the data.

[0124] Specifically, the server receives a data processing instruction for a database, and the data processing instruction is used to maintain the data in the data table, such as deleting invalid values, filling null values, and deleting data in the data table. The server parses the data processing instruction to obtain the identifier of the database storing the data to be processed and the identifier of the data table in the database. Among them, hive database and relational data can use dataX for data processing.

[0125] S706: Search the data level database for the level of the target data table corresponding to the target data table identifier according to the target database identifier and the target data table identifier.

[0126] Specifically, the server searches for matching database identifiers and data table identifiers in the data level database according to the target database identifier and the target data table identifier, and then finds the data level associated with the database identifier and the data table identifier, that is, obtains the level of the target data table corresponding to the target data table identifier.

[0127] S708, executing the data processing instruction in the target data table according to the level of the target data table.

[0128] Specifically, the server executes data processing instructions in the target data table in the target database according to the level of the target data table. For example, if the level of the target data table is low, the data in the target data table can be deleted. If the level of the target data table is high, the data in the target data table can be updated or real-time access monitoring can be performed.

[0129] In the above embodiment, by storing the calculated level of the data table in a data level database and applying the level results in reverse to the relational database, high-value data can be monitored and low-value data can be archived based on the data level, which can effectively reduce the data storage pressure of the data storage server and reduce the waste of storage resources.

[0130] It should be understood that although Figure 2-7 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-7 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0131] In one embodiment, Figure 8 As shown, a data level classification device 800 is provided, comprising: an information acquisition module 802, a dimension reduction module 804, an information acquisition module 806, a weight calculation module 808 and a level determination module 810, wherein:

[0132] The information acquisition module 802 is used to acquire the attribute information corresponding to the source data;

[0133] A dimension reduction module 804 is used to reduce the dimension of the attribute information to obtain a target dimension and a quality value corresponding to the target dimension;

[0134] An information obtaining module 806 is used to obtain target attribute information corresponding to the target dimension from the attribute information;

[0135] A weight calculation module 808 is used to calculate the weight corresponding to the target dimension according to the target attribute information;

[0136] The level determination module 810 is used to calculate the quality value of the source data according to the weight and quality value corresponding to the target dimension, and determine the level of the source data according to the quality value of the source data.

[0137] In one embodiment, the information acquisition module 802 includes:

[0138] The data table determination module is used to determine each data table in the source data and obtain the attribute information corresponding to each data table, where the attribute information is used to describe the attributes of the data table.

[0139] In one embodiment, the dimension reduction module 804 includes:

[0140] A standardization module is used to clean and standardize the attribute information to obtain standard attribute information;

[0141] A correlation calculation module is used to calculate the correlation matrix of standard attribute information;

[0142] Feature calculation module, used to calculate the characteristic roots and standardized eigenvectors of the correlation matrix;

[0143] A target dimension determination module is used to calculate the cumulative contribution according to the characteristic roots and determine the target dimension according to the cumulative contribution;

[0144] The quality value calculation module is used to obtain the target dimension load matrix according to the eigenvalue and the standardized eigenvector, and calculate the quality value corresponding to the target dimension using a regression algorithm according to the target dimension load matrix.

[0145] In one embodiment, the data level classification device 800 further includes:

[0146] The special point determination module is used to determine the special outlier points corresponding to the attribute information.

[0147] The configuration module is used to receive a level configuration instruction for a special outlier point, and obtain the level of the source data corresponding to the special outlier point according to the level configuration instruction.

[0148] In one embodiment, the weight calculation module 808 includes:

[0149] An entropy value calculation module is used to normalize the target attribute information and use the normalized target attribute information to calculate the entropy value corresponding to the target dimension;

[0150] The coefficient calculation module is used to calculate the difference coefficient corresponding to the target dimension using the entropy value corresponding to the target dimension, and use the difference coefficient to calculate the weight corresponding to the target dimension.

[0151] In one embodiment, the level determination module 810 includes:

[0152] A database identification acquisition module is used to determine the quality value of each data table in the source data and obtain the database identification corresponding to each data table;

[0153] A scenario identification acquisition module is used to obtain the business scenario identification corresponding to each data table according to the correspondence between the database identification and the business scenario identification;

[0154] The table level determination module is used to obtain the corresponding level classification rules according to the business scenario identifier, and determine the level of each data table in the source data according to the level classification rules and the quality value of each data table.

[0155] In one embodiment, the data level classification device 800 further includes:

[0156] A level storage module is used to associate and store the identifier of each data table, the identifier of the database corresponding to each data table and the level of each data table in a data level database;

[0157] An instruction receiving module is used to receive a data processing instruction, parse the data processing instruction to obtain a target database identifier and a corresponding target data table identifier;

[0158] A level search module, used to search the level of the target data table corresponding to the target data table identifier in the data level database according to the target database identifier and the target data table identifier;

[0159] The instruction execution module is used to execute the data processing instruction in the target data table according to the level of the target data table.

[0160] For the specific definition of the data level classification device, please refer to the definition of the data level classification method above, which will not be repeated here. Each module in the above-mentioned data level classification device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0161] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store source data and level data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data level classification method is implemented.

[0162] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0163] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: obtaining attribute information corresponding to source data; reducing the dimension of the attribute information to obtain a target dimension and a quality value corresponding to the target dimension; obtaining target attribute information corresponding to the target dimension from the attribute information; calculating a weight corresponding to the target dimension based on the target attribute information; calculating a quality value of the source data based on the weight and quality value corresponding to the target dimension, and determining a level of the source data based on the quality value of the source data.

[0164] In one embodiment, obtaining attribute information corresponding to source data includes: when the processor executes the computer program, it also implements the following steps: determining each data table in the source data, and obtaining attribute information corresponding to each data table, where the attribute information is used to describe the attributes of the data table.

[0165] In one of the embodiments, reducing the dimension of attribute information to obtain a target dimension and a quality value corresponding to the target dimension includes: when the processor executes the computer program, the following steps are also implemented: cleaning and standardizing the attribute information to obtain standard attribute information; calculating the correlation matrix of the standard attribute information; calculating the eigenvalues ​​and standardized eigenvectors of the correlation matrix; calculating the cumulative contribution based on the eigenvalues, and determining the target dimension based on the cumulative contribution; obtaining the target dimension load matrix based on the eigenvalues ​​and the standardized eigenvectors, and calculating the quality value corresponding to the target dimension based on the target dimension load matrix using a regression algorithm.

[0166] In one embodiment, after reducing the dimension of the attribute information and obtaining the target dimension and the quality value corresponding to the target dimension, the processor also implements the following steps when executing the computer program: determining the special outliers corresponding to the attribute information; receiving the level configuration instructions for the special outliers, and obtaining the level of the source data corresponding to the special outliers according to the level configuration instructions.

[0167] In one embodiment, calculating the weight corresponding to the target dimension based on the target attribute information includes: when the processor executes the computer program, it also implements the following steps: normalizing the target attribute information, and using the normalized target attribute information to calculate the entropy value corresponding to the target dimension; using the entropy value corresponding to the target dimension to calculate the difference coefficient corresponding to the target dimension, and using the difference coefficient to calculate the weight corresponding to the target dimension.

[0168] In one embodiment, determining the level of source data based on the quality value of the source data includes: when the processor executes the computer program, it also implements the following steps: determining the quality value of each data table in the source data and obtaining the database identifier corresponding to each data table; obtaining the business scenario identifier corresponding to each data table based on the correspondence between the database identifier and the business scenario identifier; obtaining the corresponding level classification rule based on the business scenario identifier, and determining the level of each data table in the source data according to the level classification rule and the quality value of each data table.

[0169] In one embodiment, after obtaining the corresponding grade classification rules according to the business scenario identifier and determining the grade of each data table according to the grade classification rules and the quality value of each data table, the processor also implements the following steps when executing the computer program: storing the identifier of each data table, the identifier of the database corresponding to each data table and the grade of each data table in a data grade database; receiving data processing instructions, parsing the data processing instructions to obtain the target database identifier and the corresponding target data table identifier; searching the data grade database for the grade of the target data table corresponding to the target data table identifier according to the target database identifier and the target data table identifier; and executing the data processing instructions in the target data table according to the grade of the target data table.

[0170] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: acquiring attribute information corresponding to source data; reducing the dimension of the attribute information to obtain a target dimension and a quality value corresponding to the target dimension; obtaining target attribute information corresponding to the target dimension from the attribute information; calculating a weight corresponding to the target dimension based on the target attribute information; calculating a quality value of the source data based on the weight and quality value corresponding to the target dimension, and determining a level of the source data based on the quality value of the source data.

[0171] In one embodiment, obtaining attribute information corresponding to source data includes: when the computer program is executed by a processor, the following steps are also implemented: determining each data table in the source data, obtaining attribute information corresponding to each data table, and the attribute information is used to describe the attributes of the data table.

[0172] In one embodiment, the dimension of attribute information is reduced to obtain a target dimension and a quality value corresponding to the target dimension, including: when the computer program is executed by the processor, the following steps are also implemented: the attribute information is cleaned and standardized to obtain standard attribute information; the correlation matrix of the standard attribute information is calculated; the eigenvalues ​​and standardized eigenvectors of the correlation matrix are calculated; the cumulative contribution is calculated based on the eigenvalues ​​and the standardized eigenvectors, and the target dimension is determined based on the cumulative contribution; the target dimension load matrix is ​​obtained based on the eigenvalues ​​and the standardized eigenvectors, and the quality value corresponding to the target dimension is calculated based on the target dimension load matrix using a regression algorithm.

[0173] In one embodiment, after reducing the dimension of the attribute information and obtaining the target dimension and the quality value corresponding to the target dimension, the computer program also implements the following steps when executed by the processor: determining the special outliers corresponding to the attribute information; receiving the level configuration instructions for the special outliers, and obtaining the level of the source data corresponding to the special outliers according to the level configuration instructions.

[0174] In one embodiment, the weight corresponding to the target dimension is calculated based on the target attribute information, including: when the computer program is executed by the processor, the following steps are also implemented: normalizing the target attribute information, and using the normalized target attribute information to calculate the entropy value corresponding to the target dimension; using the entropy value corresponding to the target dimension to calculate the difference coefficient corresponding to the target dimension, and using the difference coefficient to calculate the weight corresponding to the target dimension.

[0175] In one embodiment, determining the level of source data based on the quality value of the source data includes: when the computer program is executed by the processor, it also implements the following steps: determining the quality value of each data table in the source data and obtaining the database identifier corresponding to each data table; obtaining the business scenario identifier corresponding to each data table based on the correspondence between the database identifier and the business scenario identifier; obtaining the corresponding level classification rule based on the business scenario identifier, and determining the level of each data table in the source data according to the level classification rule and the quality value of each data table.

[0176] In one embodiment, after obtaining the corresponding grade classification rules according to the business scenario identifier and determining the grade of each data table according to the grade classification rules and the quality value of each data table, the computer program also implements the following steps when executed by the processor: storing the identifier of each data table, the identifier of the database corresponding to each data table and the grade of each data table in a data grade database; receiving data processing instructions, parsing the data processing instructions to obtain the target database identifier and the corresponding target data table identifier; searching the data grade database for the grade of the target data table corresponding to the target data table identifier according to the target database identifier and the target data table identifier; and executing the data processing instructions in the target data table according to the grade of the target data table.

[0177] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0178] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0179] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A data classification method, the method comprising: Get the attribute information corresponding to the source data; Reduce the dimension of the attribute information to obtain a target dimension and a quality value corresponding to the target dimension, wherein the quality value refers to an evaluation score of the target dimension after the dimension reduction, and the quality value is obtained by performing regression calculation using a maximum orthogonal rotation matrix realized by a correlation matrix of the attribute information, the attribute information of the target dimension, and a target dimension load matrix, wherein the correlation matrix is ​​used to represent the degree of linear correlation between dimensions in the standardized attribute information, and the target dimension load matrix is ​​calculated by performing principal component decomposition based on the characteristic roots of the target dimension and the standardized characteristic vectors of the target dimension; Obtaining target attribute information corresponding to the target dimension from the attribute information; Calculate the weight corresponding to the target dimension according to the target attribute information, the weight corresponding to the target dimension is calculated using the difference coefficient of the target dimension, the difference coefficient of the target dimension is calculated using the entropy value corresponding to the target dimension, the entropy value corresponding to the target dimension is calculated using the proportion, and the proportion refers to the proportion of the value of the target dimension to the total value of the target dimension; The quality value of the source data is calculated according to the weight and quality value corresponding to the target dimension, and the level of the source data is determined according to the quality value of the source data.

2. The method according to claim 1, characterized in that The acquiring of attribute information corresponding to the source data includes: Determine each data table in the source data, and obtain attribute information corresponding to each data table, where the attribute information is used to describe the attributes of the data table.

3. The method according to claim 1, characterized in that The step of reducing the dimension of the attribute information to obtain a target dimension and a quality value corresponding to the target dimension includes: Cleaning and standardizing the attribute information to obtain standard attribute information; Calculating a correlation matrix of the standard attribute information; Calculating the eigenvalues ​​and normalized eigenvectors of the correlation matrix; Calculate the cumulative contribution according to the characteristic root, and determine the target dimension according to the cumulative contribution; A target dimension load matrix is ​​obtained according to the characteristic root and the standardized eigenvector, and a quality value corresponding to the target dimension is calculated using a regression algorithm according to the target dimension load matrix.

4. The method according to claim 1, characterized in that: After reducing the dimension of the attribute information to obtain the target dimension and the quality value corresponding to the target dimension, the method further includes: Determining special outliers corresponding to the attribute information; A level configuration instruction for the special outlier is received, and the level of the source data corresponding to the special outlier is obtained according to the level configuration instruction.

5. The method according to claim 1, characterized in that The calculating the weight corresponding to the target dimension according to the target attribute information includes: Normalizing the target attribute information, and using the normalized target attribute information to calculate the entropy value corresponding to the target dimension; The entropy value corresponding to the target dimension is used to calculate the difference coefficient corresponding to the target dimension, and the weight corresponding to the target dimension is calculated using the difference coefficient.

6. The method according to claim 1, characterized in that Determining the level of the source data according to the quality value of the source data includes: Determine the quality value of each data table in the source data and obtain the database identifier corresponding to each data table; Obtaining the business scenario identifiers corresponding to the respective data tables according to the correspondence between the database identifiers and the business scenario identifiers; A corresponding grading rule is obtained according to the business scenario identifier, and the grade of each data table in the source data is determined according to the grading rule and the quality value of each data table.

7. The method according to claim 6, characterized in that After acquiring the corresponding level classification rule according to the business scenario identifier and determining the level of each data table according to the level classification rule and the quality value of each data table, the method further includes: Associatively storing the identifiers of the respective data tables, the identifiers of the databases corresponding to the respective data tables, and the levels of the respective data tables in a data level database; Receiving a data processing instruction, parsing the data processing instruction to obtain a target database identifier and a corresponding target data table identifier; According to the target database identifier and the target data table identifier, the level of the target data table corresponding to the target data table identifier is searched in the data level database; The data processing instructions are executed in the target data table according to the level of the target data table.

8. A data level classification device, characterized in that: The device comprises: An information acquisition module is used to obtain attribute information corresponding to source data; A dimension reduction module, used for reducing the dimension of the attribute information to obtain a target dimension and a quality value corresponding to the target dimension, wherein the quality value refers to an evaluation score of the target dimension after dimension reduction, and the quality value is obtained by regression calculation using a maximum orthogonal rotation matrix realized by a correlation matrix of the attribute information, the attribute information of the target dimension, and a target dimension load matrix, wherein the correlation matrix is ​​used to represent the degree of linear correlation between dimensions in the standardized attribute information, and the target dimension load matrix is ​​calculated by principal component solution based on the characteristic roots of the target dimension and the standardized characteristic vectors of the target dimension; An information obtaining module, used for obtaining target attribute information corresponding to the target dimension from the attribute information; A weight calculation module, used to calculate the weight corresponding to the target dimension according to the target attribute information, wherein the weight corresponding to the target dimension is calculated using the difference coefficient of the target dimension, which is calculated using the entropy value corresponding to the target dimension, which is calculated using a proportion, and the proportion refers to the proportion of the value of the target dimension to the total value of the target dimension; The level determination module is used to calculate the quality value of the source data according to the weight and quality value corresponding to the target dimension, and determine the level of the source data according to the quality value of the source data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images