A prediction method and device for model primary key

Abstract

The present invention discloses a method and device for predicting a model primary key, and relates to the field of big data technology. A specific implementation of the method includes: extracting data for predicting the primary key from the model; analyzing and processing the data for predicting the primary key to obtain the component proportion of each principal component; and determining the predicted model primary key according to the component proportion of each principal component and the correspondence between each principal component and each field in the model. This specific implementation can quickly predict and locate the model primary key for new models of unknown businesses and when there are many model fields, thereby facilitating subsequent model users to analyze model granularity, clarify model businesses, etc., and thus providing support for the development and design of data models.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method and device for predicting a model primary key. Background Art

[0002] The model primary key value is used to uniquely identify a record in the model. The model primary key includes the model business primary key, which is a field with business logic meaning and is used to clarify the model granularity to facilitate downstream analysis and data model design and development.

[0003] The existing method of obtaining the business primary key of a model is mainly to consult the model developer or the model usage documents they maintain. However, when the business of the new model is not understood, and there are many types and numbers of models, it is difficult to find the developer of the corresponding model, or when the model has been handed over, the new model manager does not have a complete understanding of the model information, which creates certain obstacles to obtaining the model primary key. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method for predicting a model primary key, which can quickly predict and locate the model primary key, making it convenient for subsequent model users to analyze the model granularity, clarify the model business, etc., thereby providing support for the development and design of data models.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for predicting a model primary key is provided, comprising:

[0006] Extract the data for predicting primary keys from the model;

[0007] Analyzing and processing the data of the predicted primary key to obtain the component proportion of each principal component;

[0008] The predicted model primary key is determined according to the component proportions of the principal components and the corresponding relationships between the principal components and the fields in the model.

[0009] Optionally, extract data from the model to predict the primary key, including:

[0010] Using LOF algorithm to obtain the outlier factor of the data in the model;

[0011] According to the outlier factor, data of the predicted primary key is determined.

[0012] Optionally, the outlier factor of the data in the model is obtained using a LOF algorithm, including:

[0013] Standardizing all data in the model to obtain standardized data;

[0014] Determine the data mean of each field in the model according to the standardized data;

[0015] Determine an outlier candidate set according to the data mean and the standardized data, wherein the outlier candidate set includes at least one candidate outlier;

[0016] The LOF algorithm is used to calculate the outlier factor of each candidate outlier in the outlier candidate set.

[0017] Optionally, performing standardization processing on all data in the model to obtain standardized data includes:

[0018] Convert all data in the model into numerical data;

[0019] Determine the column mean and column standard deviation of the column where the numerical data is located;

[0020] The numerical data is subtracted from the column mean, and the subtraction result is divided by the column standard deviation to obtain the standardized data.

[0021] Optionally, determining a candidate set of outliers according to the data mean and the standardized data includes:

[0022] Calculate the distance between the data mean and each data in the field and the distance mean of the field;

[0023] Data with distances greater than the mean distance are selected as candidate outliers to form the outlier candidate set.

[0024] Optionally, the outlier factor of each candidate outlier in the outlier candidate set is calculated using a LOF algorithm, including:

[0025] For each candidate outlier in the outlier candidate set,

[0026] Calculate the local reachable density of the k-th distance field of the candidate outlier and the local reachable density of the candidate outlier, where k is a given positive integer;

[0027] The average of the ratios of the local reachable density of the k-th distance area of ​​the candidate outlier to the local reachable density of the candidate outlier is determined as the outlier factor of the candidate outlier.

[0028] Optionally, determining the data of the predicted primary key according to the outlier factor includes:

[0029] Sort the outlier factors in descending order, and determine the rows where the data corresponding to the first n outlier factors are located, where n is a preset positive integer;

[0030] The row data of the row where the first n data are located is removed from the model to determine the data of the predicted primary key.

[0031] Optionally, analyzing and processing the data of the predicted primary key to obtain the component proportion of each principal component includes:

[0032] The principal component analysis method is used to analyze and process the data of the predicted primary key to obtain the component proportion of each principal component, and the component proportion is the variance contribution rate.

[0033] Optionally, principal component analysis is used to analyze and process the data of the predicted primary key to obtain the variance contribution rate of each principal component, including:

[0034] Obtaining a covariance matrix of the predicted primary key data;

[0035] Obtaining eigenvalues ​​of the covariance matrix;

[0036] According to the eigenvalues, the variance contribution rate of each principal component is determined.

[0037] Optionally, determining the predicted model primary key according to the contribution rate of each principal component and the corresponding relationship between each principal component and each field includes:

[0038] Determine a factor loading matrix of the covariance matrix according to the eigenvalues ​​and eigenvectors of the covariance matrix;

[0039] According to the factor loading matrix, determining the corresponding relationship between each principal component and each field;

[0040] Each field is accumulated one by one in descending order of the variance contribution rate to determine whether the result corresponding to the field is empty, so as to perform a primary key check on the field. If so, the check passes and it is determined that the field is the model primary key. If not, it is determined whether the number of determined fields is equal to the total number of fields. If so, it is determined that the model has no primary key. If not, it is determined whether the result corresponding to the field where the cumulative number is added to 1 is empty.

[0041] According to another aspect of an embodiment of the present invention, a model primary key prediction device is provided, comprising:

[0042] Data processing module, extracts data for predicting primary keys from the model;

[0043] A data analysis module analyzes and processes the data of the predicted primary key to obtain the component proportion of each principal component;

[0044] The primary key prediction module determines the predicted model primary key according to the component proportions of the principal components and the corresponding relationship between the principal components and each field in the model.

[0045] According to another aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0046] one or more processors;

[0047] a storage device for storing one or more programs,

[0048] When the one or more programs are executed by the one or more processors, the one or more processors implement the model primary key prediction method provided by the present invention.

[0049] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method for predicting a model primary key provided by the present invention is implemented.

[0050] An embodiment of the above invention has the following advantages or beneficial effects: by extracting the data of the predicted primary key from the model and analyzing and processing the data of the predicted primary key, the component weight of each principal component is obtained, and then the predicted model primary key is determined according to the correspondence between each principal component and each field in the model. The embodiment of the present invention combines the extraction of the data of the predicted primary key with the acquisition of the component weight of each principal component to achieve rapid prediction and positioning of the model primary key, thereby facilitating subsequent model users to analyze the model granularity, clarify the model business, etc., and further providing support for the development and design of data models.

[0051] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0053] Figure 1 is a schematic diagram of the main process of a method for predicting a model primary key according to an embodiment of the present invention;

[0054] Figure 2 is a schematic diagram of the main process of another method for predicting a model primary key according to an embodiment of the present invention;

[0055] Figure 3 is a schematic diagram of the main process of another method for predicting a model primary key according to an embodiment of the present invention;

[0056] Figure 4 It is a flowchart of a method for predicting a model primary key according to an embodiment of the present invention;

[0057] Figure 5 is a schematic diagram of main modules of a device for predicting a model primary key according to an embodiment of the present invention;

[0058] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0059] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0061] With the development of Internet technology, there are more and more data warehouse models, such as buffer models, basic models, general models, aggregation models, dimensional models and application models. The way to obtain the business primary key of the model is generally to consult the R&D personnel or the model usage documents they maintain. However, when the business of the new model is not understood and there are many model fields, it is difficult to predict which fields its primary key is; and it is difficult to find the R&D personnel of the corresponding model, especially when the corresponding model R&D is not maintained or the maintenance personnel are not model developers. For example, when extracting source data to the big data platform, the person in charge of the corresponding buffer model and basic model is the extractor, not the model R&D personnel, etc., or the model has been handed over, and the person in charge of the new model does not fully understand the model information, etc., which all cause certain obstacles to obtaining the model primary key and analyzing the model granularity. Therefore, in response to the above problems, an embodiment of the present invention provides a method for predicting the model primary key to quickly predict the model business primary key.

[0062] Figure 1 is a schematic diagram of the main process of a method for predicting a model primary key according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0063] Step S101: extracting data of predicted primary key from the model;

[0064] Step S102: Analyze and process the data of the predicted primary key to obtain the component proportion of each principal component;

[0065] Step S103: Determine the predicted model primary key according to the component weight of each principal component and the corresponding relationship between each principal component and each field in the model.

[0066] The model primary key prediction method of the embodiment of the present invention is a model business primary key method. The model business primary key is a field in the model with business logic meaning. The combination of two or more fields as the model primary key is called a joint primary key.

[0067] In the embodiment of the present invention, in step S101, the data for predicting the primary key is the data in the model used to predict the business primary key of the model. The data for predicting the primary key can be all the data in the model or the data after the abnormal data is removed from the model. Optionally, the abnormal data in the model can be removed by the LOF algorithm, the Grubbs criterion, etc. to obtain the data for predicting the primary key.

[0068] Optionally, the model can be a data table. For example, given a model, the data volume is t, that is, the total number of rows in the model is t, the total number of fields is m, that is, the number of rows in the data table is t, the number of columns is m, t≥1, m≥2.

[0069] In the embodiment of the present invention, in step S101, extracting data for predicting the primary key from the model includes: using the LOF algorithm to obtain the outlier factor of the data in the model; and determining the data for predicting the primary key according to the outlier factor.

[0070] Using the LOF algorithm to remove outliers in the model can improve the accuracy of the model's primary key prediction. The density-based LOF algorithm can effectively detect local outliers in the model with high detection accuracy.

[0071] Optionally, the data in the model can be each data in the model, that is, the LOF algorithm (Local Outlier Factor) is used to obtain the outlier factor (LOF value) of each data in the model, and then the outlier data in the model is eliminated according to the outlier factor to determine the data of the predicted primary key. Optionally, for each data in the model, the local reachable density of the k-th distance field of each data and the local reachable density of each data are calculated, where k is a given positive integer; the average of the ratio of the local reachable density of the k-th distance field of each data to the local reachable density of each data is determined as the outlier factor of the candidate outlier.

[0072] When traversing the entire model data to calculate the LOF value of each data point, since the number of outliers only accounts for a very small part of the total model data, calculating the LOF value of each data point to determine the degree of discreteness of each data point will generate a large amount of redundant calculations, resulting in low algorithm efficiency and a waste of space resources due to storing intermediate results. Therefore, before calculating the outlier factor of each data point, the data in the model is pruned to determine the candidate set of outliers.

[0073] In the embodiment of the present invention, Figure 2As shown, the LOF algorithm used to obtain the outlier factor of the data in the model includes:

[0074] Step S201: Standardize all data in the model to obtain standardized data;

[0075] Step S202: Determine the data mean of each field in the model based on the standardized data;

[0076] Step S203: determining an outlier candidate set according to the data mean and the normalized data, wherein the outlier candidate set includes at least one candidate outlier;

[0077] Step S204: Calculate the outlier factor of each candidate outlier in the outlier candidate set using the LOF algorithm.

[0078] In an embodiment of the present invention, before pruning the data in the model, all the data in the model are standardized to obtain the standardized data, including: converting all the data in the model into numerical data; determining the column mean and column standard deviation of the column where the numerical data is located; subtracting the numerical data from the column mean, dividing the difference result by the column standard deviation, and obtaining the standardized data.

[0079] Before converting all the data in the model into numeric data, it includes: judging whether there are null values ​​in each field (feature) in the model. If not, no processing is performed; if so, the null values ​​are processed to fill the null values; optionally, different null value processing methods are used for different types of fields. When the field corresponding to the null value is a character field, the string "None" can be used to fill the null value; when the field corresponding to the null value is a numeric field, the column mean of the field can be used to fill the null value.

[0080] When the field is a character field, since character data cannot be used for calculation, it is necessary to convert the character data into numeric data. For example, classification coding can be used to convert character data into numeric data. Classification coding can be encoded by object, such as ["male", "female"] can be represented by [0, 1].

[0081] After converting the data in the model into numerical data, the data is processed to eliminate the unit or magnitude influence of each field in the model. For example, for a model with a total number of rows of t and a total number of fields of m, the standardized data is obtained according to formula (1):

[0082]

[0083] In formula (1), l is the row where the numerical data is located, q is the column where the numerical data is located, is the column mean of the qth column, is the standard deviation of the qth column. is the numeric data in the lth row and qth column, The numerical data in the lth row and qth column are normalized.

[0084] In an embodiment of the present invention, after obtaining the standardized data, pruning is performed to determine the data mean of each field in the model, and then the outlier candidate set is determined based on the data mean and the standardized data, further including: calculating the distance between the data mean and each data in the field and the mean of the distance; selecting data with a distance greater than the mean of the distance as a candidate outlier to form an outlier candidate set. Among them, each data in the field is standardized data, and the data mean of the field is the average value of all standardized data in the field.

[0085] For example, the data mean of the i-th field is recorded as a(i), i is the number of columns, i≤m, and then the distance from each data in the i-th field to the data mean a(i) (that is, the absolute value of the difference between each data in the i-th field and a(i)) and the mean of the distance r(i) are calculated respectively, and then the data with a distance greater than the mean of the distance r(i) are selected as candidate outliers to form an outlier candidate set, which is recorded as set S.

[0086] In an embodiment of the present invention, after obtaining the outlier candidate set, the LOF algorithm is used to calculate the outlier factor of each candidate outlier in the outlier candidate set, including: for each candidate outlier in the outlier candidate set, calculating the local reachable density of the k-th distance field of the candidate outlier and the local reachable density of the candidate outlier, where k is a given positive integer; determining the average of the ratio of the local reachable density of the k-th distance field of the candidate outlier to the local reachable density of the candidate outlier as the outlier factor of the candidate outlier.

[0087] Optionally, the outlier factor of each candidate outlier point p (p∈S) in the outlier candidate set is calculated using formula (2):

[0088]

[0089] In formula (2), k is a given positive integer, and the kth distance of the definition point p is d k (p), in the set S, there is a point o, the distance between it and point p is denoted as d(p,o), when the following two conditions are met, d k (p) = d(p,o): (1) In the set S, there are at least k points o′ (o′∈S, o′≠p) such that d(p,o′)≤d(p,o); (2) In the set S, there are at most k-1 points o′ (o′∈S, o′≠p) such that d(p,o′)<d(p,o).

[0090] Among them, N k (p) is the kth distance range of point p, which means the distance from point p is less than or equal to d k (p) is a set of points, so the number of points in the kth region of point p | N k (p)|≥k; lrd k (o) is the kth distance field N of point p k (p) Local reachability density of midpoint o, lrd k (p) is the local reachable density of point p, which is calculated according to formula (3).

[0091]

[0092] In formula (3), rech_dist k (p,o) is the reachable distance from point o to point p, which can be calculated according to formula (4):

[0093] rech_dist k (p,o)=max{d k (o),d(p,o)} (4).

[0094] The embodiment of the present invention determines an outlier candidate set and then calculates the outlier factor of each candidate outlier in the outlier candidate set, thereby achieving pruning of data in the model. There is no need to calculate the outlier factor of each data in the model, thereby improving the operational efficiency of the LOF algorithm in detecting outliers and saving storage space.

[0095] In one implementation of an embodiment of the present invention, the data of the predicted primary key is determined based on the outlier factors, including: sorting the outlier factors in descending order, determining the rows where the data corresponding to the first n outlier factors are located, where n is a preset positive integer; and removing the row data of the rows where the first n data are located from the model to determine the data of the predicted primary key.

[0096] After obtaining the outlier factors of each candidate outlier in the outlier candidate set, it is necessary to determine the outliers to determine the data for predicting the primary key. Because the greater the LOF value is than 1, the more it is considered an outlier. Therefore, the candidate outliers can be sorted in descending order according to the size of the outlier factor, and the row data of the first n candidate outliers are removed, and the remaining data in the model is used as the data for predicting the primary key. The first n candidate outliers can be used as outliers of the model data.

[0097] In another embodiment, according to the outlier factor, determining the data for predicting the primary key includes: sorting the outlier factors in descending order, determining the candidate outliers whose outlier factors are greater than a preset threshold, and then removing the row data where the candidate outliers are located, and using the remaining data in the model as the data for predicting the primary key. The candidate outliers whose outlier factors are greater than the preset threshold can be used as outliers of the model data.

[0098] In an embodiment of the present invention, after determining the data of the predicted primary key, the data of the predicted primary key is analyzed and processed to obtain the component proportion of each principal component. Optionally, the data of the predicted primary key can be analyzed and processed using methods such as principal component analysis, cluster analysis or factor analysis.

[0099] In an optional implementation, the data of the predicted primary key is analyzed and processed to obtain the component proportion of each principal component, including: using the principal component analysis method to analyze and process the data of the predicted primary key to obtain the component proportion of each principal component, where the component proportion is the variance contribution rate.

[0100] Furthermore, principal component analysis (PCA) is used to analyze and process the data of the predicted primary key to obtain the variance contribution rate of each principal component, including: obtaining the covariance matrix of the data of the predicted primary key; obtaining the eigenvalue of the covariance matrix; and determining the variance contribution rate of each principal component based on the eigenvalue.

[0101] The covariance matrix R of the data predicting the primary key y Calculated by formula (5),

[0102]

[0103] In formula (5), Y i =(y i1 ,y i2 ,…,y im ) is the row vector of the data for predicting the primary key, tn is the amount of data for predicting the primary key, is the mean of the j-th column of data for predicting the primary key, Calculated by formula (6),

[0104]

[0105] In formula (6), y ij The data is in the i-th row and j-th column.

[0106] According to the obtained covariance matrix R y , calculate the covariance matrix R y The eigenvalue λ j (j=1,2,…,m) and the corresponding eigenvector vj (j=1,2,…,m), the eigenvalues ​​and eigenvectors can be obtained by solving the characteristic equation |λI-R y |=0, where I is a unit vector; then the eigenvalue λ j The sizes are sorted in descending order, so that λ1≥λ2≥...≥λ j ≥λ m ≥0, and then calculate the variance contribution rate and cumulative variance contribution rate of each principal component according to formula (7) and formula (8) to obtain the proportion of each principal component.

[0107]

[0108]

[0109] In formula (7), is the variance contribution rate of the bth principal component, b=1,2,…,m, in formula (8), It is the cumulative variance contribution rate of the first b principal components. The variance contribution rate of a principal component reflects how much information the principal component contains. According to the order of the eigenvalues, λ1 is the largest, indicating that the variance contribution rate of the first principal component is the largest, followed by the second principal component, and so on.

[0110] The principal components obtained after PCA analysis are the comprehensive performance of the fields in the model and have no actual business meaning. Therefore, the principal components obtained cannot be used as the primary key of the model. Since the factor loading matrix in PCA or the factor loading matrix after orthogonal rotation can reflect the relationship between the fields in the model and each principal component, the prediction of the model primary key can be completed by selecting appropriate fields with business meaning by calculating the factor loading matrix or the factor loading matrix after orthogonal rotation.

[0111] In the embodiment of the present invention, Figure 3 As shown in the figure, according to the contribution rate of each principal component and the corresponding relationship between each principal component and each field, the predicted model primary key is determined, including:

[0112] Step S301: determining the factor loading matrix of the covariance matrix according to the eigenvalues ​​and eigenvectors of the covariance matrix;

[0113] Step S302: Determine the corresponding relationship between each principal component and each field according to the factor loading matrix;

[0114] Step S303: accumulating the results of each field one by one in descending order of variance contribution rate to determine whether the result corresponding to the field is empty, so as to perform primary key verification on the field. If yes, execute step S304; if not, execute step S305;

[0115] Step S304: Verification passes, and the field is determined to be the model primary key;

[0116] Step S305: determine whether the determined number of fields is equal to the total number of fields, if so, execute step S306, if not, execute step S303;

[0117] Step S306: Determine whether the model has no primary key.

[0118] According to the covariance matrix R y The eigenvalue λ j (j=1,2,…,m) and the eigenvector v j (j=1,2,…,m), and get the covariance matrix R y The factor loading matrix A is shown in formula (9).

[0119]

[0120] According to the factor load matrix, the fields corresponding to each principal component can be determined. Then, the first j features (fields) are selected one by one according to the descending order of the eigenvalues ​​(or variance contribution rates) of each principal component to verify the model primary key. The value of the variable j is initialized to 1. It is determined whether the query results corresponding to the first j fields are empty, that is, whether there are duplicate records in the first j fields. If the query result is empty, it means that there are no duplicate records, then the verification is passed, and the first j fields are determined to be the model primary key, and the model primary key is output; if the query result corresponding to the first j fields is not empty, it means that there are duplicate records, then it is determined whether j is equal to the total number of fields m, if yes, it is determined that the model has no primary key, if not, it is determined whether the query results of the first j+1 fields are empty, and the above steps are repeated until the model primary key or the model has no primary key is obtained. When two or more fields are determined to be the model primary key, the model primary key is a joint primary key. The SQL statement for checking the model primary key for the first j fields can be: Select field1, field2, ..., fieldj From table Group by field1, field2, ..., fieldjhaving count(*)>1.

[0121] For example, the model has three fields, y1, y2, and y3. Assume that the covariance matrix R is calculated according to formula (5): y for:

[0122]

[0123] Solve the characteristic equation |λI-R|=0 to obtain the covariance matrix R y The eigenvalues ​​of are arranged from large to small as follows: λ1=1.5464,λ2=0.8536,λ3=0.6, and the corresponding eigenvectors are:

[0124]

[0125] According to the eigenvalues ​​and eigenvectors, we get the factor loading matrix, that is, the principal component contribution rate matrix A:

[0126]

[0127] Get the relationship between the model fields and the principal components:

[0128] y1=0.5717F1+0.8205F2

[0129] y2=0.7809F1-0.3003F2+0.5477F3,

[0130] y3=-0.7809F1+0.3003F2+0.5477F3

[0131] Among them, F1, F2 and F3 are the extracted common factors;

[0132] From this we can deduce:

[0133] F1=0.5717y1+0.7809y2-0.7809y3

[0134] F2=0.8205y1-0.3003y2+0.3003y3,

[0135] F3=0.5477y2+0.5477y3

[0136] The corresponding relationship between the three fields and the three principal components in the model can be obtained as shown in Table 1.

[0137] Table 1

[0138]

[0139] From the above principal component contribution rate matrix, we can select the selectable field y2 as the first principal component, field y1 as the second principal component, and field y3 as the third principal component (the second field has been selected, so we can select the third field that contributes more to the third principal component), that is, the first principal component, the second principal component, and the third principal component correspond to fields y2, y1, and y3 respectively. Then, the model primary key is verified in the order of y2, y1, and y3 to determine whether the query result of field y2 is empty. If so, the model primary key is determined to be field y2. Otherwise, determine whether the query results of y2 and y1 are empty. If so, determine that the model primary key is y2 and y1. Otherwise, determine whether the query results of y2, y1, and y3 are empty. If so, determine that the model primary key is y2, y1, and y3. Otherwise, determine whether the query results of y2, y1, and y3 are empty. If so, determine that the model primary key is y2, y1, and y3. Otherwise, determine that the model has no primary key.

[0140] like Figure 4The figure is a flow chart of a method for predicting a model primary key according to an embodiment of the present invention. The total number of rows in the model is t, the total number of fields is m, and the variables i=1, j=1 (i, j≤m) are initialized.

[0141] Step S401: standardize the data in the model to obtain standardized data;

[0142] Step S402: Calculate the data mean a(i) of all data in the i-th field, calculate the distance between all data in the i-th field and a(i) and the mean r(i) of the distance;

[0143] Step S403: taking the data in the i-th field whose distance to a(i) is greater than r(i) as candidate outliers to form an outlier candidate set;

[0144] Step S404: determine whether i is equal to the total number of fields m, if not, execute step S402, if yes, execute step S405;

[0145] Step S405: using the LOF algorithm to calculate the outlier factor of each candidate outlier in the outlier candidate set, and sorting the outlier factors in descending order;

[0146] Step S406: Eliminate the row data of the row where the first n candidate outliers are located, and determine the data of the predicted primary key;

[0147] Step S407: using PCA to perform principal component analysis on each field, and calculating the variance contribution rate and cumulative variance contribution rate corresponding to each principal component;

[0148] Step S408: Determine the fields corresponding to each principal component through the factor loading matrix;

[0149] Step S409: Determine whether the query results of the first j fields are empty to perform model primary key verification. If not, execute step S410. If yes, execute step S411.

[0150] Step S410: Determine whether j is equal to the total number of fields m, if so, execute step S412, if not, set j=j+1, and execute step S409;

[0151] Step S411: determine the first j fields as the model primary key, and output the model primary key;

[0152] Step S412: Determine whether the model has no primary key.

[0153] The model primary key prediction method provided by the embodiment of the present invention extracts the predicted primary key data in the model, analyzes and processes the predicted primary key data, determines the component proportion of each principal component, and then predicts the model primary key according to the correspondence between each principal component and the field. For example, by combining the LOF algorithm with the PCA principal component analysis method, the prediction and positioning of the model primary key can be quickly achieved. And by processing the data in the model, an outlier candidate set is obtained, and then the outlier factors of the candidate outliers in the outlier candidate set are calculated to determine the data of the predicted primary key, thereby improving the operation efficiency and the accuracy of the predicted model primary key. The model primary key prediction method of the present invention can quickly and accurately predict the model primary key when facing a new model or when there are many model fields, thereby providing support for model users to analyze model granularity and clarify model business needs.

[0154] like Figure 5 As shown, an embodiment of the present invention provides a model primary key prediction device 500, which includes:

[0155] Data processing module 501, extracting data of predicted primary key from the model;

[0156] The data analysis module 502 analyzes and processes the data of the predicted primary key to obtain the component proportion of each principal component;

[0157] The primary key prediction module 503 determines the predicted model primary key according to the component proportion of each principal component and the corresponding relationship between each principal component and each field in the model.

[0158] In the embodiment of the present invention, the data processing module 501 is further used to: use the LOF algorithm to obtain the outlier factor of the data in the model; and determine the data of the predicted primary key according to the outlier factor.

[0159] In an embodiment of the present invention, the data processing module 501 is further used to: standardize all data in the model to obtain standardized data; determine the data mean of each field in the model based on the standardized data; determine the outlier candidate set based on the data mean and the standardized data, the outlier candidate set includes at least one candidate outlier; use the LOF algorithm to calculate the outlier factor of each candidate outlier in the outlier candidate set.

[0160] In an embodiment of the present invention, the data processing module 501 is further used to: convert all data in the model into numerical data; determine the column mean and column standard deviation of the column where the numerical data is located; subtract the numerical data from the column mean, and divide the difference result by the column standard deviation to obtain standardized data.

[0161] In an embodiment of the present invention, the data processing module 501 is further used to: determine a candidate set of outliers based on the data mean and the data after normalization, including: calculating the distance between the data mean and each data in the field and the mean distance of the field; selecting data whose distance is greater than the mean distance as candidate outliers to form a candidate set of outliers.

[0162] In an embodiment of the present invention, the data processing module 501 is further used to: for each candidate outlier in the outlier candidate set, calculate the local reachable density of the k-th distance field of the candidate outlier and the local reachable density of the candidate outlier, where k is a given positive integer; determine the average of the ratio of the local reachable density of the k-th distance field of the candidate outlier to the local reachable density of the candidate outlier as the outlier factor of the candidate outlier.

[0163] In an embodiment of the present invention, the data processing module 501 is further used to: determine the data of the predicted primary key based on the outlier factors, including: sorting the outlier factors in descending order, determining the rows where the data corresponding to the first n outlier factors are located, where n is a preset positive integer; and removing the row data of the rows where the first n data are located from the model to determine the data of the predicted primary key.

[0164] In the embodiment of the present invention, the data analysis module 502 is further used to: use the principal component analysis method to analyze and process the data of the predicted primary key to obtain the component proportion of each principal component, where the component proportion is the variance contribution rate.

[0165] In the embodiment of the present invention, the data analysis module 502 is further used to: obtain a covariance matrix of the predicted primary key data; obtain eigenvalues ​​of the covariance matrix; and determine the variance contribution rate of each principal component according to the eigenvalues.

[0166] In an embodiment of the present invention, the primary key prediction module 503 is further used to: determine the factor loading matrix of the covariance matrix according to the eigenvalues ​​and eigenvectors of the covariance matrix; determine the correspondence between each principal component and each field according to the factor loading matrix; and determine whether the results corresponding to the fields are empty one by one in descending order of variance contribution rate to perform a primary key check on the fields. If so, the check passes and the field is determined to be the model primary key. If not, determine whether the number of determined fields is equal to the total number of fields. If so, determine that the model has no primary key. If not, determine whether the result corresponding to the field with the cumulative number plus 1 is empty.

[0167] An embodiment of the present invention also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the one or more processors implement the model primary key prediction method of the embodiment of the present invention.

[0168] An embodiment of the present invention further provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, the method for predicting the model primary key of the embodiment of the present invention is implemented.

[0169] Figure 6 An exemplary system architecture 600 is shown to which the method for predicting a model primary key or the device for predicting a model primary key according to an embodiment of the present invention can be applied.

[0170] like Figure 6 As shown, the system architecture 600 may include terminal devices 601, 602, 603, a network 604 and a server 605 (this architecture is only an example, and the components included in the specific architecture may be adjusted according to the specific application). The network 604 is used to provide a medium for a communication link between the terminal devices 601, 602, 603 and the server 605. The network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0171] Users can use terminal devices 601, 602, and 603 to interact with server 605 through network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0172] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0173] The server 605 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 601, 602, and 603. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0174] It should be noted that the model primary key prediction method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the model primary key prediction device is generally set in the server 605 .

[0175] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0176] Reference below Figure 7, which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0177] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0178] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0179] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0180] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0181] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0182] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a data processing module, a data analysis module and a primary key prediction module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the data processing module may also be described as a "module for extracting data for predicting primary keys from a model".

[0183] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: extracting data for predicting primary keys from the model; analyzing and processing the data for predicting primary keys to obtain the component weights of each principal component; and determining the predicted model primary key according to the component weights of each principal component and the corresponding relationship between each principal component and each field in the model.

[0184] According to the technical solution of the embodiment of the present invention, by extracting the predicted primary key data in the model and analyzing and processing the predicted primary key data, the component proportion of each principal component is determined, and then the model primary key is predicted according to the corresponding relationship between each principal component and the field. For example, by combining the LOF algorithm with the PCA principal component analysis method, the prediction and positioning of the model primary key can be quickly realized. And by processing the data in the model, the outlier candidate set is obtained, and then the outlier factor of the candidate outlier in the outlier candidate set is calculated to determine the data of the predicted primary key, thereby improving the operation efficiency and the accuracy of the predicted model primary key. The model primary key prediction method of the present invention can quickly and accurately predict the model primary key when facing a new model or when there are many model fields, thereby providing support for model users to analyze model granularity and clarify model business needs.

[0185] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for predicting a model primary key, characterized in that: include: Extract the data for predicting primary keys from the model; The principal component analysis method is used to analyze and process the data of the predicted primary key to obtain the component proportion of each principal component; the component proportion is the variance contribution rate; Determine a predicted model primary key according to the component weights of the principal components and the corresponding relationships between the principal components and the fields in the model; The principal component analysis method is used to analyze and process the data of the predicted primary key to obtain the variance contribution rate of each principal component, including: obtaining the covariance matrix of the predicted primary key data; obtaining the eigenvalue of the covariance matrix; and determining the variance contribution rate of each principal component according to the eigenvalue; According to the component proportions of the principal components and the corresponding relationship between the principal components and the fields in the model, the predicted model primary key is determined, including: determining the factor loading matrix of the covariance matrix according to the eigenvalues ​​and eigenvectors of the covariance matrix; determining the corresponding relationship between the principal components and the fields according to the factor loading matrix; accumulating each field one by one in descending order of the variance contribution rate to determine whether the result corresponding to the field is empty, so as to perform a primary key check on the field. If so, the check passes and the field is determined to be the model primary key. If not, it is determined whether the number of determined fields is equal to the total number of fields. If so, it is determined that the model has no primary key. If not, it is determined whether the result corresponding to the field increased by 1 is empty.

2. The method according to claim 1, characterized in that Extract data from the model to predict the primary key, including: Using LOF algorithm to obtain the outlier factor of the data in the model; According to the outlier factor, data of the predicted primary key is determined.

3. The method according to claim 2, characterized in that The outlier factor of the data in the model is obtained using the LOF algorithm, including: Standardizing all data in the model to obtain standardized data; Determine the data mean of each field in the model according to the standardized data; Determine an outlier candidate set according to the data mean and the standardized data, wherein the outlier candidate set includes at least one candidate outlier; The LOF algorithm is used to calculate the outlier factor of each candidate outlier in the outlier candidate set.

4. The method according to claim 3, characterized in that All data in the model are standardized to obtain standardized data, including: Convert all data in the model into numerical data; Determine the column mean and column standard deviation of the column where the numerical data is located; The numerical data is subtracted from the column mean, and the subtraction result is divided by the column standard deviation to obtain the standardized data.

5. The method according to claim 3, characterized in that: Determining an outlier candidate set according to the data mean and the standardized data includes: Calculate the distance between the data mean and each data in the field and the distance mean of the field; Data with distances greater than the mean distance are selected as candidate outliers to form the outlier candidate set.

6. The method according to claim 3, characterized in that The outlier factor of each candidate outlier in the outlier candidate set is calculated using the LOF algorithm, including: For each candidate outlier in the outlier candidate set, Calculate the local reachable density of the k-th distance field of the candidate outlier and the local reachable density of the candidate outlier, where k is a given positive integer; The average of the ratios of the local reachable density of the k-th distance area of ​​the candidate outlier to the local reachable density of the candidate outlier is determined as the outlier factor of the candidate outlier.

7. The method according to any one of claims 2 to 6, characterized in that: Determining the data of the predicted primary key according to the outlier factor includes: Sort the outlier factors in descending order, and determine the rows where the data corresponding to the first n outlier factors are located, where n is a preset positive integer; The row data of the row where the first n data are located is removed from the model to determine the data of the predicted primary key.

8. A prediction device for a model primary key, characterized in that: include: Data processing module, extracts data for predicting primary keys from the model; The data analysis module uses the principal component analysis method to analyze and process the data of the predicted primary key to obtain the component proportion of each principal component; the component proportion is the variance contribution rate; A primary key prediction module, which determines a predicted model primary key according to the component weights of the principal components and the corresponding relationships between the principal components and the fields in the model; Wherein, the data analysis module is further used to: obtain the covariance matrix of the predicted primary key data; obtain the eigenvalue of the covariance matrix; and determine the variance contribution rate of each principal component according to the eigenvalue; The primary key prediction module is further used to: determine the factor loading matrix of the covariance matrix according to the eigenvalues ​​and eigenvectors of the covariance matrix; determine the corresponding relationship between each principal component and each field according to the factor loading matrix; accumulate each field one by one in descending order of the variance contribution rate to determine whether the result corresponding to the field is empty, so as to perform a primary key check on the field. If so, the check passes and the field is determined to be the model primary key. If not, it is determined whether the number of determined fields is equal to the total number of fields. If so, it is determined that the model has no primary key. If not, it is determined whether the result corresponding to the field with the accumulated number plus 1 is empty.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images