Data processing method, device, equipment and storage medium

Abstract

The embodiment of the present application discloses a data processing method, apparatus, device and storage medium, the method comprising: obtaining field information of multiple tables in a database; determining the association information between any two tables based on the field information of the multiple tables; constructing a relationship network of the multiple tables based on the association information between the any two tables; calculating the PageRank value of each table based on the relationship network; wherein the PageRank value is used to characterize the association value of each table in the relationship network; and performing data mining on the tables in the database based on the PageRank value of each table. In this way, by extracting the table field information in the database to construct a table relationship network and obtaining the PageRank value of each table, the association information between tables and the association value of each table can be obtained, thereby simplifying subsequent data mining. The entire process does not require additional business logic or development logic support, breaks down the barriers between various industries and data platforms, and improves the degree of intelligent automation of data analysis.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a data processing method, apparatus, device and storage medium. Background Art

[0002] In existing technologies, data analysis in databases is largely still done manually, requiring analysis and processing based on business scenarios and requirements. This requires understanding the specific business context and analyzing the data in the database to generate results. However, as data volumes continue to grow, the difficulty of data mining and analysis increases. Furthermore, existing microservices architectures reduce interaction between database developers and data analysts, making it difficult for data analysts to obtain the information used to construct database tables and fields. All of these factors complicate traditional manual analysis methods. Summary of the Invention

[0003] To solve the above technical problems, the embodiments of the present application hope to provide a data processing method, apparatus, device and storage medium.

[0004] The technical solution of this application is achieved as follows:

[0005] In a first aspect, a data processing method is provided, the method comprising:

[0006] Get field information of multiple tables in the database;

[0007] Determine association information between any two tables based on field information of the multiple tables; wherein the association information includes a first association relationship and a first association weight;

[0008] Constructing a relationship network of the plurality of tables according to the association information between any two tables; wherein the relationship network includes: a second association relationship and a second association weight;

[0009] Calculating the PageRank value of each table according to the relationship network; wherein the PageRank value is used to represent the association value of each table in the relationship network;

[0010] Data mining is performed on the tables in the database based on the PageRank values ​​of the tables.

[0011] In the above solution, determining the association information between any two tables based on the field information of the multiple tables includes:

[0012] Determine, based on field information of the multiple tables, a hard connection relationship and a soft connection relationship between a first table and a second table among the multiple tables; use the hard connection relationship and the soft connection relationship between the first table and the second table as a first association relationship; and determine, based on the hard connection relationship and the soft connection relationship between the first table and the second table, a first association weight between the first table and the second table.

[0013] In the above solution, the field information includes the number of primary and foreign key associations and field names; the hard connection relationship includes the number of hard connections and hard connection weights, and the soft connection relationship includes the number of soft connections and soft connection weights.

[0014] In the above solution, determining the hard connection relationship and the soft connection relationship between the first table and the second table in the multiple tables according to the field information of the multiple tables includes:

[0015] Determine the number of hard connections between the two tables based on the number of primary and foreign key associations between the two tables; determine the hard connection weight between the two tables based on a preset hard connection coefficient and a basic connection coefficient; determine the number of soft connections between the two tables based on the number of associations of the same field name between the two tables; determine the soft connection weight between the two tables based on a preset soft connection coefficient and the basic connection coefficient;

[0016] The determining, based on the hard connection relationship and the soft connection relationship between the first table and the second table, a first association weight between the first table and the second table includes:

[0017] According to the hard connection weight and the soft connection weight between the first table and the second table, a weighted calculation is performed on the number of hard connections and the number of soft connections between the first table and the second table to obtain a first association weight between the first table and the second table.

[0018] In the above solution, constructing a relationship network of the multiple tables based on the association information between any two tables includes:

[0019] All tables in the plurality of tables that have a first association relationship with the i-th table are used as association tables of the i-th table;

[0020] Obtaining a second association relationship between the i-th table and its associated table according to the first association relationship between the i-th table and its associated table;

[0021] A second association weight between the i-th table and its associated table is obtained according to the first association weight and the basic weight coefficient between the i-th table and its associated table.

[0022] In the above solution, calculating the PageRank value of each table based on the relationship network includes:

[0023] In this iteration, at least one associated table of the j-th table is determined according to the second associated relationship;

[0024] Performing iterative processing based on the second association weight between the j-th table and the at least one association table, and the PageRank value of the at least one association table after the previous iteration, to obtain the PageRank value of the j-th table after the current iteration;

[0025] Calculate the PageRank difference between the PageRank value of each table in the relationship network after this iteration and the PageRank value after the previous iteration;

[0026] The PageRank differences of each table in the relationship network are summed and averaged to obtain the update amount;

[0027] If the update amount is less than or equal to a preset constant, stop iteration and obtain the PageRank value of each table in the relationship network;

[0028] If the update amount is greater than the preset constant, the next iteration is performed.

[0029] In the above solution, the iterative processing based on the second association weight between the j-th table and the at least one association table and the PageRank value of the at least one association table after the previous iteration to obtain the PageRank value of the j-th table after the current iteration includes:

[0030] Calculating the product of a second association weight between the j-th table and the at least one association table and a PageRank value of the at least one association table after a previous iteration;

[0031] The at least one obtained product is summed to obtain the PageRank value of the j-th table after the current iteration.

[0032] In a second aspect, a data processing device is provided, characterized in that the device includes:

[0033] The acquisition module is used to obtain field information of multiple tables in the database;

[0034] A processing module, configured to determine association information between any two tables based on field information of the plurality of tables; wherein the association information includes a first association relationship and a first association weight;

[0035] The processing module is further configured to construct a relationship network of the plurality of tables based on the association information between any two tables; wherein the relationship network includes: a second association relationship and a second association weight;

[0036] The processing module is further configured to calculate the PageRank value of each table based on the relationship network; wherein the PageRank value is used to represent the association value of each table in the relationship network;

[0037] The processing module is further configured to perform data mining on the tables in the database based on the PageRank values ​​of the tables.

[0038] According to a third aspect, a data processing device is provided, comprising: a processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to execute the steps of any of the aforementioned methods when running the computer program.

[0039] In a fourth aspect, a computer storage medium is provided, on which a computer program is stored, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0040] The present application discloses a data processing method, which includes: obtaining field information of multiple tables in a database; determining the association information between any two tables based on the field information of the multiple tables; wherein the association information includes a first association relationship and a first association weight; constructing a relationship network of the multiple tables based on the association information between the any two tables; wherein the relationship network includes: a second association relationship and a second association weight; calculating the PageRank value of each table based on the relationship network; wherein the PageRank value is used to characterize the association value of each table in the relationship network; and performing data mining on the tables in the database based on the PageRank value of each table. In this way, by extracting the table field information in the database to construct a table relationship network and obtaining the PageRank value of each table, the association information between the tables and the association value of each table in the database can be obtained, simplifying subsequent data mining. The entire process does not require additional business logic or development logic support, breaks down the barriers between various industries and data platforms, weakens the impact of specific fields on data analysis work, and improves the degree of intelligent automation of data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of the first flow chart of the data processing method in an embodiment of the present application;

[0042] Figure 2 This is a second flow chart of the data processing method in an embodiment of the present application;

[0043] Figure 3 This is a schematic diagram of the relationship network structure in the embodiment of this application;

[0044] Figure 4This is a third flow chart of the data processing method in an embodiment of the present application;

[0045] Figure 5 This is a flow chart of calculating the PageRank values ​​of each table in a relationship network in an embodiment of the present application;

[0046] Figure 6 This is a schematic diagram of the structure of a data processing device in an embodiment of the present application;

[0047] Figure 7 Schematic diagram of the structure of the data processing device in the embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0049] Figure 1 This is a first flow chart of the data processing method in the embodiment of the present application. Figure 1 As shown, the data processing method may specifically include:

[0050] Step 101: Obtain field information of multiple tables in the database.

[0051] There are a large number of tables in the database, and each table contains a large amount of field information. When processing data in the database, it is necessary to obtain the field information of the tables in the database. For example, the field information of the table may include the table field name, the table primary key, the table foreign key, and other information.

[0052] In some embodiments, before obtaining the field information of multiple tables in the database, the method also includes performing data preprocessing. Exemplarily, the data preprocessing may include: obtaining all tables and field information in the database; merging tables with exactly the same field names based on all tables and field information to obtain the multiple tables; eliminating the corresponding fields of each table in the multiple tables whose field names appear at a frequency greater than a frequency threshold to obtain the field information of the multiple tables.

[0053] For example, fields with a frequency greater than a threshold (typically 0.5) can be removed because these fields are generally used to represent basic database information, such as creation time and modification time. These fields are meaningless when building relationships between tables. Data preprocessing can simplify the table and field information in the database, reducing the amount of subsequent computation.

[0054] Step 102: Determine association information between any two tables based on the field information of the multiple tables; wherein the association information includes a first association relationship and a first association weight.

[0055] Databases contain numerous tables, some of which are related to each other. In data analysis, when extracting information from a particular table, analysts will inevitably refer to other tables as indexes or supplements to that table's information. In practical applications, the relationship between two tables can be determined based on their field information, such as whether they have the same field name, the number of identical field names, or whether one table's primary key is another table's foreign key.

[0056] Here, the first association relationship is used to indicate that there is a certain association between two tables. If there is no association between the two tables, then there is no first association relationship between the two tables. For example, if two tables have the same field name, or the primary key of one table is the foreign key of the other table, then there is a first association relationship between the two tables.

[0057] Here, the first association weight is used to quantitatively represent the degree of association between two tables having a first association relationship. A larger first association weight value indicates a stronger degree of association between the two tables.

[0058] In a database, the more a table is associated with other tables, the more important it is. By determining the association information between any two tables, we can provide a foundation for subsequent analysis of the associated value of each table in the database.

[0059] Step 103: constructing a relationship network of the plurality of tables according to the association information between any two tables; wherein the relationship network includes: a second association relationship and a second association weight.

[0060] Here, the second association relationship is used to represent the association between a table in the relationship network and all other tables that have a first association relationship with the table. Through the second association relationship, all other tables in the relationship network that have a first association relationship with a certain table can be queried.

[0061] Here, the second association weight is used to quantitatively represent the degree of association between a table and other tables that have a first association relationship with the table in a relationship network. The larger the first association weight value, the stronger the degree of association between the two tables in the relationship network.

[0062] The constructed relationship network includes the second association relationship and the second association weight between the tables in the database. For example, a relationship network may correspond to a database and be used to represent the relationship between the tables in the database.

[0063] By constructing a relationship network, you can query all other tables in the relationship network that have a first association relationship with a certain table. You can also quantitatively characterize the degree of association between a table and other tables in the relationship network that have a first association relationship with the table, providing a basis for subsequent analysis of the association value of each table in the database.

[0064] Step 104: Calculate the PageRank value of each table based on the relationship network; wherein the PageRank value is used to represent the association value of each table in the relationship network.

[0065] Here, the PageRank value of each table is used to represent the association value of each table in the relationship network. For example, the association value of a table can be obtained based on the second association relationship and the second association weight between the table and other tables in the relationship network. The larger the PageRank value of a table, the easier it is to be associated with other tables in the database, and the greater the association value of the table in the relationship network.

[0066] In some embodiments, calculating the PageRank value of each table according to the relationship network includes:

[0067] In this iteration, at least one associated table of the jth table is determined according to the second association relationship; iterative processing is performed according to the second association weight between the jth table and the at least one associated table, and the PageRank value of the at least one associated table after the previous iteration to obtain the PageRank value of the jth table after this iteration; the PageRank difference between the PageRank value of each table in the relationship network after this iteration and the PageRank value after the previous iteration is calculated; the PageRank differences of each table in the relationship network are summed and averaged to obtain an update amount; if the update amount is less than or equal to a preset constant, the iteration is stopped to obtain the PageRank value of each table in the relationship network; if the update amount is greater than the preset constant, the next iteration is executed.

[0068] Here, the jth table is any table in the database corresponding to the relationship network. For example, when performing the first iteration, the preset original PageRank value is used as the PageRank value after the previous iteration.

[0069] In some embodiments, performing iterative processing based on the second association weight between the j-th table and the at least one association table and the PageRank value of the at least one association table after the previous iteration to obtain the PageRank value of the j-th table after the current iteration includes:

[0070] Calculate the product of the second association weight between the j-th table and the at least one association table and the PageRank value of the at least one association table after the previous iteration; sum the at least one obtained product to obtain the PageRank value of the j-th table after the current iteration.

[0071] By calculating the PageRank value of each table, we can determine the importance of each table in the database's relationship network, thereby quantitatively determining the association between tables and the value of table entities in the entire database, achieving preliminary analysis of the database and simplifying subsequent data analysis and data mining.

[0072] Step 105: Perform data mining on the tables in the database based on the PageRank values ​​of the tables.

[0073] Here, data mining can be the process of searching for information hidden within large amounts of data through algorithms. For example, data mining based on the PageRank values ​​of the tables described above can include: calculating PageRank values ​​before data analysis to recommend important tables and reduce the workload of data analysis; visualizing business structures to obtain a logical relationship diagram for the business; and optimizing the database structure.

[0074] Here, the execution subject of steps 101 to 105 may be a processor of a data processing device.

[0075] The technical solution of this application extracts table field information from the database, determines the association information between any two tables, constructs a table relationship network, and obtains the PageRank value of each table. This can characterize the association value of each table in the database, simplifying subsequent data mining. The entire process does not require additional business logic or development logic support, breaking down the barriers between industries and data platforms, weakening the impact of specific fields on data analysis work, and improving the level of intelligent and automated data analysis.

[0076] In order to better reflect the purpose of this application, further examples are given based on the embodiments of this application. Figure 2 This is a second flow chart of the data processing method in the embodiment of the present application. Figure 2 As shown, the data processing method specifically includes:

[0077] Step 201: Obtain field information of multiple tables in the database.

[0078] In real-world applications, databases contain numerous tables, each containing a wealth of field information. Table field information contains basic information about the table. Obtaining field information from multiple tables in a database provides a data foundation for subsequent table relationship analysis and data mining. For example, field information can include table field names, primary keys, and foreign keys.

[0079] The multiple tables here can be all tables in the database after data preprocessing, or some tables in the database selected according to requirements.

[0080] In some embodiments, before obtaining the field information of multiple tables in the database, the method also includes performing data preprocessing. Exemplarily, the data preprocessing may include: obtaining all tables and field information in the database; merging tables with exactly the same field names based on all tables and field information to obtain the multiple tables; eliminating the corresponding fields of each table in the multiple tables whose field names appear at a frequency greater than a frequency threshold to obtain the field information of the multiple tables.

[0081] For example, fields with a frequency greater than a threshold (typically 0.5) can be removed because these fields are generally used to represent basic database information, such as creation time and modification time. These fields are meaningless when building relationships between tables. Data preprocessing can simplify the table and field information in the database, reducing the amount of subsequent computation.

[0082] Step 202: Determine a hard connection relationship and a soft connection relationship between a first table and a second table in the multiple tables according to the field information of the multiple tables.

[0083] Here, the first table and the second table are any two tables in the plurality of tables. Exemplarily, the field information may include the field names, primary keys, and foreign keys of the tables. For example, the association information between the two tables may be determined based on whether the two tables have the same field names, the number of the same field names, or whether the primary key of one table is the foreign key of another table.

[0084] If the primary key of one table is also the foreign key of another table, the two tables are said to have a hard link relationship. This relationship is a primary-foreign key relationship. Hard links are defined when the database is built and contain information the designer created when designing the database. A hard link relationship between two tables indicates a certain degree of correlation between them.

[0085] If a table and another table have the same field name, they are said to have a soft link relationship. The relationship between the two tables is a same-field name relationship. Soft links are typically not defined when building a database structure, but in business scenarios, it's common to perform join queries on tables with the same field name due to certain business requirements. The existence of a soft link relationship between two tables indicates a certain degree of correlation between them.

[0086] The association of a hard link is stronger than that of a soft link, but the strength of the association between the two tables is also related to the number of hard links and soft links.

[0087] In some embodiments, the field information includes the number of primary and foreign key associations and field names; the hard connection relationship includes the number of hard connections and hard connection weights; and the soft connection relationship includes the number of soft connections and soft connection weights.

[0088] Illustratively, in some embodiments, determining, based on field information of multiple tables, a hard link relationship and a soft link relationship between a first table and a second table in the multiple tables includes:

[0089] Determine the number of hard connections between the first table and the second table based on the number of primary and foreign key associations between the first table and the second table; determine the hard connection weight between the first table and the second table based on a preset hard connection coefficient and a basic connection coefficient; determine the number of soft connections between the first table and the second table based on the number of associations with the same field name between the any two tables; and determine the soft connection weight between the first table and the second table based on a preset soft connection coefficient and the basic connection coefficient.

[0090] For example, the hard link coefficient is greater than the soft link coefficient, and the basic link coefficient can be set to 1, which can be adjusted according to actual needs. By setting different connection weights for hard links and soft links, and combining the number of hard links and soft links, the hard link relationship and soft link relationship between the two tables can be more accurately obtained.

[0091] Step 203: The hard connection relationship and the soft connection relationship between the first table and the second table are used as a first association relationship.

[0092] Here, the first association relationship is used to represent that there is a certain association between the two tables. If there is no association between the two tables, then there is no first association relationship between the two tables.

[0093] The hard connection relationship and the soft connection relationship between the first table and the second table are regarded as the first association relationship. If there is no hard connection relationship and the soft connection relationship between the first table and the second table, then there is no first association relationship between the first table and the second table.

[0094] Step 204: Determine a first association weight between the first table and the second table based on the hard connection relationship and the soft connection relationship between the first table and the second table.

[0095] In some embodiments, determining the first association weight between the first table and the second table according to the hard connection relationship and the soft connection relationship between the first table and the second table includes:

[0096] According to the hard connection weight and the soft connection weight between the first table and the second table, a weighted calculation is performed on the number of hard connections and the number of soft connections between the first table and the second table to obtain a first association weight between the first table and the second table.

[0097] By dividing the relationship between two tables into hard connections and soft connections and combining different connection weights, we can obtain the relationship information between the two tables. This relationship information can more accurately represent the strength of the relationship between the two tables.

[0098] Step 205: All tables in the plurality of tables that have a first association relationship with the i-th table are used as association tables of the i-th table.

[0099] Here, the multiple tables are all tables in the database, and the i-th table is any table in the database.

[0100] Step 206: Obtain a second association relationship between the i-th table and its associated tables based on the first association relationship between the i-th table and its associated tables.

[0101] Here, the second association relationship is the relationship between the i-th table and all its associated tables in the database. Through the second association relationship, all tables in the database that have a first association relationship with the i-th table can be determined.

[0102] Step 207: Obtain a second association weight between the i-th table and its associated table according to the first association weight and the basic weight coefficient between the i-th table and its associated table.

[0103] For example, Figure 3 A schematic diagram of a relational network structure is shown. In the diagram, a black dot represents a table in the database. The line between any two black dots represents the first association relationship between the tables represented by the two black dots. If there is no line between the two black dots, it means that there is no first association relationship between the two tables. All lines connecting any black dot to other black dots represent the second association relationship between the table represented by the black dot and its associated table. Figure 3 You can quickly determine all the related tables of a table in the relationship network.

[0104] Step 208: Calculate the PageRank value of each table based on the relationship network; wherein the PageRank value is used to represent the association value of each table in the relationship network.

[0105] Here, the PageRank value of each table is used to represent the association value of each table in the relationship network. For example, the association value of a table can be obtained based on the second association relationship and the second association weight between the table and other tables in the relationship network. The larger the PageRank value of a table, the easier it is to be associated with other tables in the database, and the greater the association value of the table in the relationship network.

[0106] In some embodiments, calculating the PageRank value of each table according to the relationship network includes:

[0107] In this iteration, at least one associated table of the jth table is determined according to the second association relationship; iterative processing is performed according to the second association weight between the jth table and the at least one associated table, and the PageRank value of the at least one associated table after the previous iteration to obtain the PageRank value of the jth table after this iteration; the PageRank difference between the PageRank value of each table in the relationship network after this iteration and the PageRank value after the previous iteration is calculated; the PageRank differences of each table in the relationship network are summed and averaged to obtain an update amount; if the update amount is less than or equal to a preset constant, the iteration is stopped to obtain the PageRank value of each table in the relationship network; if the update amount is greater than the preset constant, the next iteration is executed.

[0108] Here, the jth table is any table in the database corresponding to the relationship network. For example, when performing the first iteration, the preset original PageRank value is used as the PageRank value after the previous iteration.

[0109] In some embodiments, performing iterative processing based on the second association weight between the j-th table and the at least one association table and the PageRank value of the at least one association table after the previous iteration to obtain the PageRank value of the j-th table after the current iteration includes:

[0110] Calculate the product of the second association weight between the j-th table and the at least one association table and the PageRank value of the at least one association table after the previous iteration; sum the at least one obtained product to obtain the PageRank value of the j-th table after the current iteration.

[0111] By calculating the PageRank value of each table, we can determine the importance of each table in the database's relationship network, thereby quantitatively determining the association between tables and the value of table entities in the entire database, achieving preliminary analysis of the database and simplifying subsequent data analysis and data mining.

[0112] Step 209: Perform data mining on the tables in the database based on the PageRank values ​​of the tables.

[0113] Here, data mining can be the process of searching for information hidden within large amounts of data through algorithms. For example, data mining based on the PageRank values ​​of the tables described above can include: calculating PageRank values ​​before data analysis to recommend important tables and reduce the workload of data analysis; visualizing business structures to obtain a logical relationship diagram for the business; and optimizing the database structure.

[0114] Here, the execution subject of steps 201 to 209 may be a processor of a data processing device.

[0115] The technical solution of the present application analyzes the hard-link relationship and soft-link relationship between any two tables, and determines the association information between any two tables based on the hard-link relationship and soft-link relationship, so that the association information can accurately represent the degree of association between the two tables. By constructing a table relationship network and obtaining the PageRank value of each table, the association value of each table in the database is quantitatively given, simplifying subsequent data mining. The entire process does not require additional business logic or development logic support, breaking down the barriers between various industries and data platforms, weakening the impact of specific fields on data analysis work, and improving the degree of intelligent automation of data analysis.

[0116] In order to better reflect the purpose of this application, further examples are given based on the above embodiments of this application. Figure 4 This is a third flow chart of the data processing method in the embodiment of the present application, as shown in FIG. Figure 4 As shown, the data processing method specifically includes:

[0117] Step 401: Data preprocessing.

[0118] Specifically, connect to the database to obtain the table and field information in the database; merge tables with exactly the same field names based on the content of the table fields; count the frequency of each field in the database table and eliminate fields with a frequency greater than a preset threshold.

[0119] Step 402: Extract pre-processed table and field information.

[0120] Step 403: Calculate the first association weight between tables.

[0121] Specifically, the first association weights between tables are calculated using hard connection relationships and soft connection relationships.

[0122] Specifically, the primary and foreign key association is usually called a hard connection. If the primary key of table a corresponds to the foreign key of table b, that is, T a (primary_key)=T b (foreign_key), then increase the weight of table a and table b to w h =α·w connection relationship; where w h is the hard connection weight, α is the hard connection coefficient, and w is the basic connection coefficient.

[0123] The same field name association is usually called a soft connection. If table a and table b have a common field name, that is, T a (field)=T b (field), then add weight w to table a and table b s =β·w connection relationship, where w s is the soft connection weight, β is the soft connection coefficient, and w is the basic connection coefficient.

[0124] The hard connection relationship and soft connection relationship between any two tables are weighted with different weights.

[0125] According to the formula w ab =iw h +jw s

[0126] Get w between table a and table b ab Among them, if w ab If it is not 0, use w ab Represents the first association weight between table a and table b. If w is calculated according to the formula ab If it is 0, it means that there is no first association relationship and first association weight between table a and table b.

[0127] Where i and j are the number of hard links and soft links between table a and table b, respectively, and w h and w s These are the hard link weight and soft link weight between table a and table b, respectively. Typically, two tables have at most one primary and foreign key relationship, but can have multiple relationships with the same field name. The number of hard links ranges from 0 or 1, and the number of soft links ranges from 0 to N, where N is an integer greater than or equal to 1.

[0128] Step 404: Build a relationship network.

[0129] Specifically, according to step 402, the first association weight between the two tables in the database is obtained, and the first association set E={e ab ,e ac ,…} to represent all the first association relationships in the relationship network. The number of elements in the set E is the number of all the first association relationships in the relationship network.

[0130] According to the formula w' ab =(iw h +jw s )·e, and obtain the second association weight w' between table a and table b ab , where i and j are the number of hard links and soft links between table a and table b, respectively, and w h and w s are the hard connection weight and soft connection weight between table a and table b respectively. e represents the basic connection weight, which is a constant preset for the database and is generally set to 1. It can be adjusted according to needs. In the same database, e is the same. Then calculate the second association weight corresponding to all the first association relationships in the relationship network, and obtain the second association weight set W = {w' ab ,w' ac ,…}, the number of elements in the set W is the number of all first association relationships in the relationship network.

[0131] A table relationship network G = (V, E, W) is obtained based on the first association relationship set E and the second association weight set W. Where V represents the set, E represents the first association relationship set, and W represents the second association weight set.

[0132] By forming a relationship network consisting of a set of tables, a first association relationship set, and a second association weight set, the association and degree of association between tables in the database can be intuitively and quantitatively displayed, providing a basis for subsequently obtaining the association value of each table in the database.

[0133] Step 405: Calculate the PageRank value of each table in the relationship network.

[0134] Specifically, Figure 5 This figure shows a flow chart of calculating the PageRank value of each table in a relational network. Figure 5 As shown,

[0135] Step 501: Initialize the PageRank value of each table.

[0136] Specifically, initialize the PageRank value of each table to PangRank(d i )1, where i is the number of the table in the database relational network, and the number of tables is N.

[0137] Step 502: Update the PageRank value of the table according to the second association weight and the first association relationship.

[0138] Specifically, according to the PageRank value of the current table in the relationship network and the second association weights between all the associated tables of the current table, the formula

[0139]

[0140] Traverse the tables in the relationship network and calculate the updated PageRank value after iteration. i ) k+1 It is table d i The PageRank value of the k+1th time after the update, M(d i ) is the same as table d i The set of association tables, w' ij is the same as Table D i With table d j Here, Table d i The association table is the relationship table with the relationship network and table d i The table with the first association relationship.

[0141] Step 503: Calculate the update amount Δ.

[0142] Specifically, the difference between the PageRank value of all tables in the relationship network after the update iteration and the previous one is calculated by the formula

[0143]

[0144] The update amount Δ is obtained by summing and averaging. In the formula, D represents the set of all tables in the relational network, and N represents the number of all tables in the relational network.

[0145] Step 504: The update amount Δ is less than or equal to the preset constant. If so, execute step 505; if not, return to execute step 502;

[0146] Step 505: Stop iteration and return the PageRank value of the table.

[0147] Through iteration, the sum of the degrees of association between a table and its associated tables in the relational network is calculated, i.e., the table's association value, which quantitatively represents the importance of the table in the database. The higher the PageRank value of a table, the greater its association value in the database, and the more important the table is when analyzing the database. This provides a foundation for subsequent data analysis and data mining of the database.

[0148] Step 406: Quantitatively analyze the database tables based on the PageRank values ​​of the tables.

[0149] Specifically, based on the PageRank values ​​of the tables, we can quantitatively determine the association value of each table in the database and the importance of each table in the database. We can then analyze the database based on the importance of each table, such as sorting the importance of tables and analyzing the most associated tables in the database. This entire process automatically analyzes the importance of each table in the database without requiring additional business logic or demand information. This allows users to prioritize analyzing tables of higher importance during subsequent data mining, effectively simplifying the data mining process.

[0150] Here, the execution subject of the above steps may be a processor of a data processing device.

[0151] The technical solution of the present application divides the association between any two tables into hard connection relationships and soft connection relationships, and determines the association weight between any two tables based on the hard connection relationship and the soft connection relationship, so that the association weight can accurately and quantitatively express the degree of association between the two tables. By constructing a table relationship network, the association value of each table in the database and the importance of each table in the database can be quantitatively obtained. The entire process does not require additional business logic and demand information support, and can automatically analyze the importance of each table in the database and obtain results. It breaks through the barriers between various industries and data platforms, weakens the impact of specific fields on data analysis work, and improves the degree of intelligent automation of data analysis.

[0152] In order to implement the method of the embodiment of the present application, the embodiment of the present application further provides a data processing device 60, such as Figure 6 As shown, the device includes:

[0153] Acquisition module 601, used to obtain field information of multiple tables in the database;

[0154] Processing module 602 is configured to determine association information between any two tables based on field information of the multiple tables; wherein the association information includes a first association relationship and a first association weight;

[0155] The processing module 602 is further configured to construct a relationship network of the plurality of tables based on the association information between the arbitrary two tables; wherein the relationship network includes: a second association relationship and a second association weight;

[0156] The processing module 602 is further configured to calculate the PageRank value of each table based on the relationship network; wherein the PageRank value is used to represent the association value of each table in the relationship network;

[0157] The processing module 602 is further configured to perform data mining on the tables in the database based on the PageRank values ​​of the tables.

[0158] By extracting table field information from the database, determining the association between any two tables, constructing a table relationship network, and obtaining the PageRank value of each table, this technical solution can characterize the association value of each table in the database, simplifying subsequent data mining. This entire process requires no additional business logic or development logic support, breaking down the barriers between industries and data platforms, reducing the impact of specific fields on data analysis, and improving the level of intelligent and automated data analysis.

[0159] In some embodiments, the processing module 602 is used to determine the hard connection relationship and soft connection relationship between the first table and the second table in the multiple tables based on the field information of the multiple tables; use the hard connection relationship and soft connection relationship between the first table and the second table as the first association relationship; and determine the first association weight between the first table and the second table based on the hard connection relationship and soft connection relationship between the first table and the second table.

[0160] In some embodiments, the field information includes the number of primary and foreign key associations and field names; the hard connection relationship includes the number of hard connections and hard connection weights; and the soft connection relationship includes the number of soft connections and soft connection weights.

[0161] In some embodiments, the processing module 602 is configured to determine the number of hard connections between the two tables based on the number of primary and foreign key associations between the two tables; determine the hard connection weight between the two tables based on a preset hard connection coefficient and a basic connection coefficient; determine the number of soft connections between the two tables based on the number of associations between the two tables having the same field name; and determine the soft connection weight between the two tables based on a preset soft connection coefficient and the basic connection coefficient.

[0162] Used to perform weighted calculation on the number of hard connections and the number of soft connections between the first table and the second table according to the hard connection weight and the soft connection weight between the first table and the second table, so as to obtain a first association weight between the first table and the second table.

[0163] In some embodiments, the processing module 602 is used to take all tables in the multiple tables that have a first association relationship with the i-th table as associated tables of the i-th table; obtain the second association relationship between the i-th table and its associated tables based on the first association relationship between the i-th table and its associated tables; and obtain the second association weight between the i-th table and its associated tables based on the first association weight and basic weight coefficient between the i-th table and its associated tables.

[0164] In some embodiments, the processing module 602 is used to determine at least one associated table of the jth table according to the second association relationship in this iteration; perform iterative processing according to the second association weight between the jth table and the at least one associated table, and the PageRank value of the at least one associated table after the previous iteration to obtain the PageRank value of the jth table after this iteration; calculate the PageRank difference between the PageRank value of each table in the relationship network after this iteration and the PageRank value after the previous iteration; sum and average the PageRank differences of each table in the relationship network to obtain an update amount; if the update amount is less than or equal to a preset constant, stop the iteration and obtain the PageRank value of each table in the relationship network; if the update amount is greater than the preset constant, execute the next iteration.

[0165] In some embodiments, the processing module 602 is used to calculate the product of the second association weight between the j-th table and the at least one association table and the PageRank value of the at least one association table after the previous iteration; and sum the at least one product obtained to obtain the PageRank value of the j-th table after the current iteration.

[0166] Based on the hardware implementation of each unit in the above data processing, the embodiment of the present application also provides another data processing device 70, such as Figure 7 As shown, the data processing device includes: a processor 701 and a memory 702 configured to store a computer program that can be run on the processor; wherein the processor 701 is configured to execute the method steps in the above embodiments when running the computer program.

[0167] Of course, in actual application, Figure 7 As shown, the various components in the data processing device are coupled together via a bus system 703. It is understood that the bus system 703 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 703 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 7 Various buses are labeled as bus system 703.

[0168] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understood that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0169] The above-mentioned memory can be a volatile memory (volatile memory), such as a random-access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above types of memory, and provides instructions and data to the processor.

[0170] In an exemplary embodiment, the present application also provides a computer-readable storage medium, such as a memory including a computer program, which can be executed by a processor of a data processing device to complete the steps of the aforementioned method.

[0171] It should be understood that the terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. The expressions "having", "can have", "including" and "comprising", or "can include" and "can include" in this application can be used to indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations or components), but do not exclude the presence of additional features.

[0172] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various types of information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another and are not necessarily used to describe a specific order or precedence. For example, first information could be referred to as second information, and similarly, second information could be referred to as first information without departing from the scope of the present invention.

[0173] The technical solutions described in the embodiments of this application can be combined arbitrarily unless there is any conflict.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed methods, devices and equipment can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0175] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0176] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0177] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Get field information of multiple tables in the database; Determining association information between any two tables based on field information of the multiple tables; wherein the association information includes a first association relationship and a first association weight; determining the association information between any two tables based on the field information of the multiple tables includes: determining a hard connection relationship and a soft connection relationship between a first table and a second table in the multiple tables based on the field information of the multiple tables; using the hard connection relationship and the soft connection relationship between the first table and the second table as a first association relationship; and determining a first association weight between the first table and the second table based on the hard connection relationship and the soft connection relationship between the first table and the second table; Constructing a relationship network of the multiple tables based on the association information between any two tables; wherein the relationship network includes: a second association relationship and a second association weight; constructing the relationship network of the multiple tables based on the association information between any two tables includes: using all tables in the multiple tables that have a first association relationship with the i-th table as association tables of the i-th table; obtaining a second association relationship between the i-th table and its associated tables based on the first association relationship between the i-th table and its associated tables; and obtaining a second association weight between the i-th table and its associated tables based on the first association weight and a basic weight coefficient between the i-th table and its associated tables. Calculating the PageRank value of each table according to the relationship network; wherein the PageRank value is used to characterize the association value of each table in the relationship network; calculating the PageRank value of each table according to the relationship network includes: in this iteration, determining at least one associated table of the j-th table according to the second association relationship; performing iterative processing according to the second association weight between the j-th table and the at least one associated table, and the PageRank value of the at least one associated table after the previous iteration, to obtain the PageRank value of the j-th table after this iteration; calculating the PageRank difference between the PageRank value of each table in the relationship network after this iteration and the PageRank value after the previous iteration; summing and averaging the PageRank differences of each table in the relationship network to obtain an update amount; if the update amount is less than or equal to a preset constant, stopping the iteration and obtaining the PageRank value of each table in the relationship network; if the update amount is greater than the preset constant, executing the next iteration; Data mining is performed on the tables in the database based on the PageRank values ​​of the tables.

2. The method according to claim 1, characterized in that The field information includes the number of primary and foreign key associations and field names; The hard connection relationship includes the number of hard connections and the hard connection weight, and the soft connection relationship includes the number of soft connections and the soft connection weight.

3. The method according to claim 2, characterized in that Determining the hard connection relationship and the soft connection relationship between the first table and the second table in the multiple tables according to the field information of the multiple tables includes: determining the number of hard connections between the first table and the second table according to the number of primary and foreign key associations between the first table and the second table; determining a hard connection weight between the first table and the second table according to a preset hard connection coefficient and a basic connection coefficient; Determining the number of soft connections between the first table and the second table according to the number of associations of the same field names between the two tables; Determining a soft connection weight between the first table and the second table according to a preset soft connection coefficient and the basic connection coefficient; The determining, based on the hard connection relationship and the soft connection relationship between the first table and the second table, a first association weight between the first table and the second table includes: According to the hard connection weight and the soft connection weight between the first table and the second table, a weighted calculation is performed on the number of hard connections and the number of soft connections between the first table and the second table to obtain a first association weight between the first table and the second table.

4. The method according to claim 1, wherein The iterative processing according to the second association weight between the j-th table and the at least one association table, and the PageRank value of the at least one association table after the previous iteration, to obtain the PageRank value of the j-th table after the current iteration, includes: Calculating the product of a second association weight between the j-th table and the at least one association table and a PageRank value of the at least one association table after a previous iteration; The at least one obtained product is summed to obtain the PageRank value of the j-th table after the current iteration.

5. A data processing device, characterized in that: The device comprises: The acquisition module is used to obtain field information of multiple tables in the database; a processing module, configured to determine association information between any two tables based on field information of the multiple tables; wherein the association information includes a first association relationship and a first association weight; the processing module determining the association information between any two tables based on the field information of the multiple tables includes: determining a hard connection relationship and a soft connection relationship between a first table and a second table in the multiple tables based on the field information of the multiple tables; using the hard connection relationship and the soft connection relationship between the first table and the second table as a first association relationship; and determining a first association weight between the first table and the second table based on the hard connection relationship and the soft connection relationship between the first table and the second table; The processing module is further configured to construct a relationship network of the multiple tables based on the association information between the arbitrary two tables; wherein the relationship network includes: a second association relationship and a second association weight; the processing module constructing the relationship network of the multiple tables based on the association information between the arbitrary two tables includes: using all tables in the multiple tables that have a first association relationship with the i-th table as association tables of the i-th table; obtaining a second association relationship between the i-th table and its associated tables based on the first association relationship between the i-th table and its associated tables; and obtaining a second association weight between the i-th table and its associated tables based on the first association weight and a basic weight coefficient between the i-th table and its associated tables. The processing module is further configured to calculate the PageRank value of each table according to the relationship network; wherein the PageRank value is used to characterize the association value of each table in the relationship network; the processing module calculates the PageRank value of each table according to the relationship network, including: in this iteration, determining at least one associated table of the j-th table according to the second association relationship; performing iterative processing according to the second association weight between the j-th table and the at least one associated table, and the PageRank value of the at least one associated table after the previous iteration, to obtain the PageRank value of the j-th table after this iteration; calculating the PageRank difference between the PageRank value of each table in the relationship network after this iteration and the PageRank value after the previous iteration; summing and averaging the PageRank differences of each table in the relationship network to obtain an update amount; if the update amount is less than or equal to a preset constant, stopping the iteration to obtain the PageRank value of each table in the relationship network; if the update amount is greater than the preset constant, executing the next iteration; The processing module is further configured to perform data mining on the tables in the database based on the PageRank values ​​of the tables.

6. A data processing device, characterized in that: The device comprises: a processor and a memory configured to store a computer program capable of being executed on the processor, Wherein, the processor is configured to execute the steps of the method according to any one of claims 1 to 4 when running the computer program.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images