Method of processing data and related apparatus

Abstract

The application provides a method for processing data and a related device. The method comprises the following steps: determining data blood relations of N tables, each of the N tables comprising at least one piece of data, N being a positive integer greater than or equal to 2; determining M tables having blood relations from the N tables according to the data blood relations of the N tables, M being a positive integer greater than or equal to 2 and smaller than or equal to N; and determining an association relationship between the M tables according to the data blood relations of the M tables and the data in the M tables. The above technical solution can conveniently determine the association relationship between tables based on data blood relations.

Description

Technical Field

[0001] This application relates to the field of information technology, and more specifically, to methods and related apparatus for processing data. Background Technology

[0002] Dimensional modeling is a data modeling method used in data warehouse construction. Simply put, dimensional modeling involves building a data warehouse based on fact tables and dimension tables (also known as dimension tables).

[0003] Dimensional modeling can intuitively reflect the business problems in a business model. However, the existing dimensional modeling process is complex, which hinders its widespread adoption. Summary of the Invention

[0004] This application provides a method and related apparatus for processing data, which can conveniently determine the management relationships between data.

[0005] In a first aspect, embodiments of this application provide a data processing method, comprising: determining the data lineage of N tables, each of the N tables including at least one piece of data, where N is a positive integer greater than or equal to 2; determining M tables with a lineage relationship from the N tables based on the data lineage of the N tables, where M is a positive integer greater than or equal to 2 and less than or equal to N; and determining the association relationship between the M tables based on the data lineage of the M tables and the data in the M tables. The above technical solution can conveniently determine the association relationship between tables based on data lineage.

[0006] In conjunction with the first aspect, in one possible implementation of the first aspect, determining the relationships between the M tables based on their data lineage and data includes: determining M1 dimension tables and M2 fact tables from the M tables based on their data lineage and data, where M1 and M2 are positive integers greater than or equal to 1, and the sum of M1 and M2 is less than or equal to M; and determining the relationships between the M tables based on the M1 dimension tables and the M2 fact tables. Using the above technical solution, dimension tables and fact tables can be determined based on data lineage, and then the relationships between the tables can be determined based on the dimension tables and fact tables.

[0007] In conjunction with the first aspect, in one possible implementation of the first aspect, determining M1 dimension tables and M2 fact tables from the M tables based on their data lineage and data includes: determining at least one first source table based on the data lineage of the M tables, where the first source table is a source table related to multiple target tables, and M is a positive integer greater than or equal to 2 and less than or equal to N; determining the M1 dimension tables from the at least one first source table based on the data and the data in the target tables of the at least one first source table; and determining the M2 fact tables from M3 tables other than the M1 dimension tables, where M3 is a positive integer greater than or equal to 1, and the sum of M3 and M1 is M. The characteristic of a dimension table is that its data is referenced by multiple tables. Through the above technical solution, dimension tables can be effectively determined from multiple source tables.

[0008] In conjunction with the first aspect, in one possible implementation of the first aspect, determining the M1 dimension tables from the at least one first source table based on data from the at least one first source table and data from the at least one first source table's destination table includes: determining the number of times data from each of the at least one first source table is repeated in each of the destination tables of each of the at least one first source table; and determining the M1 dimension tables from the at least one first source table based on the determined number of repetitions, wherein at least one piece of data from each of the M1 dimension tables is repeated more than a preset threshold in at least one destination table of each dimension table. By determining whether data in the source table is repeated in the destination table, the probability of misclassifying a source table as a dimension table can be effectively reduced.

[0009] In conjunction with the first aspect, in one possible implementation of the first aspect, determining the M2 fact tables from the M3 tables other than the M1 dimension tables includes: determining the target table in the M3 tables as the fact table.

[0010] In conjunction with the first aspect, in one possible implementation of the first aspect, determining the relationships between the M1 dimension tables and the M2 fact tables includes: determining the K corresponding to the j-th dimension table in the M1 dimension tables. j A fact table, in which K j Each fact table in the fact tables includes at least one piece of data from the j-th dimension table, j = 1, ..., M1, K j It is a positive integer greater than or equal to 1 and less than or equal to M2.

[0011] In conjunction with the first aspect, in one possible implementation of the first aspect, determining the relationships between the M1 dimension tables and the M2 fact tables further includes: determining the relationships between the K... jEach fact table in the fact table includes the number of times the data in the j-th dimension table is repeated, determining the K. j The data duplication relationship between the M1 dimension tables and the M2 fact tables is determined; a target model is established to indicate the correspondence and data duplication relationship between the M1 dimension tables and the M2 fact tables. This technical solution can intuitively reflect the relationship between the dimension tables and the fact tables through a model, and can also intuitively reflect whether data in the dimension tables appears repeatedly in the corresponding fact tables.

[0012] Secondly, embodiments of this application provide a computer device that includes units for implementing the first aspect or any possible implementation of the first aspect.

[0013] Thirdly, embodiments of this application provide a computer device including a processor for coupling with a memory to read and execute instructions and / or program code in the memory to perform the first aspect or any possible implementation of the first aspect.

[0014] Fourthly, embodiments of this application provide a chip system including logic circuitry for coupling with an input / output interface to transmit data via the input / output interface, thereby executing the first aspect or any possible implementation thereof.

[0015] Fifthly, embodiments of this application provide a computer-readable storage medium storing program code that, when executed on a computer, causes the computer to perform the first aspect or any possible implementation thereof. Attached Figure Description

[0016] Figure 1 This is a schematic flowchart of a data processing method provided according to an embodiment of this application.

[0017] Figure 2 The bloodline diagram is obtained based on the blood relationships in Tables 1 to 8.

[0018] Figure 3 It is another bloodline diagram obtained based on the bloodline relationships in Tables 1 to 8.

[0019] Figure 4 It is another bloodline diagram obtained based on the bloodline relationships in Tables 1 to 8.

[0020] Figure 5 The star-shaped model was obtained based on the blood relations and data in Tables 1 to 8.

[0021] Figure 6 It is a computer device provided according to an embodiment of this application.

[0022] Figure 7 This is a schematic structural block diagram of a computer device provided according to an embodiment of this application. Detailed Implementation

[0023] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0024] This application will present various aspects, embodiments, or features relating to systems that may include multiple devices, components, modules, etc. It should be understood and appreciated that individual systems may include additional devices, components, modules, etc., and / or may not include all the devices, components, modules, etc. discussed in conjunction with the accompanying drawings. Furthermore, combinations of these approaches are also possible.

[0025] Furthermore, in the embodiments of this application, the words "exemplary," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplary" is intended to present the concept in a concrete manner.

[0026] In the embodiments of this application, "corresponding" and "corresponding" can sometimes be used interchangeably. It should be noted that when the distinction is not emphasized, their intended meanings are consistent.

[0027] In the embodiments of this application, sometimes the subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0028] The network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0029] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0030] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0031] To help those skilled in the art better understand the technical solutions of this application, some concepts involved in the embodiments of this application will be briefly introduced first.

[0032] 1. Data lineage

[0033] From the generation, processing, integration, circulation, and eventual disappearance of data, a relationship naturally forms between the data. Borrowing a similar relationship from human society to express this relationship between data, we call it data lineage. Data lineage can also be referred to as data kinship, data lineage information, etc.

[0034] Data lineage includes information such as the flow and history of data throughout the data management lifecycle, recording the origin and development of data. Data lineage can be used to trace the source of data and track the data processing process.

[0035] Common methods for obtaining data lineage include automatic parsing, system tracing, and machine learning.

[0036] Automatic parsing is the process of using Structured Query Language (SQL) to parse a table, or a file containing a table from a stored procedure or during the Extract-Transform-Load (ETL) process, to obtain the table's data lineage.

[0037] System tracking occurs during the data processing flow, where the data processing program is responsible for sending data lineage information. For example, descriptive information about lineage can be added to the data annotation section.

[0038] Machine learning calculates the data similarity between tables based on the dependencies between them.

[0039] 2. Source table and target table

[0040] A source table is a table from which data flows in a data lineage.

[0041] The destination table refers to the table from which data flows in the data lineage.

[0042] For example, if table A is the source table of table B, then table B is a destination table. Table B can be called the destination table of table A. Conversely, if table B is the destination table of table A, then table A is a source table. In this case, table A can be called the source table of table B.

[0043] A table can be both a source table and a destination table. For example, table B is the destination table of table A, but table B can also be the source table of table C.

[0044] A source table can have one or more destination tables.

[0045] A destination table can also have one or more source tables.

[0046] If there are multiple tables, and any one of these tables is a source table or a destination table for one or more other tables, then these multiple tables can be said to be related.

[0047] For ease of description, in the embodiments of this application, if one table corresponds to another (or more) tables, it means that the former table is the source table or the destination table of the latter (or more) tables. For example, if table A corresponds to table B, it means that table A is the source table or the destination table of table B. If table A corresponds to both table B and table C, it means that table A is the source table or the destination table of both table B and table C.

[0048] Figure 1 This is a schematic flowchart of a data processing method provided according to an embodiment of this application.

[0049] 101. Determine the data lineage of N tables, where N is a positive integer greater than or equal to 2.

[0050] These N tables may originate from a database. They can come from the same database or different databases. These N tables can be one or more of the following: product inventory tables for different regions, product sales tables for different regions, product inventory tables of different types, product sales tables of different types, product sales tables of different years, product inventory tables of different years, statistical tables of different types, statistical tables of different years, and statistical tables of different years and different regions.

[0051] The data lineage of these N tables can be obtained through automatic parsing, system tracing, or machine learning.

[0052] Optionally, in some embodiments, if some of the N tables lack a data lineage, the user can be prompted that the data lineage collection was unsuccessful. In this case, the user can choose to ignore the tables for which no data lineage was collected, or manually determine the data lineage of these tables.

[0053] 102. Based on the bloodline data in the N tables, determine M tables that have bloodline relationships from the N tables, where M is a positive integer greater than or equal to 2 and less than or equal to N.

[0054] Some of the N tables obtained in step 101 may not be related to other tables. Therefore, these tables have no association with other tables and need to be excluded. The remaining M tables are related tables. In other words, any one of these M tables is a source table or a destination table for one or more other tables in these M tables.

[0055] Of course, in some cases, these N tables may all be related. In this case, the value of M is equal to N.

[0056] 103. Based on the data lineage of the M tables, determine the source tables from the M tables that contain multiple destination tables.

[0057] As mentioned above, the M tables are related. Therefore, any one of the M tables is either a source table or a destination table for one or more other tables. Furthermore, as mentioned above, a table can be both a source table for one table and a destination table for another.

[0058] For ease of description, table attributes can be divided into the following three types: attribute 1, attribute 2, and attribute 3. Tables with attribute 1 are only used as source tables, tables with attribute 2 are only used as destination tables, and tables with attribute 3 can be used as both source and destination tables.

[0059] As mentioned above, a source table may have one or more destination tables. Step 103 is to identify the source tables among the M tables that have multiple destination tables. For ease of description, the source table with multiple destination tables can be called the first source table, and the source table with only one destination table can be called the second source table.

[0060] 104. Determine whether the data in each first source table appears multiple times in one or more destination tables.

[0061] The phrase "data appears multiple times in one or more destination tables" means that each of the one or more destination tables includes at least α data points. The value of α can be set as needed. Typically, α can be a positive integer greater than or equal to 2. For example, in some embodiments, the value of α can be 2, 4, or 6.

[0062] 105. Based on the determination result of step 104, determine the dimension table from at least one determined first source table.

[0063] If data in a first source table appears multiple times in one or more destination tables, then the first source table can be identified as a dimension table. It can be understood that these multiple destination tables are the destination tables of the first source table.

[0064] If data in a first source table does not appear multiple times in any of the multiple destination tables, then the first source table is determined not to be a dimension table.

[0065] For example, suppose a first source table A1 has three destination tables: destination table B1, destination table B2, and destination table B3. Data D1 is a data point in the first source table A1. Assume the value of α is 2. If destination table B1 contains two data points D1, then data D1 can be said to appear multiple times in destination table B1. If destination table B2 contains only one data point D1, then data D1 does not appear multiple times in destination table B2. If destination table B3 contains ten data points D1, then data D1 can be said to appear multiple times in destination table B3. Since data D1 in the first source table A1 appears multiple times in both destination tables B1 and B3, the first source table A1 can be identified as a dimension table.

[0066] For example, suppose a source table A2 has two destination tables, destination table B4 and destination table B5. Suppose destination table B4 contains only one piece of data from source table A2, and destination table B5 also contains multiple pieces of data from source table A2, but each piece of data appears only once in destination table B5. In this case, it can be determined that source table A2 is not a dimension table.

[0067] 106. Determine the fact table from the M tables excluding the dimension table.

[0068] For ease of description, assume that there are M1 dimension tables in the M tables. Then, we need to determine the fact table from the M3 tables other than the M1 dimension tables in the M tables, where M3 is a positive integer greater than or equal to 1 and the sum of M3 and M1 is M.

[0069] The destination table in these M3 tables is the fact table. In other words, among the remaining tables besides the dimension tables, if a table has a source table, then that table is the fact table.

[0070] For ease of description, we can assume that there are M2 fact tables in the M tables, where M2 is an integer greater than or equal to 1 and M2 is less than or equal to M3.

[0071] 107. Based on the number of times the data in each of the M1 dimension tables appears in the corresponding fact table, determine the data duplication relationship between the M1 dimension tables and the M2 fact table.

[0072] If a data point in a dimension table appears twice or more in the corresponding fact table, then the data duplication relationship between the dimension table and the fact table can be denoted as 1:T. 1:T indicates that at least one data point in the dimension table appears repeatedly in the corresponding fact table.

[0073] If any data in a dimension table appears 1 or 0 times in its corresponding fact table, then the data duplication relationship between the dimension table and the fact table can be represented as 1:1. 1:T means that the data in the dimension table does not appear repeatedly in the corresponding fact table.

[0074] 108. Determine the target model, which is used to indicate the correspondence and data duplication relationships between the M1 dimension tables and the M2 fact tables.

[0075] The target model can be a star (constellation) model.

[0076] The following uses Tables 1 to 10 as examples to illustrate how... Figure 1 The embodiments shown are described below.

[0077] Table 1

[0078] hairstyle Long hair Zhongfa short hair

[0079] Table 2

[0080] gender male female unknown

[0081] Table 3

[0082] gender hairstyle male Long hair female Long hair unknown Long hair male Zhongfa female Zhongfa male short hair female short hair

[0083] Table 4

[0084] gender hairstyle address price male Long hair Beijing 100 female Zhongfa Shanghai 200 female short hair Shenzhen 100 male Zhongfa Beijing 50 male short hair Beijing 30

[0085] Table 5

[0086] year 2016 2017 2018

[0087] Table 6

[0088] year hairstyle address price 2016 short hair Beijing 30 2016 Zhongfa Beijing 40 2016 Long hair Beijing 50 2017 short hair Beijing 40 2017 Zhongfa Beijing 50 2017 Long hair Beijing 60

[0089] Table 7

[0090] year gender hairstyle address price 2016 male short hair Beijing 30 2017 female short hair Beijing 60 2018 unknown short hair Beijing 100

[0091] Table 8

[0092] gender hairstyle address year percentage male Long hair Beijing 2016 1 male Zhongfa Beijing 2016 75 male short hair Beijing 2016 24 male Long hair Beijing 2017 2 male Zhongfa Beijing 2017 75 male short hair Beijing 2017 23 male Long hair Beijing 2018 6 male Zhongfa Beijing 2018 74 male short hair Beijing 2018 20

[0093] Table 9

[0094] age under 3 years old 3-12 years old 13-18 years old Over 18 years old

[0095] Table 10

[0096] Shop Name Statistical quantity A-shape 85 B Style 112 Studio C 23

[0097] As shown in Tables 1 to 10, Table 1 is a hairstyle table, Table 2 is a gender table, Table 3 is a hairstyle gender table, Table 4 is a styling price table for various regions, Table 5 is an annual information table to be surveyed, Table 6 is a short hairstyle price table for Beijing in various years, Table 7 is a short hairstyle price table for different genders in Beijing in various years, Table 8 is a table of the percentage of men in Beijing using different hairstyles in various years, Table 9 is an age statistics table, and Table 10 is a table of shop names and sample sizes.

[0098] By analyzing Tables 1 to 10, we can determine the data lineage of Tables 1 to 10. Based on the data lineage of Tables 1 to 10, we can confirm that Tables 1 to 8 are related, while Tables 9 and 10 are not related, nor are they related to Tables 1 to 8.

[0099] Specifically, the table-level kinship relationships in Tables 1 to 8 are as follows:

[0100] Table 1, Table 2 -> Table 3;

[0101] Table 2, Table 3 -> Table 4;

[0102] Table 4, Table 5 -> Table 6;

[0103] Table 6, Table 2 -> Table 7;

[0104] Table 3, Table 5 -> Table 8.

[0105] Table 1, Table 2 -> Table 3 means that Table 1 and Table 2 are the source tables of Table 3. In other words, Table 3 is the destination table of Table 1 and Table 2.

[0106] As can be seen from the above relationships, some tables in Tables 1 to 8 can only be source tables, such as Table 1; some tables can only be destination tables, such as Tables 7 and 8; and some tables can be both source and destination tables, such as Tables 3 and 4.

[0107] Figures 2 to 4 It is a bloodline diagram derived from the above relationships.

[0108] By analyzing the lineage relationships in Tables 1 to 8, we can identify the following tables that can serve as source tables (i.e., tables with attributes 1 and 3): Table 1, Table 2, Table 3, Table 4, Table 5, and Table 6. Table 1 has only one destination table, Table 3; Table 2 has three destination tables, Table 3, Table 4, and Table 7; Table 3 has two destination tables, Table 4 and Table 5; Table 4 has one destination table, Table 6; Table 5 has two destination tables, Table 6 and Table 8; and Table 6 has one destination table, Table 7.

[0109] Therefore, Tables 2, 3, and 5 contain multiple source tables. Thus, Tables 2, 3, and 5 are the primary source tables.

[0110] By traversing the data in Table 2 and its three target tables, it can be determined that data in Table 2 appears multiple times in Tables 3 and 4. For example, the data "male" in Table 2 appears 3 times in Table 3 and 3 times in Table 4; the data "female" in Table 2 appears 3 times in Table 3 and 2 times in Table 4. Therefore, it can be determined that Table 2 is a dimension table.

[0111] Similarly, it can be determined that the data in Table 3 appears repeatedly in Tables 4 and 8, and the data in Table 5 appears repeatedly in Tables 6 and 8.

[0112] Therefore, it can be determined that Tables 2, 3, and 5 are dimension tables.

[0113] Excluding the dimension tables (Tables 2, 3, and 5) from Tables 1 to 8, the remaining tables are Tables 1, 4, 6, 7, and 8. Among Tables 1, 4, 6, 7, and 8, Table 1 has attribute 1, meaning it serves only as a source table; Table 8 has attribute 2, meaning it serves only as a destination table; and Tables 4, 6, and 7 have attribute 3, meaning they can serve as both source and destination tables. Therefore, the tables that can serve as destination tables are Tables 4, 6, 7, and 8. Thus, it can be determined that Tables 4, 6, 7, and 8 are fact tables.

[0114] Tables 4 and 7 are the target tables of Table 2 and are also fact tables; therefore, Tables 4 and 7 are fact tables corresponding to dimension table Table 2. Similarly, Tables 4 and 8 are the target tables of Table 3 and are also fact tables; therefore, Tables 4 and 8 are fact tables corresponding to Table 3. Tables 6 and 8 are the target tables of Table 5 and are also fact tables; therefore, Tables 6 and 8 are fact tables corresponding to Table 5.

[0115] As mentioned above, the data in Table 2 appears repeatedly in Table 4, therefore, the data repetition relationship between Table 2 and Table 4 is 1:T; the data in Table 2 appears in Table 7 but is not repeated, therefore, the data repetition relationship between Table 2 and Table 7 is 1:1.

[0116] The data in Table 3 appears repeatedly in Table 4, therefore the data repetition relationship between Table 3 and Table 4 is 1:T; the data in Table 3 appears repeatedly in Table 8, therefore the data repetition relationship between Table 3 and Table 8 is 1:T.

[0117] The data in Table 5 appears repeatedly in Table 6, therefore, the data repetition relationship between Table 5 and Table 6 is 1:T; the data in Table 5 appears repeatedly in Table 8, therefore, the data repetition relationship between Table 5 and Table 8 is 1:T.

[0118] Furthermore, although the data in Table 2 appears repeatedly in Table 3, both Table 2 and Table 3 are dimension tables. Therefore, the data duplication relationship between Table 2 and Table 3 can be disregarded.

[0119] Based on the correspondence between the dimension table and the fact table, as well as the data duplication relationship, we can obtain the following: Figure 5 Therefore, the star (constellation) model.

[0120] Figure 5 The star schema shown can conveniently reflect the relationship between the dimension tables and fact tables in Tables 1 to 8.

[0121] In some embodiments, a dimension table can be a subset of another dimension table. For example, table 3' is also a dimension table and table 3' is a subset of table 3.

[0122] Table 3'

[0123] gender hairstyle male Long hair female Long hair male Zhongfa female Zhongfa male short hair female short hair

[0124] Comparing Table 3 and Table 3', we can see that Table 3' contains all the content of Table 3. Therefore, Table 3' is a subset of Table 3. In other words, if there are two dimension tables, Table A and Table B, and a field in Table A contains a field in Table B (i.e., a field in Table A is greater than or equal to a field in Table B, such as Table 3 and Table 3'), then Table B is a subset of Table A. Furthermore, if a parent table outputs data to a child table, then the child table is a subset of the parent table.

[0125] If there is a subset of dimension tables, then the model obtained based on the relationship between the dimension tables and the fact tables can be a snowflake model.

[0126] In the above embodiments, source tables with only one destination table (i.e., Table 1) are excluded during the determination of dimension tables. In other embodiments, if data from a source table that has only one destination table appears repeatedly in its destination table, then the source table can also be determined to be a dimension table. In still other embodiments, for source tables with only one destination table, the user can be prompted to manually determine whether the source table is a dimension table.

[0127] In the above embodiments, the fact table is determined by identifying any table other than the dimension tables that can serve as the destination table. In other embodiments, the fact table can also be determined by first excluding the dimension tables in Tables 1 to 8, then excluding tables that do not have a corresponding source table, and the remaining tables are the fact tables.

[0128] In the above embodiments, the relationship between the dimension table and the fact table is achieved through methods such as... Figure 5 The star schema shown illustrates this. In other embodiments, the relationship between dimension tables and fact tables can also be presented in other ways. For example, the relationship between dimension tables and fact tables can be presented in a table as shown in Table 11.

[0129] Table 11

[0130]

[0131] For example, the relationship between dimension tables and fact tables can also be represented in the following form:

[0132] {[Table 2, Table 4], 1: T;

[0133] [Table 2, Table 7], 1:1;

[0134] [Table 3, Table 4], 1:T;

[0135] [Table 3, Table 8], 1:T;

[0136] [Table 5, Table 6], 1:T;

[0137] [Table 5, Table 8], 1:T}.

[0138] [Table 2, Table 4], 1:T indicates that Table 2 is a dimension table, Table 4 is a fact table corresponding to Table 2, and the data duplication relationship between Table 2 and Table 4 is 1:T; [Table 2, Table 7], 1:1 indicates that Table 2 is a dimension table, Table 7 is a fact table corresponding to Table 2, and the data duplication relationship between Table 2 and Table 7 is 1:1, and so on.

[0139] Optionally, in other embodiments, the association between the dimension table and the fact table may not include, for example, Figure 5 The data shown in Table 11 does not repeat the relationships shown in Table 11, but only reflects the correspondence between the dimension table and the fact table.

[0140] Optionally, in some embodiments, the data repetition relationship 1:T between the dimension table and the fact table reflects how many times the data in the dimension table is repeated in the fact table. In other embodiments, T can also be represented by a specific numerical value, which can be the number of times the data with the most repetition in the dimension table is repeated in the fact table, or it can be the average of all data with repetition.

[0141] Figure 6 This is a computer device provided according to an embodiment of this application. For example... Figure 6 The computer device 600 shown includes an acquisition unit 601 and a processing unit 602.

[0142] The acquisition unit 601 is used to acquire N tables, each of which contains at least one piece of data, where N is a positive integer greater than or equal to 2.

[0143] Processing unit 602 is used to analyze the data lineage of the N tables.

[0144] The processing unit 602 is further configured to determine M tables with a blood relationship from the N tables based on the data lineage of the N tables, where M is a positive integer greater than or equal to 2 and less than or equal to N.

[0145] The processing unit 602 is further configured to determine the association relationship between the M tables based on the data lineage of the M tables and the data in the M tables.

[0146] Optionally, in some embodiments, the processing unit 602 is specifically configured to determine M1 dimension tables and M2 fact tables from the M tables based on the data lineage of the M tables and the data in the M tables, wherein M1 and M2 are positive integers greater than or equal to 1, and the sum of M1 and M2 is less than or equal to M; and determine the association relationship between the M tables based on the M1 dimension tables and the M2 fact tables.

[0147] Optionally, in some embodiments, the processing unit 602 is specifically configured to: determine at least one first source table based on the data lineage of the M tables, wherein the first source table is a source table that has a lineage relationship with the multiple destination tables, and M is a positive integer greater than or equal to 2 and less than or equal to N; determine the M1 dimension tables from the at least one first source table based on the data of the at least one first source table and the data of the destination tables of the at least one first source table; and determine the M2 fact tables from the M3 tables other than the M1 dimension tables, wherein M3 is a positive integer greater than or equal to 1, and the sum of M3 and M1 is M.

[0148] Optionally, in some embodiments, the processing unit 602 is specifically configured to determine the number of times data in each of the at least one first source tables is repeated in each destination table of each of the at least one first source tables; and to determine the M1 dimension tables from the at least one first source table based on the determined number of repetitions, wherein at least one piece of data in each of the M1 dimension tables is repeated more than a preset threshold in at least one destination table of each dimension table.

[0149] Optionally, in some embodiments, the processing unit 602 is specifically used to determine that the target table in the M3 tables is the fact table.

[0150] Optionally, in some embodiments, the processing unit 602 is specifically used to determine K corresponding to the j-th dimension table in the M1 dimension tables. j A fact table, in which K j Each fact table in the fact tables includes at least one piece of data from the j-th dimension table, j = 1, ..., M1, K j It is a positive integer greater than or equal to 1 and less than or equal to M2.

[0151] Optionally, in some embodiments, the processing unit 602 is specifically configured to, according to the K j Each fact table in the fact table includes the number of times the data in the j-th dimension table is repeated, determining the K. j The data duplication relationship between the M1 fact tables and the j-th dimension table is determined; the target model is used to indicate the correspondence and data duplication relationship between the M1 dimension tables and the M2 fact tables.

[0152] Figure 7This is a schematic structural block diagram of a computer device according to an embodiment of this application. The computer device 700 includes a bus 701, a processor 702, a communication interface 703, and a memory 704. The processor 702, the memory 704, and the communication interface 703 communicate via the bus 701. The processor 702 can be a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system-on-chip (SoC), a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or other integrated chips. The memory 704 stores executable code included in the pathological image processing system. The processor 702 reads the executable code from the memory 704 to execute... Figure 1 The method is shown. The memory 704 may also include other software modules required for running processes, such as an operating system. The operating system can be LINUX. TM UNIX TM WINDOWS TM wait.

[0153] The computer devices (e.g., computer device 600, computer device 700) in the embodiments of this application can be laptops, desktop computers, tablets, etc., or servers.

[0154] This application also provides a chip system, including: a logic circuit, the logic circuit being coupled to an input / output interface, through which data is transmitted to perform actions such as... Figure 1 The method described.

[0155] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0156] It should be noted that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method embodiments can be completed by the integrated logic circuitry in the processor's hardware or by instructions in software form. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0157] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0158] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute... Figure 1 The method of the illustrated embodiment.

[0159] According to the method provided in the embodiments of this application, this application also provides a computer-readable medium storing program code, which, when run on a computer, causes the computer to perform... Figure 1 The method of the illustrated embodiment.

[0160] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0161] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0162] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0163] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0164] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0165] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Determine the data lineage of N tables, where each of the N tables contains at least one piece of data, and N is a positive integer greater than or equal to 2; Based on the bloodline data in the N tables, determine M tables that have bloodline relationships from the N tables, where M is a positive integer greater than or equal to 2 and less than or equal to N; Based on the data lineage of the M tables, at least one first source table is determined. The first source table is a source table that has a lineage relationship with multiple destination tables, and M is a positive integer greater than or equal to 2 and less than or equal to N. Determine the number of times data from each of the at least one first source tables is repeated in each destination table of each first source table; Based on the determined number of repetitions, M1 dimension tables are determined from the at least one first source table, wherein at least one piece of data in each of the M1 dimension tables has a repetition count greater than a preset threshold in at least one destination table of each dimension table. M2 fact tables are determined from M3 tables other than the M1 dimension tables, wherein the M2 fact tables are the target tables in the M3 tables, and M1 and M2 are positive integers greater than or equal to 1, and the sum of M1 and M2 is less than or equal to M; Based on the M1 dimension tables and the M2 fact tables, determine the relationships between the M tables.

2. The method as described in claim 1, characterized in that, The step of determining the relationships between the M1 dimension tables and the M2 fact tables includes: Determine K corresponding to the j-th dimension table in the M1 dimension tables. j A fact table, wherein K j Each of the fact tables includes at least one piece of data from the j-th dimension table, j = 1, ..., M1, K j It is a positive integer greater than or equal to 1 and less than or equal to M2.

3. The method as described in claim 2, characterized in that, The step of determining the relationships between the M1 dimension tables and the M2 fact tables further includes: According to the K j Each fact table in the fact tables includes the number of repetitions of data from the j-th dimension table, determining the K. j Data duplication relationships between the fact table and the j-th dimension table; Determine the target model, which is used to indicate the correspondence and data duplication relationships between the M1 dimension tables and the M2 fact tables.

4. A computer device, characterized in that, The computer device includes: The acquisition unit is used to acquire N tables, each of which includes at least one piece of data, where N is a positive integer greater than or equal to 2; The processing unit is used to analyze the data lineage of the N tables; The processing unit is further configured to determine M tables with a blood relationship from the N tables based on the data lineage of the N tables, where M is a positive integer greater than or equal to 2 and less than or equal to N; The processing unit is further configured to: Based on the data lineage of the M tables, at least one first source table is determined. The first source table is a source table that has a lineage relationship with multiple destination tables, and M is a positive integer greater than or equal to 2 and less than or equal to N. Determine the number of times data from each of the at least one first source tables is repeated in each destination table of each first source table; Based on the determined number of repetitions, M1 dimension tables are determined from the at least one first source table, wherein at least one piece of data in each of the M1 dimension tables has a repetition count greater than a preset threshold in at least one destination table of each dimension table. M2 fact tables are determined from M3 tables other than the M1 dimension tables, wherein the M2 fact tables are the target tables in the M3 tables, and M1 and M2 are positive integers greater than or equal to 1, and the sum of M1 and M2 is less than or equal to M; Based on the M1 dimension tables and the M2 fact tables, determine the relationships between the M tables.

5. The computer device as described in claim 4, characterized in that, The processing unit is specifically used to determine K corresponding to the j-th dimension table in the M1 dimension tables. j There are Kj fact tables, wherein each of the Kj fact tables includes at least one piece of data from the j-th dimension table, j = 1, ..., M1, K. j It is a positive integer greater than or equal to 1 and less than or equal to M2.

6. The computer device as described in claim 5, characterized in that, The processing unit is specifically used to process the K j Each fact table in the fact tables includes the number of repetitions of data from the j-th dimension table, determining the K. j Data duplication relationships between the fact table and the j-th dimension table; Determine the target model, which is used to indicate the correspondence and data duplication relationships between the M1 dimension tables and the M2 fact tables.

7. A computer device, characterized in that, include: A processor configured to be coupled to a memory, read and execute instructions and / or program code in the memory to perform the method as described in any one of claims 1-3.

8. A chip system, characterized in that, include: A logic circuit for coupling with an input / output interface, through which data is transmitted to perform the method as described in any one of claims 1-3.

9. A computer-readable medium, characterized in that, The computer-readable medium stores program code that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-3.

Images