Data table generation method and device, equipment, storage medium and program product

Abstract

The invention discloses a data table generation method and device, equipment, a storage medium and a program product, which are used for efficiently and accurately generating a data table. The data table generation method comprises the steps of determining first configuration information of a wide table field based on first metadata information of a plurality of data source tables; aiming at each data source table, based on a plurality of preset primary key screening conditions, splitting the data source table into a plurality of first data tables; the structures of the multiple first data tables are the same, and each first data table corresponds to a primary key screening condition; based on the first configuration information and the multiple first data tables, multiple second data tables are generated, each second data table comprises the wide table field, and primary key screening conditions corresponding to the first data tables belonging to the same second data table are the same; and combining the plurality of second data tables to obtain a target data table.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data table generation method, device, equipment, storage medium and program product. Background Art

[0002] In the process of business processing, it is usually necessary to splice multiple data source tables into a data table with larger data volume. Due to the large scale of data source tables, there is an urgent need for a solution to generate data tables efficiently and accurately. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a data table generation method, device, equipment, storage medium and program product for efficiently and accurately generating a data table.

[0004] In order to achieve the above objectives, the present application embodiment adopts the following technical solutions:

[0005] In a first aspect, an embodiment of the present application provides a method for generating a data table, comprising:

[0006] Determine first configuration information of a wide table field based on first metadata information of a plurality of data source tables;

[0007] For each of the data source tables, based on a plurality of preset primary key screening conditions, the data source table is split into a plurality of first data tables; the plurality of first data tables have the same structure, and each of the first data tables corresponds to a primary key screening condition;

[0008] Based on the first configuration information and the plurality of the first data tables, a plurality of second data tables, each of the second data tables including the wide table field, and the primary key screening conditions corresponding to the first data tables in the same second data table are the same;

[0009] The multiple second data tables are merged to obtain a target data table.

[0010] In a second aspect, an embodiment of the present application provides a data table generating device, including:

[0011] A determination module, configured to determine first configuration information of a wide table field based on first metadata information of a plurality of data source tables;

[0012] a splitting module, for splitting each of the data source tables into a plurality of first data tables based on a plurality of preset primary key screening conditions; the plurality of first data tables have the same structure, and each of the first data tables corresponds to a primary key screening condition;

[0013] A connection module, configured to, based on the first configuration information and a plurality of the first data tables, a plurality of second data tables, each of the second data tables including the wide table field, and the first data tables belonging to the same second data table having the same primary key screening condition;

[0014] The merging module is used to merge the multiple second data tables to obtain a target data table.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0016] processor;

[0017] a memory for storing instructions executable by the processor;

[0018] The processor is configured to execute the instructions to implement the data table generating method provided in the first aspect.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the data table generation method provided in the first aspect.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps in the data table generation method provided in the first aspect.

[0021] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0022] The data table is generated by adopting the technical route of first vertical splitting, then horizontal connection, and finally vertical merging. First, for each data source table, based on multiple preset primary key screening conditions, the data source table is split into multiple first data tables, each first data table corresponds to a primary key screening condition, and the vertical splitting of each data source table is realized, ensuring that the first data tables split from the same data source have the same structure and the same structure as the data source table. Then, based on the first configuration information of the wide table field and multiple first data tables, multiple second data tables are generated, each second data table contains a wide table field, and the primary key screening condition corresponding to the first data table in the same second data table, thereby realizing horizontal connection. In this way, only part of the data of the data source table needs to be scanned during the horizontal connection process, rather than the full amount of data, which helps to improve the efficiency of data table generation; secondly, since each second data packet contains a wide table field, all second data tables have the same structure and have the same structure as the data table to be generated; on this basis, by combining multiple second data tables, vertical merging is realized, and the target data table can be generated quickly and accurately. In addition, based on the first metadata information of multiple data source tables, the first configuration information of the wide table field is determined, so that the first configuration information can automatically adapt to the field changes of the data source table, and each second data table obtained based on the first configuration information can also automatically adapt to the field changes of the data source table, and finally the target data table generated based on multiple second data tables can also automatically adapt to such field changes. In this way, there is no need for manual intervention to modify the code, which further improves the efficiency and accuracy of data table generation and improves the system processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] Figure 1 A schematic diagram of a data table generation process provided for an embodiment of the present application;

[0025] Figure 2 A flowchart of a method for generating a data table provided in accordance with an embodiment of the present application;

[0026] Figure 3 A schematic diagram of a primary key mapping method provided for an embodiment of the present application;

[0027] Figure 4 A schematic diagram of a generation process of a connection operation statement provided for an embodiment of the present application;

[0028] Figure 5 A schematic diagram of the structure of a data table generating device provided in one embodiment of the present application;

[0029] Figure 6 A schematic diagram of the structure of an electronic device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0031] The terms "first", "second", etc. in this specification and claims are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in this specification and claims means at least one of the connected objects, and the character " / " generally means that the objects associated with each other are in an "or" relationship.

[0032] Related concept description:

[0033] Data model: also called table, is an abstraction of data features. It describes the static features, dynamic behaviors and constraints of the system from an abstract level, and provides an abstract framework for information representation and operation of the database system. The content described by the data model consists of three parts: data structure, data operation and data constraints.

[0034] Hash algorithm: generally translated as hash, hash, or transliterated as hash, is to transform an input of arbitrary length (also called pre-image) into an output of fixed length through a hash algorithm, and the output is the hash value. This conversion is a compression mapping, that is, the space of hash values ​​is usually much smaller than the space of inputs, and different inputs may hash to the same output, so it is impossible to determine the unique input value from the hash value. Simply put, it is a function that compresses a message of arbitrary length into a message digest of a fixed length. The hash algorithm can convert a data into a mark, and this mark has a very close relationship with every byte of the source data.

[0035] Modulo operation: Modulo operation is to find the remainder after dividing two numbers. It is also called modular operation or remainder operation. It is one of the commonly used mathematical operations in mathematics and computer science. In mathematics, modulo operation refers to the remainder obtained by dividing one number by another number; in computer science, modulo operation refers to the remainder obtained by division operation. The symbol "%" is usually used to represent modulo operation, such as: 5%3=2.

[0036] Metadata: Metadata is "data about data", that is, data that describes data and exists to describe relevant information about the data.

[0037] The applicant has studied the data table generation process and found that a data table can be generated by sequentially connecting multiple data source tables. For example, if a data table is to be generated based on n data source tables, table1 to tablen, table2 is first left-connected with table1 to obtain the first data table 1; then, table3 is left-connected with the first data 1 to obtain the first data 2; and so on, until all data source tables are connected to obtain the final data table table.

[0038] However, in this way, each connection operation needs to scan the entire data source table, which will greatly affect the efficiency of data table generation. In addition, when the fields of the data source table change, manual intervention is required to modify the code to make the generated data wide table adapt to such field changes, which will not only further reduce the efficiency of data table generation, but also affect the accuracy of data table generation, resulting in poor processing performance of the entire system.

[0039] In view of this, the embodiment of the present application proposes a data table generation scheme, which adopts the technical route of first vertical splitting, then horizontal connection, and finally vertical merging to generate a data table. First, for each data source table, based on multiple preset primary key screening conditions, the data source table is split into multiple first data tables, each first data table corresponds to a primary key screening condition, and the vertical splitting of each data source table is realized, ensuring that the first data table split from the same data source has the same structure and the same structure as the data source table. Then, based on the first configuration information of the wide table field and multiple first data tables, multiple second data tables are generated, each second data table contains a wide table field, and the primary key screening condition corresponding to the first data table in the same second data table, thereby realizing horizontal connection. In this way, only part of the data of the data source table needs to be scanned during the horizontal connection process, rather than the full amount of data, which helps to improve the efficiency of data table generation; secondly, since each second data packet contains a wide table field, all second data tables have the same structure and have the same structure as the data table to be generated; on this basis, by combining multiple second data tables, vertical merging is realized, and the target data table can be generated quickly and accurately. In addition, based on the first metadata information of multiple data source tables, the first configuration information of the wide table field is determined, so that the first configuration information can automatically adapt to the field changes of the data source table, and each second data table obtained based on the first configuration information can also automatically adapt to the field changes of the data source table, and finally the target data table generated based on multiple second data tables can also automatically adapt to such field changes. In this way, there is no need for manual intervention to modify the code, which further improves the efficiency and accuracy of data table generation and improves the system processing performance.

[0040] For example, Figure 1 As shown, taking multiple data source tables table0 to table3 as an example, data source table table0 is vertically split into first data tables table0_0 to table0_4, data source table table1 is vertically split into first data tables table1_0 to table1_4, data source table table2 is vertically split into first data tables table2_0 to table2_4, and data source table table3 is vertically split into first data tables table3_0 to table3_4. Among them, the first data tables table0_0 to table3_0 correspond to the same primary key screening condition, the first data tables table0_1 to table3_1 correspond to the same primary key screening condition, the first data tables table0_2 to table3_2 correspond to the same primary key screening condition, the first data tables table0_3 to table3_3 correspond to the same primary key screening condition, and the first data tables table0_4 to table3_4 correspond to the same primary key screening condition.

[0041] Further, based on the first configuration information of the wide table field, a connection operation is performed on multiple first data tables table0_0~table0_4 to obtain a second data table wide_view0; similarly, a connection operation is performed on multiple first data tables table0_1~table3_1 to obtain a second data table wide_view1; a connection operation is performed on multiple first data tables table0_2~table3_2 to obtain a second data table wide_view2; a connection operation is performed on multiple first data tables table0_3~table3_3 to obtain a second data table wide_view3; a connection operation is performed on multiple first data tables table0_4~table3_4 to obtain a second data table wide_view4.

[0042] Finally, all the second data tables wide_view0 to wide_view4 are vertically merged to obtain the target data table wide_table.

[0043] It should be understood that the data table generation method provided in the embodiment of the present application can be executed by an electronic device. The electronic device referred to here may include a terminal device, such as a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device, a smart home appliance, a smart watch, a vehicle terminal, an aircraft, etc.; or, the electronic device may also include a server, such as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The electronic device is independent of the backend server.

[0044] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0045] Please refer to Figure 2 , is a flow chart of a data table generation method provided by an embodiment of the present application, the method comprising the following steps:

[0046] S202: Determine first configuration information of a wide table field based on first metadata information of a plurality of data source tables.

[0047] A data source table refers to a data table obtained from a data source. Each data source table has corresponding first metadata information. The first metadata information of a data source table includes metadata of the data source table, which is used to describe the structure of the data source table, such as but not limited to field information of the data source table, table name of the data source table, etc. Among them, the field information of the data source table is used to describe the properties of the fields contained in the data source table, such as but not limited to the field name, type and order of the fields contained in the data source table.

[0048] In an embodiment of the present application, the first metadata information of each data source table can be pre-configured by relevant personnel, or can be obtained by periodically retrieving metadata from the data source table. Exemplarily, multiple data source tables are traversed through a python script, and a metadata retrieval statement such as "showcolumns from table[i]" is executed on the currently traversed data source table using the Spark engine to obtain the field information of the data source table, where table[i] is the table name of the traversed data source table; then the field information and table name of the data source table are used as the first metadata information of the data source table. In this way, it is possible to adapt to the dynamic changes of the data source table, ensure the accuracy of the first metadata information, and provide reliable data support for the subsequent generation of data tables.

[0049] The wide table field refers to the field included in the data table to be generated. The first configuration information of the wide table field is used to describe the attributes of the wide table field, such as but not limited to the field name, type, order, etc. of the wide table field, which is not limited in the embodiment of the present application.

[0050] In an embodiment of the present application, the first configuration information can be pre-configured by relevant personnel, or it can be continuously updated based on the first metadata information of each data source table, so as to automatically sense and adapt to field changes in the data source table, so that the second data table generated thereby and the target data table finally generated can adapt to such field changes.

[0051] Exemplarily, the above S202 includes the following steps: based on the first metadata information, determining the field information of each data source table; comparing the field information with the second configuration information of the wide table field to obtain field difference information, wherein the second configuration information is the configuration information of the wide table field obtained before determining the first configuration information of the wide table field; updating the second configuration information based on the field difference information to obtain the first configuration information.

[0052] In practical applications, the steps included in the above S202 can be executed periodically.

[0053] As an example, the method for updating the second configuration information includes: determining an update keyword that matches the second configuration information based on field difference information; generating an update statement based on the field difference information and the update keyword, the update statement being used to update the second configuration information; and executing the update statement to obtain the first configuration information.

[0054] For example, the update keyword may be a keyword used in SQL to modify table fields, such as but not limited to: alter table add columns, alter table replace columns, etc. By comparing the field information of each data source table with the second configuration information, the field with differences is determined; if the field exists in the data source table but does not exist in the second configuration information, the field is determined to be a newly added field of the data source table, and the update keyword is determined to be alter table add columns; further, the table in the update keyword is replaced with the identifier of the second configuration information (that is, the table name of the data table to be generated), and the columns in the update keyword is replaced with the field name of the field, so that the update statement can be obtained, and the first configuration information containing the newly added field can be obtained by executing the update statement.

[0055] If the field does not exist in the data source table but exists in the second configuration information, the field is determined to be a deleted field of the data source table, and the update keyword is determined to be alter table replace columns; further, the table in the update keyword is replaced with the identifier of the second configuration information (that is, the table name of the target data table to be generated), and the columns in the update keyword is replaced with the field name of the field, and an update statement can be obtained. By executing the update statement, the first configuration information that does not contain the deleted field can be obtained.

[0056] It is worth noting that the above method for updating the second configuration information can be implemented by calling spark.

[0057] S204: For each data source table, based on a plurality of preset primary key screening conditions, split the data source table into a plurality of first data tables.

[0058] The structures of the multiple first data tables are the same, and each first data table corresponds to a primary key screening condition.

[0059] In this way, each first data table is obtained by vertically splitting the data source table to which it belongs. Each first data table has the same structure as the data source table to which it belongs. Specifically, the fields contained in each first data table are the same as the fields contained in the data source table to which it belongs. For example, if the data source table includes a primary key user_id and multiple non-primary key fields var1 to var20, each first data table obtained by dividing the data source table also includes a primary key user_id and multiple non-primary key fields var1 to var20.

[0060] In the above S204, the above primary key screening condition can be set according to actual needs.

[0061] As an example, each primary key screening condition includes the number of digits of a primary key. For example, the data source table includes 100 data, and five first data tables are obtained by vertically dividing the data source, wherein the first first data table includes data with primary key values ​​in the data source table at the 1st to 20th positions, the second first data table includes data with primary key values ​​in the data source table at the 21st to 40th positions, the third first data table includes data with primary key values ​​in the data source table at the 41st to 60th positions, the fourth first data table includes data with primary key values ​​in the data source table at the 61st to 80th positions, and the fifth first data table includes data with primary key values ​​in the data source table at the 81st to 100th positions.

[0062] As another example, each primary key screening condition includes an integer obtained by mapping the primary key value. For example, assuming that five first data tables are to be split from each data source table, the following five primary key screening conditions are pre-set:

[0063] Primary key screening condition 1: The integer obtained by mapping the primary key value is 0;

[0064] Primary key screening condition 2: The integer obtained by mapping the primary key value is 1;

[0065] Primary key screening condition 3: The integer obtained by mapping the primary key value is 2;

[0066] Primary key screening condition 4: The integer obtained by mapping the primary key value is 3;

[0067] Primary key screening condition 5: The integer obtained by mapping the primary key value is 4.

[0068] Based on the above primary key filtering conditions, for each data source table, the primary key value of each row of data in the data source table is mapped to obtain an integer corresponding to each row of data; then, based on the integer corresponding to each row of data, the primary key filtering conditions satisfied by each row of data are determined; further, the data corresponding to the same primary key filtering conditions are divided into the same group to obtain multiple groups of data; finally, a corresponding first data table is generated based on each group of data, that is, each group of data corresponds to a first data table.

[0069] In the above implementation, the method used to map the primary key value of each row of data can be determined based on the type of the primary key. Figure 3 As shown, if the type of the primary key is a numeric type (such as an integer, a floating point tree type, a Boolean value type, etc.), mapping the primary key value of each row of data includes: performing a modulo operation on the primary key value; if the type of the primary key is a character type (such as a string type), mapping the primary key value of each row of data includes: performing a hash operation on the primary key value to obtain a hash value for each row of data, and then performing a modulo operation on the hash value for each row of data.

[0070] In actual applications, the type of the primary key can be determined using the Hive built-in function CAST. For example, assuming the primary key is user_id, if the result returned by the function CAST(user_id as DOUBLE) is not empty, the type of user_id is numeric, otherwise the type of user_id is character. Modulo operations can be implemented using the pmod function in the Structured Query Language (SQL). For example, if there are 5 primary key screening conditions, 5 is used as the dividend, and a modulo operation is performed using the SQL statement "pmod(user_id,5)".

[0071] Through the above implementation, each data source table is divided into the same number of first data tables.

[0072] In order to improve the overall computing performance, for each data source table, in the above S204, based on the amount of data in the data source table, the splitting method of the data source table is determined; based on the splitting method and primary key of the data source table, the data source table is split into multiple first data tables.

[0073] In the embodiment of the present application, the above splitting methods include physical splitting and logical splitting. Among them, physical splitting refers to filtering data from the data source table and then storing it in the database to generate a new data table, and the new data table is the first data table. Logical splitting refers to filtering data from the data source table by filtering method and directly using it for subsequent calculations without storing it in the database to generate a new data table. In the logical splitting method, each time the data is filtered, it is a logical table splitting process.

[0074] Specifically, for each data source table, if the amount of data in the data source table exceeds a preset order of magnitude, for example, the number of rows in the data source table exceeds a preset number of rows or the number of columns exceeds a preset number of columns, then it is determined that the splitting method of the data table is physical splitting; further, for each primary key filtering condition, based on the primary key of the data source table, the data that meets the primary key filtering condition is filtered out, and a new data table is generated after these data are stored in the database, thereby obtaining the first data table corresponding to the primary key filtering condition.

[0075] It is understandable that when the amount of data in the data source table is large, the subsequent calculation complexity is usually large. By physically splitting the data source table into separate tables, the overall calculation performance can be improved.

[0076] For each data source table, if the amount of data in the data source table does not exceed a preset order of magnitude, for example, the number of rows in the data source table does not exceed the preset number of rows and the number of columns is not longer than the preset number of columns, then it is determined that the splitting method of the data table is logical splitting; further, for each primary key filtering condition, based on the primary key of the data source table, the data that meets the primary key filtering condition is filtered out as the data in the first data table corresponding to the primary key filtering condition for subsequent calculation.

[0077] In actual applications, under the physical split method, the data that is stored in the database can be stored in a secondary partitioning manner. Specifically, the condition identifier of the primary key screening condition is used as the value of the secondary partition partition, and the data in the data source table is stored in the distributed file system (Hadoop Distributed File System, HDFS) according to the primary and secondary partition blocks. The fields based on each level of partition can be selected according to actual needs, and this embodiment of the application does not limit this.

[0078] It is understandable that when the amount of data in the data source table is not large, the subsequent calculation complexity is usually small. By splitting the data source table into separate tables through logical splitting, redundant storage caused by data expansion can be saved, and the amount of data scanned in subsequent calculations can also be controlled, thereby improving the overall computing performance.

[0079] S206: Generate multiple second data tables based on the first configuration information and the multiple first data tables.

[0080] Each second data table includes wide table fields. The primary key screening conditions corresponding to the first data tables in the same second data table are the same.

[0081] In the above S206, multiple second data tables can be generated in various appropriate ways, which can be selected according to actual needs, and the embodiment of the present application is not limited to this.

[0082] In one embodiment, the above S206 includes the following steps: S261a, based on multiple primary key filtering conditions, multiple first data tables are grouped to obtain multiple task groups, and the primary key filtering conditions corresponding to the first data tables in the same task group are the same; S262a, for each task group, based on the primary key and first configuration information of the first data table in the task group, each first data table in the task group is connected in turn.

[0083] For example, Figure 1Taking the first data table shown as an example, 5 task groups can be obtained through the above method. The first task group includes multiple first data tables table0_0~table3_0, the second task group includes multiple first data tables table0_1~table3_1, the third task group includes multiple first data tables table0_2~table3_2, the fourth task group includes multiple first data tables table0_3~table3_3, and the fifth task group includes multiple first data tables table0_4~table3_4.

[0084] Taking the first task group as an example, the same primary key value is used as a connection condition, and a connection operation is performed on multiple first data tables table0_0 to table3_0 in sequence to obtain the second data table wide_view0.

[0085] Since the structure of the first data table in the same data source table is the same, the connection operation for each task group is similar, and the only difference is the table name of the first data table and the table name of the second data table to be generated. Therefore, the connection operation can be performed on all task groups in parallel, thereby greatly improving the efficiency of the connection operation.

[0086] In another embodiment, the above S206 includes the following steps:

[0087] S261b, grouping the first data tables based on multiple primary key screening conditions to obtain multiple task groups.

[0088] The primary key screening conditions corresponding to the first data tables belonging to the same task group are the same.

[0089] S262b: For each task group, based on the primary key of the first data table in the task group, the first configuration information, and the connection statement template, generate a connection operation statement for the task group.

[0090] The connection operation statement is used to perform a connection operation on the first data table in the task group.

[0091] S263b, execute the connection operation statement of each task group to obtain the second data table corresponding to each task group.

[0092] The connection operation statement is used to perform a connection operation on the task group. The connection statement template refers to a template used to generate a connection operation statement, such as an SQL statement template. The connection operation statement can be obtained by filling the primary key and the first configuration information of the first data table in the task group into the corresponding positions of the connection statement template.

[0093] As an example, in order to ensure that the generated connection operation statement can efficiently and accurately perform connection operations on the task group, the above S262b includes the following steps: Step A1, based on the field information of the data source table to which the first data table in the task group belongs, determine the correspondence between the wide table field and the first data table in the task group; Step A2, based on the correspondence, the primary key of the first data table in the task group and the connection statement template, generate the connection operation statement of the task group.

[0094] In the above step A1, the corresponding relationship is used to indicate from which first data table the field value of the wide table field is obtained during the connection operation. Since each first data table has the same structure as the data source table to which it belongs, the fields contained in each first data table are the same as the fields contained in the data source table to which it belongs. In this case, by matching the fields contained in each data source table with the wide table fields, the data source table to which the wide table field belongs can be determined; further, a corresponding relationship between the first data table and the wide table field in the data source table is established.

[0095] For example, continue with Figure 1 Taking the data source table shown in the figure as an example, assuming that the wide table fields include var1 to var10, the fields contained in each data source table are as follows:

[0096] table0: var1~var3

[0097] table1: var4~var5

[0098] table2: var6~var8

[0099] table3: var9~var10

[0100] For the first task group {table0_0, table_0, table2_0, table3_0}, the following correspondence can be determined: {table0_0, var1~var3}, {table1_0, var4~var5}, {table2_0, var6~var8}, {table3_0, var9~var10}. For the remaining four task groups, the correspondence between the first data table and the wide table field in each task group can be determined in a similar manner.

[0101] In the above step A2, the connection statement template includes: the first selection keyword, the connection keyword, the condition keyword, and the starting point keyword. Accordingly, the connection operation statement of the task group can be generated in the following manner:

[0102] Step A21: Generate a first selection clause of the task group based on the corresponding relationship and the first selection keyword.

[0103] The first selection clause is used to obtain the field value of the wide table field from the first data table included in the task group. The first selection keyword can be the keyword SELECT in SQL, and the string after SELECT is used as the query result, which is consistent with the structure of the target data table to be generated. Among them, the first string after SELECT represents the primary key of the target data table, and from the second string onwards, the wide table fields and the first data table used to obtain the field value of the wide table fields are represented in sequence.

[0104] Exemplarily, in order to ensure that the primary key of the target data table is consistent with the primary key of each first data table in the task group, the COALESCE function can be used to globally collect the primary keys of all first data tables in the task group, and the return result of the COALESCE function is used as the primary key of the target data table.

[0105] For example, taking the first task group {table0_0, table1_0, table2_0, table3_0} and their corresponding relationships shown above as an example, where the primary key of each first data table is user_id, the first selection clause of the task group obtained in the above manner is:

[0106] SELECT COALESCE(table0_0.user_id,table1_0.user_id,table2_0.user_id,table3_0.user_id)as user_id,table0_0.var1~var3,table1_0.var4~var5,table2_0.var6~var8,table3_0.var9~var10

[0107] Step A22: determining a first candidate data table and a second candidate data table from the task group based on the order of the first data tables included in the task group.

[0108] The first candidate data table refers to the first data table as the starting point of the connection operation. The second candidate data table refers to the first data table to be connected based on the first candidate data table.

[0109] Step A23: Generate a start clause based on the table name of the first candidate data table and the start keyword.

[0110] The start clause is used to indicate that the first candidate data table is used as the starting point for the connection operation. The starting point keyword can be FROM in SQL, and the string after FROM is the table name of the first candidate data table.

[0111] Continuing with the first task group {table0_0,table1_0,table2_0,table3_0} shown above, table0_0 is the first candidate data table, and the starting clause of the task group is as follows:

[0112] FROM table0_0

[0113] Step A24: Generate a join clause of the second candidate data table based on the table name and the join keyword of the second candidate data table.

[0114] The join clause is used to perform a join operation on the second candidate data table. The join keyword can be the keyword FULL JOIN in SQL, and the string after FULL JOIN is the table name of the second candidate data table. Using FULL JOIN to generate a join clause based on the above SELECT can ensure that the primary keys of all first data tables in the task group are taken as a union during the join operation, and all fields of the first data table are written to the target data table, so that the join operation of the task group can be completed by scanning each first data table only once.

[0115] Continuing with the first task group {table0_0, table1_0, table2_0, table3_0} shown above as an example, table1_0, table2_0, table3_0 are the second candidate data tables, and the join clauses of these second candidate data tables are as follows:

[0116] FULL JOIN table1_0

[0117] FULL JOIN table2_0

[0118] FULL JOIN table3_0

[0119] Step A25 , generating a conditional clause of the second candidate data table based on the primary key of the second candidate data table, the table name of the first data table that is sequentially located before the second candidate data table, and the conditional keyword.

[0120] The conditional clause is used to indicate the condition for performing a join operation on the second candidate data table. The conditional keyword may be the keyword ON in SQL, and the string after ON indicates the condition.

[0121] For example, continuing with the first task group {table0_0, table1_0, table2_0, table3_0} shown above, table1_0, table2_0, table3_0 are the second candidate data tables, and the conditional clauses of these second candidate data tables are as follows:

[0122] ON table0_0.user_id=table1_0.user_id

[0123] ON COALESCE(table0_0.user_id,table1_0.user_id)=table2_0.user_id

[0124] ON COALESCE(table0_0.user_id,table1_0.user_id,table2_0.user_id)=table3_0.user_id

[0125] In practical applications, the starting clause of the task group, the connection clause of each second candidate data table, and the conditional clause can be generated by traversing the task group in sequence. Specifically, when traversing to the first first data table, the starting clause is generated based on the table name and the starting keyword of the first data table; thereafter, each time a first data table is traversed, the connection clause and conditional clause of the first data table are generated.

[0126] Step A26, generating a connection operation statement of the task group based on the first selection clause, the start clause, the connection clause of the second candidate data table and the conditional clause.

[0127] As an example, the first selection clause, the start clause, the connection clause of the second candidate data table and the conditional clause are concatenated to obtain the connection operation statement of the task group.

[0128] For example, continuing with the first task group {table0_0,table1_0,table2_0,table3_0} shown above, by concatenating the above clauses, the connection operation statement for the task group is as follows:

[0129] SELECT COALESCE(table0_0.user_id,table1_0.user_id,table2_0.user_id,table3_0.user_id)as user_id,table0_0.var1~var3,table1_0.var4~var5,table2_0.var6~var8,table3_0.var9~var10

[0130] FROM table0_0

[0131] FULL JOIN table1_0ON table0_0.user_id=table1_0.user_id

[0132] FULL JOIN table2_0COALESCE(table0_0.user_id,table1_0.user_id)=table2_0.user_id

[0133] FULL JOIN table3_0COALESCE(table0_0.user_id,table1_0.user_id,table2_0.user_id)=table3_0.user_id

[0134] In this way, the connection operation statements can be automatically generated and executed without manually writing a large amount of SQL code, thereby significantly improving the efficiency of the connection operation.

[0135] Optionally, the connection statement template may also include a second selection keyword. The above step A2 may also include: before the above step A25, based on the second selection keyword and the table name and splitting method of the data source table to which the first data table in the task group belongs, generate a second selection clause for the first data table in the task group. The second selection clause is used to obtain the first data table. Accordingly, in the above step A25, the first selection clause, the start clause, the connection clause and the conditional clause of the second candidate data table are combined to obtain a candidate connection statement; based on the splitting method of the data source table to which the first data table in the task group belongs, the second selection clause of the first data table in the task group and the candidate connection statement are combined to obtain the connection operation statement of the task group.

[0136] More specifically, if the data source table to which the first data table in the task group belongs is split in a logical splitting manner, the table name of the first data table is replaced with the connection operation statement of the first data table in the candidate connection statement to obtain the connection operation statement of the task group; if the data source table to which the first data table in the task group belongs is split in a physical splitting manner, the second selection clause of the first data table is spliced ​​with the candidate connection statement to obtain the connection operation statement of the task group.

[0137] For example, the second selection clause may include the keywords SELECT*FROM and WHERE in SQL. Taking the first task group {table0_0, table1_0, table2_0, table3_0} as an example, the second selection clause of each first data table in the task group is as follows:

[0138] SELECT*FROM table0 WHERE pmod(user_id,5)=0->table0_0

[0139] SELECT*FROM table1 WHERE pmod(user_id,5)=0->table1_0

[0140] SELECT*FROM table2 WHERE pmod(user_id,5)=0->table2_0

[0141] SELECT*FROM table3 WHERE pmod(user_id,5)=0->table3_0

[0142] Assume that the candidate join statements are as follows:

[0143] SELECT COALESCE(table0_0.user_id,table1_0.user_id,table2_0.user_id,table3_0.user_id)as user_id,table0_0.var1~var3,table1_0.var4~var5,table2_0.var6~var8,table3_0.var9~var10

[0144] FROM table0_0

[0145] FULL JOIN table1_0ON table0_0.user_id=table1_0.user_id

[0146] FULL JOIN table2_0COALESCE(table0_0.user_id,table1_0.user_id)=table2_0.user_id

[0147] FULL JOIN table3_0COALESCE(table0_0.user_id,table1_0.user_id,table2_0.user_id)=table3_0.user_id

[0148] If the splitting mode of the data source table to which the first data table table0_0 belongs is logical splitting, and the splitting modes of the data source tables to which the other first data tables belong are physical splitting, then replace table0_0 in the above candidate connection statement with the second selection clause of the first data table, and concatenate the second selection clauses of the other first data tables with the above candidate connection statement to obtain the following connection operation statement:

[0149] SELECT*FROM table1 WHERE pmod(user_id,5)=0->table1_0

[0150] SELECT*FROM table2 WHERE pmod(user_id,5)=0->table2_0

[0151] SELECT*FROM table3 WHERE pmod(user_id,5)=0->table3_0

[0152] SELECT COALESCE(table0_0.user_id,table1_0.user_id,table2_0.user_id,table3_0.user_id)as user_id,table0_0.var1~var3,table1_0.var4~var5,table2_0.var6~var8,table3_0.var9~var10

[0153] FROM table0_0

[0154] FULL JOIN(SELECT*FROM table0 WHERE pmod(user_id,5)=0->table0_0)ONtable0_0.user_id=table1_0.user_id

[0155] FULL JOIN table2_0COALESCE(table0_0.user_id,table1_0.user_id)=table2_0.user_id

[0156] FULL JOIN table3_0COALESCE(table0_0.user_id,table1_0.user_id,table2_0.user_id)=table3_0.user_id

[0157] In this way, based on the above connection operation statement, the entire process from filtering the first data table of the data source table to generating the second data table can be automatically executed, thereby significantly improving the connection operation efficiency. In addition, in the process of generating the connection operation statement, the predicate push-down technology is also introduced to replace the table name of the first data table in the logically split source data table with the second selection clause of the first data table. Therefore, when executing the connection operation on the first data table, it is completed as close as possible to the time when the first data table is read, which can greatly reduce the input data of the connection operation and improve the overall computing performance.

[0158] In practical applications, such as Figure 4 As shown, the above method for generating a connection operation statement can be abstracted into a class, which is executed by the instantiated object spark of SparkSession, and the spark.DataFrame.createOrReplaceTempView method is called to register the temporary data table view wide_view_{element} at the SparkSession level to obtain the second data table.

[0159] It is worth noting that, on the one hand, the above connection operation statement generation method does not actually write all SQL texts at the code level. Each iteration and each traversal constructs dynamic text characters through the Python scripting language and assigns them to a Python variable, and automatically builds and passes variables. The various clauses in the process and the final generated connection operation statement are all Python variables in the memory, so the amount of code is relatively small; on the other hand, the life cycle of the second data table is synchronized with the SparkSession session life cycle, that is, it is destroyed when the Spark program ends, does not occupy actual physical storage, and does not bring redundant storage costs.

[0160] In addition, in practical applications, Figure 4 As shown, the above class is instantiated into an object object, the connection operation for each task group is used as a calculation branch, the object operation connection statement generation method is called respectively, and the spark execution connection generation method is called to generate multiple second data tables. The above multiple branches can be executed in parallel.

[0161] S208: Merge the multiple second data tables to obtain a target data table.

[0162] Since each second data table has the same structure and contains wide table fields, the target data table can be quickly obtained by merging multiple second data tables.

[0163] In one embodiment, the above S208 includes the following steps: based on the table name and merge keyword of each second data table, generating a merge clause for each second data table, the merge clause is used to merge the corresponding second data tables; splicing the merge clauses of each second data table to obtain a merge operation statement, the merge operation statement is used to merge multiple second data tables; executing the merge operation statement to obtain the target data table.

[0164] For example, the merge keyword may be the SQL keyword UNION ALL. Figure 1 Taking the second data table shown as an example, the following merge operation statement can be generated:

[0165] SELECT*FROM wide_view0 UNION ALL

[0166] SELECT*FROM wide_view1 UNION ALL

[0167] SELECT*FROM wide_view2 UNION ALL

[0168] SELECT*FROM wide_view3 UNION ALL

[0169] SELECT * FROM wide_view4

[0170] Among them, the first line is the clause of the second data table wide_view0, the second line is the clause of the second data table wide_view1, the third line is the clause of the second data table wide_view2, the fourth line is the clause of the second data table wide_view3, and the fifth line is the clause of the second data table wide_view4.

[0171] Through the above implementation, the automatic construction and execution of the merge operation statement can be realized, which significantly improves the efficiency and accuracy of data table generation compared with the method of manually writing statements.

[0172] The embodiment of the present application shows some implementation methods of the above S208. Of course, it should be understood that the above S208 can also be implemented in other ways, and the embodiment of the present application does not limit this.

[0173] The data table generation method provided in the embodiment of the present application adopts the technical route of first vertical splitting, then horizontal connection, and finally vertical merging to generate a data table. First, for each data source table, based on multiple preset primary key screening conditions, the data source table is split into multiple first data tables, each first data table corresponds to a primary key screening condition, and the vertical splitting of each data source table is realized, ensuring that the first data table split from the same data source has the same structure and the same structure as the data source table. Then, based on the first configuration information of the wide table field and multiple first data tables, multiple second data tables are generated, each second data table contains a wide table field, and the primary key screening condition corresponding to the first data table in the same second data table, thereby realizing horizontal connection. In this way, only part of the data of the data source table needs to be scanned during the horizontal connection process, rather than the full amount of data, which helps to improve the efficiency of data table generation; secondly, since each second data packet contains a wide table field, all second data tables have the same structure and have the same structure as the data table to be generated; on this basis, by combining multiple second data tables, vertical merging is realized, and the target data table can be generated quickly and accurately. In addition, based on the first metadata information of multiple data source tables, the first configuration information of the wide table field is determined, so that the first configuration information can automatically adapt to the field changes of the data source table, and each second data table obtained based on the first configuration information can also automatically adapt to the field changes of the data source table, and finally the target data table generated based on multiple second data tables can also automatically adapt to such field changes. In this way, there is no need for manual intervention to modify the code, which further improves the efficiency and accuracy of data table generation and improves the system processing performance.

[0174] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0175] Figure 5 This is a schematic diagram of the structure of a data table generating device according to an embodiment of the present application. Figure 5 In a software implementation, the data table generating device 500 may include: a determination module 510, a splitting module 520, a connection module 530, and a merging module 540.

[0176] The determination module 510 is configured to determine first configuration information of a wide table field based on first metadata information of a plurality of data source tables.

[0177] The splitting module 520 is used to split the data source table into multiple first data tables based on multiple preset primary key screening conditions for each of the data source tables; the multiple first data tables have the same structure, and each of the first data tables corresponds to a primary key screening condition.

[0178] The connection module 530 is used to generate multiple second data tables based on the first configuration information and multiple first data tables, each of the second data tables contains the wide table field, and the primary key screening conditions corresponding to the first data tables in the same second data table are the same.

[0179] The merging module 540 is configured to merge the multiple second data tables to obtain a target data table.

[0180] In another embodiment, the splitting module is used to:

[0181] Determine a splitting method of the data source table based on the amount of data in the data source table;

[0182] Based on the splitting mode and primary key of the data source table, the data source table is split into multiple first data tables.

[0183] In another embodiment,

[0184] The determination module is used for:

[0185] Based on the first metadata information, determining the field information of each of the data source tables;

[0186] Comparing the field information with second configuration information of the wide table field to obtain field difference information, where the second configuration information is configuration information of the wide table field obtained before determining the first configuration information of the wide table field;

[0187] The second configuration information is updated based on the field difference information to obtain the first configuration information.

[0188] In another embodiment, when the determination module updates the second configuration information based on the field difference information to obtain the first configuration information, the determination module performs the following steps:

[0189] Based on the field difference information, determining an update keyword that matches the second configuration information;

[0190] generating an update statement based on the field difference information and the update keyword, wherein the update statement is used to update the second configuration information;

[0191] Execute the update statement to obtain the first configuration information.

[0192] In another embodiment, the connection module is used to:

[0193] Based on the multiple primary key screening conditions, the multiple first data tables are grouped to obtain multiple task groups, and the primary key screening conditions corresponding to the first data tables in the same task group are the same;

[0194] For each of the task groups, based on the primary key of the first data table in the task group, the first configuration information and a connection statement template, a connection operation statement of the task group is generated, wherein the connection operation statement is used to perform a connection operation on the first data table in the task group;

[0195] Execute the connection operation statement of each task group to obtain the second data table corresponding to each task group.

[0196] In another embodiment, the connection module performs the following steps when generating a connection operation statement for the task group based on the primary key of the first data table in the task group, the first configuration information, and a connection statement template:

[0197] Determine the correspondence between the wide table field and the first data table in the task group based on the field information of the data source table to which the first data table in the task group belongs;

[0198] A connection operation statement for the task group is generated based on the corresponding relationship, the primary key of the first data table in the task group, and the connection statement template.

[0199] In another embodiment, the connection statement template includes: a first selection keyword, a connection keyword, a condition keyword, and a starting point keyword;

[0200] When the connection module generates a connection operation statement for the task group based on the corresponding relationship, the primary key of the first data table in the task group, and the connection statement template, the connection module performs the following steps:

[0201] Based on the corresponding relationship and the first selection keyword, generating a first selection clause of the task group, wherein the first selection clause is used to obtain a field value of the wide table field from a first data table included in the task group;

[0202] Determining a first candidate data table and a second candidate data table from the task group based on an order of the first data tables included in the task group;

[0203] Based on the table name of the first candidate data table and the starting point keyword, generating a starting clause, wherein the starting clause is used to indicate that the first candidate data table is used as a starting point for a connection operation;

[0204] Based on the table name of the second candidate data table and the connection keyword, a connection clause of the second candidate data table is generated, wherein the connection clause is used to perform a connection operation on the second candidate data table;

[0205] generating a conditional clause of the second candidate data table based on the primary key of the second candidate data table, the table name of the first data table sequentially located before the second candidate data table, and the conditional keyword, wherein the conditional clause is used to indicate a condition for performing a join operation on the second candidate data table;

[0206] A join operation statement for the task group is generated based on the first selection clause, the start clause, the join clause of the second candidate data table, and the conditional clause.

[0207] In another embodiment, the connection statement template further includes a second selection keyword;

[0208] The connection module is also used for:

[0209] Based on the second selection keyword and the table name and splitting method of the data source table to which the first data table in the task group belongs, generate a second selection clause for the first data table in the task group, wherein the second selection clause is used to obtain the first data table;

[0210] The generating the connection operation statement of the task group based on the first selection clause, the start clause, the connection clause of the second candidate data table and the conditional clause includes:

[0211] Combining the first selection clause, the start clause, the connection clause of the second candidate data table and the conditional clause to obtain a candidate connection statement;

[0212] Based on the splitting mode of the data source table to which the first data table in the task group belongs, the second selection clause of the first data table in the task group and the candidate connection statement are combined to obtain the connection operation statement of the task group.

[0213] In another embodiment, the connection module performs the following steps when combining the second selection clause of the first data table in the task group and the candidate connection statement based on the splitting mode of the data source table to which the first data table in the task group belongs to obtain the connection operation statement of the task group:

[0214] If the splitting mode of the data source table to which the first data table in the task group belongs is logical splitting, then in the candidate connection statement, the table name of the first data table is replaced with a connection operation statement of the first data table to obtain a connection operation statement of the task group;

[0215] If the splitting mode of the data source table to which the first data table in the task group belongs is logical splitting, then in the candidate connection statement, the table name of the first data table is replaced with the second selection clause of the first data table to obtain the connection operation statement of the task group;

[0216] If the splitting mode of the data source table to which the first data table in the task group belongs is physical splitting, the second selection clause of the first data table is concatenated with the candidate connection statement to obtain the connection operation statement of the task group.

[0217] In another embodiment, the merging module is used to:

[0218] Based on the table name and the merge keyword of each second data table, generate a merge clause of each second data table;

[0219] splicing the merge clauses of each of the second data tables to obtain a merge operation statement, where the merge operation statement is used to merge the multiple second data tables;

[0220] Execute the merge operation statement to obtain the target data table.

[0221] Obviously, the data table generating device in the embodiment of this specification can be used as the above Figure 2 The execution body of the data table generation method shown in FIG. Figure 2 The functions realized are as follows. Since the principles are the same, they will not be described here.

[0222] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 6 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0223] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0224] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0225] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a data table generation device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0226] Determine first configuration information of a wide table field based on first metadata information of a plurality of data source tables;

[0227] For each of the data source tables, based on a plurality of preset primary key screening conditions, the data source table is split into a plurality of first data tables; the plurality of first data tables have the same structure, and each of the first data tables corresponds to a primary key screening condition;

[0228] Based on the first configuration information and the plurality of the first data tables, a plurality of second data tables, each of the second data tables including the wide table field, and the primary key screening conditions corresponding to the first data tables in the same second data table are the same;

[0229] The multiple second data tables are merged to obtain a target data table.

[0230] The above application Figure 2The method performed by the data table generating device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0231] The electronic device may also perform Figure 2 The method and the data table generating device are implemented in Figures 1 to 4 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0232] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0233] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 2 The method of the embodiment shown is specifically used to perform the following operations:

[0234] Determining first configuration information of a wide table field based on first metadata information of a plurality of data source tables;

[0235] For each of the data source tables, based on a plurality of preset primary key screening conditions, the data source table is split into a plurality of first data tables; the plurality of first data tables have the same structure, and each of the first data tables corresponds to a primary key screening condition;

[0236] Based on the first configuration information and the plurality of the first data tables, a plurality of second data tables, each of the second data tables including the wide table field, and the primary key screening conditions corresponding to the first data tables in the same second data table are the same;

[0237] The multiple second data tables are merged to obtain a target data table.

[0238] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps in the data table generation method provided in the embodiment of the present application.

[0239] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0240] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0241] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0242] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0243] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A data table generation method, characterized in that: include: Determine first configuration information of a wide table field based on first metadata information of a plurality of data source tables; For each of the data source tables, based on a plurality of preset primary key screening conditions, split the data source table into a plurality of first data tables; The structures of the multiple first data tables are the same, and each of the first data tables corresponds to a primary key screening condition; Based on the first configuration information and the plurality of the first data tables, a plurality of second data tables are generated, each of the second data tables includes the wide table field, and the primary key screening conditions corresponding to the first data tables in the same second data table are the same; The multiple second data tables are merged to obtain a target data table.

2. The method according to claim 1, characterized in that For each of the data source tables, splitting the data source table into multiple first data tables based on multiple preset primary key screening conditions includes: Determine a splitting method of the data source table based on the amount of data in the data source table; Based on the splitting mode and primary key of the data source table, the data source table is split into multiple first data tables.

3. The method according to claim 1, characterized in that The determining, based on the first metadata information of the plurality of data source tables, first configuration information of the wide table field includes: Based on the first metadata information, determining the field information of each of the data source tables; Comparing the field information with second configuration information of the wide table field to obtain field difference information, where the second configuration information is configuration information of the wide table field obtained before determining the first configuration information of the wide table field; The second configuration information is updated based on the field difference information to obtain the first configuration information.

4. The method according to claim 3, characterized in that: The updating the second configuration information based on the field difference information to obtain the first configuration information includes: Based on the field difference information, determining an update keyword that matches the second configuration information; generating an update statement based on the field difference information and the update keyword, wherein the update statement is used to update the second configuration information; Execute the update statement to obtain the first configuration information.

5. The method according to claim 1, characterized in that The generating a plurality of second data tables based on the first configuration information and the plurality of first data tables comprises: Based on the multiple primary key screening conditions, the multiple first data tables are grouped to obtain multiple task groups, and the primary key screening conditions corresponding to the first data tables in the same task group are the same; For each of the task groups, based on the primary key of the first data table in the task group, the first configuration information and a connection statement template, a connection operation statement of the task group is generated, wherein the connection operation statement is used to perform a connection operation on the first data table in the task group; Execute the connection operation statement of each task group to obtain the second data table corresponding to each task group.

6. The method according to claim 5, characterized in that The generating a connection operation statement for the task group based on the primary key of the first data table in the task group, the first configuration information, and a connection statement template includes: Determine the correspondence between the wide table field and the first data table in the task group based on the field information of the data source table to which the first data table in the task group belongs; A connection operation statement for the task group is generated based on the corresponding relationship, the primary key of the first data table in the task group, and the connection statement template.

7. A data table generating device, characterized in that: include: A determination module, configured to determine first configuration information of a wide table field based on first metadata information of a plurality of data source tables; A splitting module, configured to split each of the data source tables into a plurality of first data tables based on a plurality of preset primary key screening conditions; The structures of the multiple first data tables are the same, and each of the first data tables corresponds to a primary key screening condition; A connection module, configured to generate a plurality of second data tables based on the first configuration information and the plurality of first data tables, wherein each second data table includes the wide table field, and primary key screening conditions corresponding to the first data tables in the same second data table are the same; The merging module is used to merge the multiple second data tables to obtain a target data table.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data table generating method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the data table generating method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute part or all of the steps in the data table generating method according to any one of claims 1 to 6.

Images