Data table processing method and device, computer device and storage medium

Abstract

The application relates to a data table processing method and device, computer equipment and a storage medium. The method relates to big data, and specifically comprises the following steps: acquiring meta information and splicing mode information of each source data table to be spliced; wherein the meta information of each source data table at least comprises the field name of each source data table; each source data table is a data table in a big data warehouse; each field name is organized into a data table according to the splicing mode information, so as to obtain a corresponding wide table view; a data protocol is generated according to the meta information of each source data table and the wide table view; and a splicing result obtained by splicing each source data table through the generated data protocol is represented. The method can effectively avoid the phenomenon of memory overflow during wide table splicing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data table processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the development of computer technology, machine learning technology has emerged. Machine learning requires model training, and the training data needed for model training is usually stored in a big data warehouse. In a big data warehouse, data is distributed across different tables according to different themes.

[0003] In traditional techniques, models typically only receive two-dimensional matrix data as training data. Therefore, analyzing this training data requires concatenating the data tables into a single, wide table. However, in big data warehouses, the number of tables is enormous, and the amount of data involved is massive. Concatenating these tables leads to a rapid increase in the number of table columns and computational metadata, often exceeding the database warehouse system's limits and causing memory overflow. Summary of the Invention

[0004] Therefore, it is necessary to provide a data table processing method, apparatus, computer equipment, and storage medium that can effectively avoid memory overflow when merging wide tables, in order to address the above-mentioned technical problems.

[0005] A data table processing method includes: obtaining metadata and concatenation method information for each source data table to be concatenated; the metadata of each source data table includes at least the field names of each source data table; organizing the field names into a data table according to the concatenation method information to obtain a corresponding wide table view; generating a data protocol based on the metadata of each source data table and the wide table view; and representing the concatenation result obtained by concatenating the source data tables through the generated data protocol.

[0006] A data table processing apparatus includes: an information acquisition module for acquiring metadata and concatenation method information of each source data table to be concatenated; the metadata of each source data table includes at least the field names of each source data table; a data concatenation module for organizing the field names into a data table according to the concatenation method information to obtain a corresponding wide table view; a data protocol generation module for generating a data protocol based on the metadata of each source data table and the wide table view; and a concatenation result representation module for representing the concatenation result obtained by concatenating the source data tables through the generated data protocol.

[0007] In some embodiments, the data protocol generation module is further configured to construct a logical layer based on the metadata of each of the source data tables; construct a user layer based on the wide table view; and generate the data protocol based on the logical layer and the user layer.

[0008] In some embodiments, the splicing result representation module is further configured to receive initial data transformation information for the splicing result through a wide table view corresponding to the data protocol; determine the target data table corresponding to the initial data transformation information according to the field names of each of the source data tables, and generate target data transformation information for the target data table according to the initial data transformation information; the target data transformation information is used to characterize the corresponding data transformation performed on the target data table; and record the target data transformation information through the data protocol to update the data protocol.

[0009] In some embodiments, the splicing result representation module is further configured to receive a computation graph for the splicing result through a wide table view corresponding to the data protocol; wherein the computation graph includes at least two graph nodes; the graph node is any one of row selection operator, column selection operator, and column generation operator; determine the target data table corresponding to the computation graph according to the field names of each of the source data tables, and generate target data transformation information for the target data table according to the computation graph.

[0010] In some embodiments, when the graph node is a row selection operator; the concatenation result representation module is further configured to compare the column information corresponding to the row selection operator with the field names of each of the source data tables, determine the target data table based on the comparison result; generate target row selection information for the target data table based on the row selection operator; the target row selection information is used to characterize the corresponding row selection operation performed on the target data table; and record the target row selection information through the data protocol to update the data protocol.

[0011] In some embodiments, when the graph node is a column selection operator, the concatenation result representation module is further configured to compare the column information corresponding to the column selection operator with the field names of each of the source data tables, determine the target data table based on the comparison result, generate target column selection information for the target data table based on the column selection operator, the target column selection information is used to characterize the corresponding column selection operation performed on the target data table, record the target column selection information through the data protocol to update the data protocol, and perform the column selection operation on the wide table view.

[0012] In some embodiments, when the graph node is a column generation operator; the concatenation result representation module is further configured to compare the column information corresponding to the column generation operator with the field names of each of the source data tables, determine the target data table based on the comparison result; generate target column generation information for the target data table based on the column generation operator; the target column generation information is used to characterize the corresponding column generation operation performed on the target data table; record the target column generation information through the data protocol to update the data protocol; and perform the column generation operation on the wide table view.

[0013] In some embodiments, the target data tables include multiple ones; the splicing result representation module is further configured to determine logical splicing information corresponding to the multiple target data tables; generate column generation information for the target spliced ​​data tables corresponding to the logical splicing information according to the column generation operator; and determine the logical splicing information and the column generation information of the target spliced ​​data tables as the target column generation information for the multiple target data tables.

[0014] In some embodiments, the source data tables to be concatenated form a source data table set; the data table processing device further includes: a sending module, which is used to send a data landing request to the server; the data landing request carries metadata, target data transformation information, and data landing mode information for each of the source data tables; the target data transformation information is generated by converting the initial data transformation information received by the wide table view of the data protocol for the concatenation result into source data tables in the source data table set; the metadata of each source data table includes the location information of each of the source data tables; the server is used to perform a data transformation operation on the target source data table according to the target data transformation information and the location information of the target source data table corresponding to the target data transformation information, so as to update the source data table set; when the data landing mode information is a first preset information, the server obtains and stores each source data table in the updated source data table set according to the location information of each source data table.

[0015] In some embodiments, the server is further configured to, when the data persistence mode information is the second preset information, obtain each source data table in the updated source data table set according to the metadata of each source data table, group each source data table in the updated source data table set to obtain multiple data table groups; the server concatenates the data tables in each data table group to obtain the current concatenated data table for each data table group; the server groupes each currently concatenated data table to obtain multiple data table groups; the server repeatedly enters the step of concatenating the data tables in each data table group until a target wide table containing all source data in the updated source data table set is obtained; the server stores the target wide table.

[0016] A computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: acquiring metadata and concatenation method information of each source data table to be concatenated; the metadata of each source data table includes at least the field names of each source data table; organizing the field names into a data table according to the concatenation method information to obtain a corresponding wide table view; generating a data protocol based on the metadata of each source data table and the wide table view; and representing the concatenation result obtained by concatenating the source data tables through the generated data protocol.

[0017] A computer-readable storage medium storing a computer program, which, when executed by a processor, performs the following steps: obtaining metadata and concatenation method information for each source data table to be concatenated; the metadata of each source data table includes at least the field names of each source data table; organizing the field names into a data table according to the concatenation method information to obtain a corresponding wide table view; generating a data protocol based on the metadata of each source data table and the wide table view; and representing the concatenation result obtained by concatenating the source data tables through the generated data protocol.

[0018] The aforementioned data table processing method, apparatus, computer equipment, and storage medium acquire the metadata and splicing method information of each source data table to be spliced. The metadata of each source data table includes at least the field names of each source data table. Further, the field names are organized into a data table according to the splicing method information to obtain a corresponding wide table view. A data protocol is generated based on the metadata and wide table view of each source data table. The generated data protocol then represents the splicing result obtained by splicing the source data tables. Since the data protocol only needs to splice the field names of the source data tables during generation, and does not need to splice all the data in the source data tables, it greatly saves computer memory and effectively avoids the memory overflow phenomenon caused by directly splicing all the data in the source data tables.

[0019] A data table processing method, characterized in that the method includes: acquiring a generated data protocol; wherein the data protocol is generated based on the metadata and wide table view of each of the source data tables to be concatenated; the metadata of each of the source data tables includes at least the field names of each of the source data tables; the wide table view is obtained by organizing the field names into a data table according to the acquired concatenation method information; receiving initial data transformation information for the concatenation result through the wide table view corresponding to the data protocol; determining the target data table corresponding to the initial data transformation information based on the field names of each of the source data tables, and generating target data transformation information for the target data table based on the initial data transformation information; the target data transformation information is used to characterize the corresponding data transformation performed on the target data table; recording the target data transformation information through the data protocol; when concatenating the source data tables, sending the metadata of each of the source data tables and the recorded target data transformation information to a server; the server is used to perform data transformation operations based on the metadata of each of the source data tables and the target data transformation information, and concatenate the source data tables to obtain a target wide table.

[0020] A data table processing apparatus, characterized in that the apparatus comprises: a data protocol acquisition module, configured to acquire a generated data protocol; wherein the data protocol is generated based on the metadata and wide table view of each of the source data tables to be concatenated; the metadata of each of the source data tables includes at least the field names of each of the source data tables; the wide table view is obtained by organizing the field names into a data table according to the acquired concatenation method information; an information receiving module, configured to receive initial data transformation information for the concatenation result through the wide table view corresponding to the data protocol; and an information transformation module, configured to determine the initial data based on the field names of each of the source data tables. The system generates target data transformation information for the target data table based on the target data table corresponding to the transformation information and the initial data transformation information. The target data transformation information is used to characterize the corresponding data transformation performed on the target data table. An information recording module is used to record the target data transformation information through the data protocol. An information sending module is used to send the metadata of each source data table and the recorded target data transformation information to the server when splicing the source data tables. The server is used to perform data transformation operations based on the metadata of each source data table and the target data transformation information, and splice the source data tables to obtain a target wide table.

[0021] A computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: acquiring a generated data protocol; wherein the data protocol is generated based on the metadata and wide table view of each of the source data tables to be concatenated; the metadata of each of the source data tables includes at least the field names of each of the source data tables; the wide table view is obtained by organizing the field names into a data table according to the acquired concatenation method information; receiving initial data transformation information for the concatenation result through the wide table view corresponding to the data protocol; and based on the metadata of each of the source data tables... The field name determines the target data table corresponding to the initial data transformation information, and generates target data transformation information for the target data table based on the initial data transformation information; the target data transformation information is used to characterize the corresponding data transformation performed on the target data table; the target data transformation information is recorded through the data protocol; when concatenating the various source data tables, the metadata of each source data table and the recorded target data transformation information are sent to the server; the server is used to perform data transformation operations based on the metadata of each source data table and the target data transformation information, and concatenate the various source data tables to obtain the target wide table.

[0022] A computer-readable storage medium storing a computer program thereon, the computer program, when executed by a processor, performs the following steps: acquiring a generated data protocol; wherein the data protocol is generated based on the metadata and wide table view of each of the source data tables to be concatenated; the metadata of each of the source data tables includes at least the field names of each of the source data tables; the wide table view is obtained by organizing the field names into a data table according to the acquired concatenation method information; receiving initial data transformation information for the concatenation result through the wide table view corresponding to the data protocol; and determining based on the field names of each of the source data tables. The initial data transformation information corresponds to the target data table, and target data transformation information for the target data table is generated based on the initial data transformation information; the target data transformation information is used to characterize the corresponding data transformation performed on the target data table; the target data transformation information is recorded through the data protocol; when splicing the various source data tables, the metadata of each source data table and the recorded target data transformation information are sent to the server; the server is used to perform data transformation operations based on the metadata of each source data table and the target data transformation information, and splice the various source data tables to obtain the target wide table.

[0023] The aforementioned data table processing method, apparatus, computer equipment, and storage medium acquire a generated data protocol, which is generated based on the metadata and wide table view of each source data table to be concatenated. The metadata of each source data table includes at least the field names of each source data table. The wide table view is obtained by organizing the field names into a data table according to the acquired concatenation method information. Thus, the initial data transformation information for the concatenation result is received through the wide table view corresponding to the data protocol. The target data table corresponding to the initial data transformation information is determined based on the metadata of each source data table. Target data transformation information for the target data table is generated based on the initial data transformation information. The target data transformation information is used to characterize the corresponding data transformation performed on the target data table. The target data transformation information is recorded through a data protocol. When concatenating the various source data tables, the metadata of each source data table and the recorded target data transformation information are sent to the server. The server performs data transformation operations based on the metadata of each source data table and the target data transformation information, and concatenates the various source data tables to obtain the target wide table. Thus, by using a data protocol to represent the actual concatenated wide table, the concatenation of the wide table can be delayed as much as possible. Furthermore, by converting the data transformations applied to the wide table into those applied to the source data tables, the amount of data during wide table concatenation can be reduced as much as possible, thereby improving the execution efficiency of wide table concatenation. Attached Figure Description

[0024] Figure 1 This is an application environment diagram of a data table processing method in one embodiment;

[0025] Figure 2 This is a flowchart illustrating a data table processing method in one embodiment;

[0026] Figure 3 This is a schematic diagram of a data protocol in one embodiment;

[0027] Figure 4 This is a flowchart illustrating the splicing result using a data protocol in one embodiment;

[0028] Figure 5 This is a schematic diagram of the data transformation process in one embodiment;

[0029] Figure 6 This is a schematic diagram illustrating the operation of an operator on a data protocol in one embodiment;

[0030] Figure 7 This is a schematic diagram illustrating the operation of an operator on a data protocol in another embodiment;

[0031] Figure 8 This is a flowchart illustrating a data table processing method in one embodiment;

[0032] Figure 8AThis is a schematic diagram of a process for wide table concatenation based on the Apache Spark computing framework in one embodiment;

[0033] Figure 9 This is a structural block diagram of a data table processing device in one embodiment;

[0034] Figure 10 Here is a structural block diagram of a data table processing method in another embodiment;

[0035] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0037] The data table processing method provided in this application can involve data tables in a big data warehouse. Big data refers to data sets that cannot be captured, managed, and processed within a certain timeframe using conventional software tools. It represents massive, rapidly growing, and diverse information assets that require new processing models to achieve stronger decision-making, insight discovery, and process optimization capabilities. With the advent of the cloud era, big data has attracted increasing attention. Big data requires specialized technologies to effectively process large amounts of data within a tolerable timeframe. Technologies suitable for big data include massively parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the Internet, and scalable storage systems.

[0038] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0039] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0040] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0041] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0042] The solutions provided in this application relate to big data processing technology and machine learning in artificial intelligence, specifically the processing of training data during the model training process in machine learning. This application is illustrated through the following embodiments:

[0043] The data table processing method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network, and server 104 communicates with data warehouse 106 via a network.

[0044] Terminal 102 obtains the metadata and splicing method information of each source data table to be spliced. The metadata of each source data table includes at least the field names of each source data table. The field names are organized into a data table according to the splicing method information to obtain the corresponding wide table view. A data protocol is generated based on the metadata and wide table view of each source data table. The generated data protocol represents the splicing result obtained by splicing the source data tables.

[0045] During the process of representing the splicing result through the generated data protocol, terminal 102 receives initial data transformation information for the splicing result, determines the target data table corresponding to the initial data transformation information based on the field names of each source data table, and generates target data transformation information for the target data table based on the initial data transformation information. Terminal 102 further records the target data transformation information through the data protocol. When splicing each source data table, terminal 102 sends the metadata of each source data table and the recorded target data transformation information to server 104. After server 104 performs data transformation operations based on the metadata of each source data table and the target data transformation information, it retrieves these source data tables from data warehouse 106 based on the location information included in the metadata of each source data table and splices these source data tables to obtain the target wide table.

[0046] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not restricted herein.

[0047] In some embodiments, such as Figure 2 As shown, a data table processing method is provided, which can be applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps:

[0048] Step 202: Obtain the metadata and splicing method information of each source data table to be spliced; the metadata of each source data table shall include at least the field names of each source data table.

[0049] The source data table to be concatenated refers to the source data table that needs to be merged into a wide table. The source data table stores the training data used during model training. Training data is typically distributed across different source data tables according to different themes. For example, order-related information is organized in the "Order Table," containing fields such as user ID, order ID, and order amount, while user-related information is organized in the "User Table," containing fields such as user ID, name, and age. The metadata of the source data table refers to information describing data properties, including the location information of the source data table, the table structure, the primary key, etc. Location information represents the storage location of the source data table in the data warehouse; specifically, it can be a storage path. The same source data table can have multiple storage paths, representing that the source data table is distributed across different computer devices. The table structure includes the table name, field names, and field types, etc. In practical applications, the field names are usually the column names of each column in the data table, used to generalize the data in that column. The concatenation method information describes the concatenation method used when concatenating the source data tables into a wide table. The concatenation method can be any of the following: left join, right join, and full join.

[0050] Specifically, the terminal can receive splicing information input by the user. The splicing information includes the table names and splicing method information of each source data table to be spliced. The terminal can query the metadata of each data table to be spliced ​​from the server based on the table names. The metadata includes at least the field names of the data tables.

[0051] Step 204: Organize the field names into a data table according to the concatenation method information to obtain the corresponding wide table view.

[0052] Specifically, the terminal organizes the field names into a data table according to the splicing method in the splicing method information to obtain the corresponding wide table view. This wide table view only includes the field names of each source data table and does not include the data records of each source data table.

[0053] In some embodiments, the terminal first organizes the field names of each source data table into sub-data tables according to their order in the source data tables, and then concatenates these sub-data tables according to the concatenation method information to obtain a data table, which is the obtained wide table view.

[0054] For example, assuming the source data tables to be joined are Table A and Table B below, and the join method is left join, the resulting wide table view is shown in Table C below:

[0055] Table 1

[0056]

[0057] Table 2

[0058]

[0059] Table 3

[0060]

[0061] Step 206: Generate a data protocol based on the metadata and wide table view of each source data table.

[0062] Here, a data protocol refers to a data structure that can store information about the various source data tables that make up the wide table, including the metadata of each source data. Users can manipulate this data structure using operators, which can be described in pseudocode as follows:

[0063] Class DataProto:

[0064] Src_tables: List[Table] # Source data tables, for example [Table1, Table2, Table3]

[0065] Specifically, after obtaining the wide table view, the terminal generates a data protocol based on the metadata of each source data table and the obtained wide table view.

[0066] In some embodiments, the terminal constructs a logical layer based on the metadata of each source data table, constructs a user layer based on the obtained wide table view, and generates a data protocol based on the logical layer and the user layer.

[0067] Step 208: The splicing result obtained by splicing the various source data tables using the generated data protocol representation.

[0068] The concatenation result refers to the wide table obtained by actually concatenating the various source data tables. For example, the wide table obtained by actually concatenating Table 1 and Table 2 above is as follows: Figure 4 As shown:

[0069] Table 4

[0070]

[0071] Specifically, after generating the data protocol, the terminal can use the generated data protocol to represent the splicing result obtained by splicing the data from each source data table. During the representation process, the terminal can respond to the user's input of data transformation information for the splicing result through the data protocol. The user can use the data protocol like a normal wide table, where a normal wide table refers to a wide table obtained by actually splicing the data from each source data table.

[0072] In some embodiments, data transformation operations include, but are not limited to, row selection operations, column selection operations, column generation operations, etc.

[0073] In some embodiments, when the terminal responds to the user's input of data transformation information for the splicing result, it can determine the target data table on which the data transformation operation actually applies, and convert the data transformation operation for the splicing result into a data transformation operation for the target data table. Typically, the amount of data in the target data table is significantly reduced compared to the amount of data in the spliced ​​wide table, which can improve the execution efficiency of the data transformation operation.

[0074] In some embodiments, after the terminal converts the data transformation information for the splicing result into target data transformation information for the target data table, it can record the data transformation information for the target data table through a data protocol. Then, when data is stored, the terminal can send this target data transformation information to the server. The server can first perform data transformation operations on the corresponding target data table according to the target data transformation information to update the target data table. After the data transformation operation is completed, the wide table splicing is performed. Since the amount of data during splicing is usually reduced after the data transformation is completed, the execution efficiency of wide table splicing can be improved by delaying the wide table splicing as much as possible.

[0075] In the above data table processing method, the metadata and concatenation method information of each source data table to be concatenated are obtained. The metadata of each source data table includes at least the field names of each source data table. The field names are then organized into a data table according to the concatenation method information to obtain the corresponding wide table view. A data protocol is generated based on the metadata and wide table view of each source data table. The generated data protocol represents the concatenation result obtained by concatenating the source data tables. Since the data protocol only needs to concatenate the field names of the source data tables during the generation process, and does not need to concatenate all the data of the source data tables, it greatly saves computer memory and effectively avoids the memory overflow phenomenon caused by directly concatenating all the data of each source data table.

[0076] In some embodiments, generating a data protocol based on the metadata of each source data table and a wide table view includes: constructing a logical layer based on the metadata of each source data table; constructing a user layer based on the wide table view; and generating a data protocol based on the logical layer and the user layer.

[0077] Specifically, the terminal records the metadata of each source data table to build the logical layer, and at the same time displays the generated wide table view to build the user layer. The logical layer and the user layer are connected through a data interface to generate a data protocol.

[0078] like Figure 3The diagram shown is a schematic representation of the data protocol in some embodiments. (Refer to...) Figure 3 The logic layer records the metadata of Table 1, Table 2, ... Table N, while the user layer displays the generated wide table view. This enables the support of wide table representations with millions of columns without modifying the existing data warehouse system. At the same time, it hides the source data table of the logic layer from the user, allowing the user to use the data protocol as if it were a normal wide table.

[0079] In some embodiments, such as Figure 4 As shown, the concatenation result obtained by combining the various source data tables through the generated data protocol includes:

[0080] Step 402: Receive the initial data transformation information for the splicing result through the wide table view corresponding to the data protocol.

[0081] The initial data transformation information indicates the data transformation operations performed on the spliced ​​result. These operations refer to changes made to the data in the data table, including but not limited to row selection, column selection, and column generation. Row selection refers to selecting several rows from the data table, or filtering rows based on conditions, such as selecting users whose registration date is greater than 2020. Column selection refers to selecting several columns from the data table, such as selecting columns related to monetary amounts. Column generation refers to creating new columns based on existing columns in the data table; for example, calculating the average amount for the first quarter involves averaging the amounts from January, February, and March, and using this average as the new column.

[0082] Since the initial data transformation information indicates a transformation operation on data in a data table, it typically carries identification information to identify the data to be transformed. This identification information can be a field name, which in practice can be a column name. Furthermore, since the order of columns in a wide table view is fixed, this identification information can also be the order of the columns where the data resides. For example, in a row selection operation involving selecting users whose gender is female, the initial data transformation information would carry the field name related to gender, such as "Gender". If the "Gender" field is located in the third column of the wide table view, the identification information carried in the initial data transformation information could also be "col3".

[0083] It is understood that the wide table view corresponding to the data protocol in this application is essentially a data table. Due to the support of the data protocol, even when the source data tables are not actually spliced ​​together, the user can use this wide table view as a substitute for the spliced ​​wide table. The user can then input the initial data transformation information for the splicing result (i.e., the wide table obtained by splicing the source data tables) on this wide table view. The terminal can then receive the initial data transformation information for the splicing result through the wide table view corresponding to the data protocol.

[0084] Step 404: Determine the target data table corresponding to the initial data transformation information based on the field names of each source data table, and generate target data transformation information for the target data table based on the initial data transformation information.

[0085] Specifically, the terminal can determine the field name of the data involved in the transformation operation based on the identification information carried in the initial data transformation information, and then match this field name with the field names in the metadata of each source data table. Based on the matching result, the terminal can determine the source data table where the data involved in the transformation operation resides, and then identify this source data table as the target data table. It can be understood that there can be one or more target data tables, with at least two. For example, the terminal can determine that the field name of the data involved in the transformation operation is "gender" based on the identification information carried in the initial data transformation information. If the metadata of source data table 3 includes the field name "gender," then the terminal can successfully match the metadata of source data table 3, and thus identify source data table 3 as the target data table.

[0086] The terminal can generate target data transformation information for the target data table based on the initial data transformation information. The generated target data transformation information carries the target data table identifier and is used to characterize the corresponding data transformation performed on the target data table.

[0087] For example, if the initial data transformation information is: select(col1,col2), where col1 and col2 involve the first and second columns of table1 respectively, then the generated target data transformation information is select(col1, col2) from table1.

[0088] Step 406: Record the target data transformation information through the data protocol to update the data protocol.

[0089] After generating the target data transformation information for the target data table, the terminal records the target data transformation information through the data protocol to update the data protocol. Thus, when data is written to the target data table, the server can perform data transformation operations according to the updated data protocol. Ultimately, this allows the server to respond to the initial data transformation information for the splicing result simply by performing data transformation on the target data table.

[0090] In some embodiments, the terminal records target data transformation information through the logical layer of the data protocol to update the logical layer of the data protocol, thereby updating the data protocol.

[0091] In the above embodiments, since the initial data transformation information for the splicing result is converted into target data transformation information for the target data table, and this target data transformation information is recorded through a data protocol, the data range of the data transformation can be reduced, and the execution efficiency can be greatly improved. Simultaneously, since the data transformation is performed first, the splicing of the wide table is delayed as much as possible, and the amount of data is minimized during subsequent wide table splicing, which can improve the execution efficiency during wide table splicing. In some embodiments, receiving the initial data transformation information for the splicing result through the wide table view corresponding to the data protocol includes: receiving a computation graph for the splicing result through the wide table view corresponding to the data protocol; wherein, the computation graph includes at least two graph nodes; the graph nodes are any one of row selection operators, column selection operators, and column generation operators; determining the target data table corresponding to the initial data transformation information based on the field names of each source data table, and generating target data transformation information for the target data table based on the initial data transformation information includes: determining the target data table corresponding to the computation graph based on the field names of each source data table, and generating target data transformation information for the target data table based on the computation graph.

[0092] The computation graph includes at least two graph nodes; each node can be any one of the row selection, column selection, and column generation operators. An operator is defined as a single data transformation. For example... Figure 5 The diagram shown illustrates the operation of an operator on a data protocol in one embodiment. It can be seen that the original data protocol is updated after data transformation by the operator, resulting in a new data protocol.

[0093] Depending on the use case, operators can be divided into three categories: row selection operators, column selection operators, and column generation operators. A row selection operator is defined as a single row selection operation, which selects several rows from a data table; a column selection operator is defined as a single column selection operation, which selects several columns from a data table; and a column generation operator is defined as a single column generation operation, which generates new columns based on existing columns from one or more data tables. It can be understood that the nodes in a computational graph are typically organized in a certain order, and the nodes are connected by directed edges, with each directed edge pointing from one node to the next. There is usually a dependency between two nodes with a sequential order, meaning that the data transformation of a later node is based on the data transformation result of the earlier node.

[0094] Specifically, the terminal receives the computation graph input by the user through a wide table view of the data protocol. This computation graph is for the splicing result, which is a wide table obtained by actually splicing the various source data. In this case, the data transformations described by each graph node in the computation graph are all for the spliced ​​data table. However, in this application, when the terminal receives the computation graph, the various source data have not actually been spliced. Therefore, the terminal can directly locate the target data table where the data transformations described by each graph node in the computation graph are located, and then convert the data transformations described by each graph node into data transformations for the corresponding target data table in sequence to obtain the target data transformation information.

[0095] In some embodiments, determining the target data table corresponding to the computation graph based on the field names of each source data table, and generating target data transformation information for the target data table based on the computation graph includes: comparing the column information corresponding to the first graph node with the field names of each source data table, determining the first target data table based on the comparison results, and converting the initial data transformation information corresponding to the first graph node into target data transformation information for the first target data table; determining the current data table to be concatenated based on the target data transformation information for the first target data table; determining the next graph node as the current graph node; comparing the current graph node with the current data table to be concatenated, determining the current target data table based on the comparison results, and converting the initial data transformation information corresponding to the current graph node into target data transformation information for the current target data table; determining the current data table to be concatenated based on the target data transformation information for the current target data table; and repeating the step of determining the next graph node as the current graph node until the current graph node is the last graph node.

[0096] In the above embodiments, the terminal receives the computation graph for the splicing result through the wide table view corresponding to the data protocol, determines the target data table corresponding to the computation graph based on the metadata of each source data table, and generates target data transformation information for the target data table based on the computation graph. Since the computation graph abstracts the data transformation process, it abstracts the data transformation into three types of operators: row selection, column selection, and column generation, which simplifies the data transformation operation.

[0097] In some embodiments, when the graph node is a row selection operator, determining the target data table corresponding to the computation graph based on the field names of each source data table, and generating target data transformation information for the target data table based on the computation graph includes: comparing the column information corresponding to the row selection operator with the field names of each source data table, and determining the target data table based on the comparison result; generating target row selection information for the target data table based on the row selection operator; the target row selection information is used to characterize the corresponding row selection operation performed on the target data table; and recording the target data transformation information through the data protocol to update the data protocol includes: recording the target row selection information through the data protocol to update the data protocol.

[0098] Specifically, since the row selection operator describes a row selection operation, which refers to selecting a number of rows from a data table, the row selection operator includes column information, which serves as the basis for the row selection operation. The row selection operator can be a WHERE operator, for example: WHERE start_date < 2007-01-01, where the column information included is "start_date".

[0099] The terminal compares the column information corresponding to the row selection operator with the field names of each source data table. Based on the comparison results, the target data table is determined, and the row selection operator is applied to the target data table to generate the target row selection information for the target data table. The target row selection information is used to represent the corresponding row selection operation performed on the target data table. After the row selection operation is actually completed, the target data table is replaced with the selection result corresponding to the row selection operation to update the target data table.

[0100] For example, such as Figure 6 As shown, if the row selection operator needs to select users registered after 2020, and the registration information exists in Table 2, then it is only necessary to select the relevant rows in Table 2 according to the condition "registered after 2020". At this time, the terminal can apply the row selection operator to Table 2, that is, generate the target row selection information for Table 2 according to the row selection operator. The target row selection information is used to describe the selection of relevant rows from Table 2 that meet the condition "registered after 2020". The selected data replaces the original Table 2, thereby updating Table 2.

[0101] Furthermore, since the target row selection information needs to be applied to the target data table, the terminal can record this information through the data protocol. At this point, the data protocol is updated. During subsequent data storage, the terminal can send the information recorded in the updated data protocol to the server. The server can then perform row selection operations on the target data table based on the target row selection information recorded in the data protocol to update the target data table. In some specific embodiments, the terminal can record the target row selection information through the logic layer of the data protocol to update the logic layer.

[0102] Understandably, the column information for row selection will not change, therefore the wide table view in the user layer will not change.

[0103] In the above embodiments, when the graph node is a row selection operator, the terminal compares the column information corresponding to the row selection operator with the field names of each source data table, determines the target data table based on the comparison results, further generates target row selection information for the target data table based on the row selection operator, and finally records the target row selection information through the data protocol to update the data protocol. Since the target row selection information for the target data table can be generated based on the row selection operator and recorded through the data protocol, the data range during the row selection operation can be reduced, and the execution efficiency of row selection is greatly improved.

[0104] In some embodiments, when the graph node is a column selection operator; determining the target data table corresponding to the computation graph based on the field names of each source data table, and generating target data transformation information for the target data table based on the computation graph includes: comparing the column information corresponding to the column selection operator with the field names of each source data table, and determining the target data table based on the comparison result; generating target column selection information for the target data table based on the column selection operator; the target column selection information is used to characterize the corresponding column selection operation performed on the target data table; recording the target data transformation information through a data protocol to update the data protocol includes: recording the target column selection information through a data protocol to update the data protocol; representing the splicing result obtained by splicing the various source data tables through the generated data protocol also includes: performing a column selection operation on the wide table view.

[0105] Specifically, since column selection operators describe column selection operations, which refer to selecting several columns from a data table, the column selection operator includes column information, which serves as the basis for selection in the column selection operation. A column selection operator can be a select operator, such as SELECT (col1, col2), where the column information included is "col1" and "col2".

[0106] The terminal compares the column information corresponding to the column selection operator with the field names of each source data table. Based on the comparison results, the target data table is determined, and the column selection operator is applied to the target data table to generate target column selection information for the target data table. The target column selection information is used to represent the corresponding column selection operation performed on the target data table. After the column selection operation is actually completed, the target data table is replaced with the selection result corresponding to the column selection operation to update the target data table.

[0107] For example, such as Figure 6 As shown, if the column selection operator needs to select data related to amount, and the data related to amount exists in Table 2, then it is only necessary to select the columns related to amount in Table 2. At this time, the terminal can apply the column selection operator to Table 2, that is, generate target column selection information for Table 2 according to the column selection operator. This target column selection information is used to describe selecting the columns related to amount from Table 2 and deleting the data in other columns of Table 2.

[0108] Furthermore, since the target column selection information needs to be applied to the target data table, the terminal can record this information through the data protocol. At this point, the data protocol is updated. When data is subsequently written, the terminal can send the information recorded in the updated data protocol to the server. The server can then perform a column selection operation on the target data table based on the target column selection information recorded in the data protocol to update the target data table. In some specific embodiments, the terminal can record the target column selection information through the logic layer of the data protocol to update the logic layer.

[0109] It is understandable that after the column selection operation is applied to the target data table, the data in other columns will be deleted except for the columns selected by the column selection operation. Accordingly, in order for the data protocol to better represent the splicing result, the terminal needs to perform the same column selection operation on the wide table view of the data protocol to delete the same columns from the wide table view of the data protocol.

[0110] In the above embodiments, since target column selection information for the target data table can be generated according to the column selection operator and the target column selection information is recorded through the data protocol, the data range during column selection operations can be reduced, and the execution efficiency of column selection can be greatly improved. At the same time, since the column selection operation corresponding to the column selection operator is performed on the wide table view, the terminal can better represent the splicing result through the data protocol, allowing users to use the data protocol like a normal wide table, thereby improving the user experience.

[0111] In some embodiments, when the graph node is a column generation operator; determining the target data table corresponding to the computation graph based on the field names of each source data table, and generating target data transformation information for the target data table based on the computation graph includes: comparing the column information corresponding to the column generation operator with the field names of each source data table, and determining the target data table based on the comparison result; generating target column generation information for the target data table based on the column generation operator; the target column generation information is used to characterize the corresponding column generation operation performed on the target data table; recording the target data transformation information through the data protocol to update the data protocol includes: recording the target column generation information through the data protocol to update the data protocol; representing the splicing result obtained by splicing the various source data tables through the generated data protocol also includes: performing a column generation operation on the wide table view.

[0112] Specifically, column generation operators describe column generation operations, which refer to generating new columns based on existing columns. The column generation operator includes column information, which serves as the basis for the column generation operation. A specific column generation operator can be the NewCol operator, which carries both the column information of the existing columns and the column information of the newly generated column. It should be noted that the existing columns can be columns from the same data table or columns from different data tables.

[0113] The terminal compares the column information corresponding to the column generation operator with the field names of each source data table. Based on the comparison results, the target data table is determined. When there is only one target data table, the column generation operator is applied to the target data table to generate the target column generation information for the target data table. The target column generation information is used to characterize the corresponding column generation operation performed on the target data table. After the column generation operation is actually completed, a new column will be added to the target data table.

[0114] Furthermore, since the target column generation information needs to be applied to the target data table, the terminal can record this target column generation information through the data protocol. At this point, the data protocol is updated. When data is subsequently written, the terminal can send the information recorded in the updated data protocol to the server. The server can then perform column generation operations on the target data table based on the target column generation information recorded in the data protocol to update the target data table. In some specific embodiments, the terminal can record the target column generation information through the logic layer of the data protocol to update the logic layer.

[0115] It is understandable that since the column generation operation will generate new columns after being applied to the target data table, the terminal needs to perform the same column generation operation on the wide table view of the data protocol in order to make the data protocol better represent the splicing result. That is, the same column (which only includes the column name and does not include the actual data) needs to be generated in the same position in the wide table view of the data protocol.

[0116] In some embodiments, when the target data tables involved in the column generation operator include multiple target data tables, generating target column generation information for the target data tables based on the initial column generation information corresponding to the column generation operator includes: determining logical concatenation information corresponding to the multiple target data tables; generating column generation information for the target concatenation data tables corresponding to the logical concatenation information based on the column generation operator; and determining the logical concatenation information and the column generation information of the target concatenation data tables as the target column generation information for the multiple target data tables.

[0117] The logical concatenation information refers to the information used to represent concatenating multiple target data tables. It can be understood that these target data tables are not actually concatenated at this stage. For example, the logical concatenation information could be `New_tableN=table1join table2 on table1.id= table2.id`. The target concatenation data table corresponding to the logical concatenation information refers to the data table obtained by concatenating these target data tables.

[0118] Specifically, the terminal determines the logical concatenation information corresponding to multiple target data tables, and generates column generation information for the target concatenation data table corresponding to the logical concatenation information according to the column generation operator. This column generation information is applied to the target concatenation data table to represent the generation of new columns from multiple columns in the target concatenation data table.

[0119] For example, such as Figure 7 As shown, suppose the column generation operator needs to generate a new column to represent the average daily transaction amount based on the transaction amount and the number of transaction days. The number of transaction days exists in Table 1, and the transaction amount exists in Table 2. To obtain the average daily transaction amount, it is necessary to first concatenate Table 1 and Table 2 to obtain Table 1-2, and then divide the amount by the number of days to obtain the average daily transaction amount as the new column. At this time, the terminal generates concatenation logic information to concatenate Table 1 and Table 2, and applies the column generation operation to the concatenated Table 1-2, that is, it generates target column generation information for Table 1-2. This target column generation information is used to instruct that the average daily transaction amount column be generated in Table 1-2 based on the transaction day column and the transaction amount column. At the same time, the average daily transaction amount column needs to be added in the wide table view.

[0120] In the above embodiments, when the graph node is a column generation operator, since the target column generation information of the target data table can be generated according to the column generation operator and the target column generation information is recorded through the data protocol, the column generation operation can be responded to before the wide table is actually spliced, delaying the wide table splicing as much as possible and improving the execution efficiency. At the same time, since the column generation operation corresponding to the column generation operator is executed on the wide table view, the terminal can better represent the splicing result through the data protocol, allowing the user to use the data protocol like a normal wide table, thereby improving the user experience.

[0121] In some embodiments, the source data tables to be concatenated form a source data table set. After concatenating the source data tables using the generated data protocol representation, the method further includes: sending a data persistence request to the server; the data persistence request carries metadata of each source data table, target data transformation information, and data persistence mode information; the target data transformation information is generated by converting the initial data transformation information received by the wide table view of the data protocol for the concatenated result into source data tables in the source data table set; the metadata of each source data table includes the location information of each source data table; the server performs the following steps: performing a data transformation operation on the target source data table according to the target data transformation information and the location information of the target source data table corresponding to the target data transformation information to update the source data table set; and performing a data persistence operation on the updated source data table set according to the data persistence mode information.

[0122] The data persistence mode information refers to the data persistence mode specified by the user. There are two data persistence modes: strict mode and non-strict model. In strict mode, the source data tables are required to be actually concatenated, while in non-strict mode, the source data tables do not need to be actually concatenated.

[0123] Specifically, the terminal sends a data landing request to the server. The data landing request carries the metadata of each source data table, the target data transformation information, and the data landing mode information. After receiving the data landing request, the server parses the data landing request, obtains the metadata, target data transformation information, and data landing mode information carried in it, locates the corresponding source data table in the data warehouse according to the target data transformation information and the location information recorded in the metadata of each data table, and performs data transformation operations on the source data table to update the source data table set.

[0124] Specifically, the target data transformation information is generated by converting the initial data transformation information for the splicing result received by the wide table view of the data protocol into source data tables in the source data table set. Specifically, the terminal receives the initial data transformation information for the splicing result through the wide table view corresponding to the data protocol, determines the target data table corresponding to the initial data transformation information according to the field names of each source data table, and generates target data transformation information for the target data table based on the initial data transformation information.

[0125] In some embodiments, the target data transformation information includes at least one of target row selection information, target column selection information, and target column generation information. When the server receives target row selection information, it performs a row selection operation on the target data table corresponding to the target row selection information to update the target data table. When the server receives target column selection information, it performs a column selection operation on the target data table corresponding to the target column selection information to update the target data table. When the server receives target column generation information, it performs a column generation operation on the target data table corresponding to the target column generation information to update the target data table. For details, please refer to the description in the above embodiments; further elaboration is not required here.

[0126] Furthermore, the server determines whether the data persistence mode is strict or non-strict based on the data persistence mode information. If the data persistence mode is strict, the data persistence operation is performed on the updated source data table set according to the strict mode; if the data persistence mode is non-strict, the data persistence operation is performed on the updated source data table set according to the non-strict mode.

[0127] In the above embodiments, after the terminal sends a data landing request to the server, the server first performs a data transformation operation based on the target data transformation information in the data landing request, and then performs a data landing operation based on the data landing mode information carried in the data landing request. Since the source data to be spliced ​​is optimized through the data transformation operation, the execution efficiency of the data landing operation can be improved.

[0128] In some embodiments, when the data storage mode information is the first preset information, the server stores each source data table in the updated source data table set.

[0129] The first preset information is used to characterize the data landing mode as a non-strict mode. The first preset information can be identification information used to characterize the non-strict mode, for example, the identification information can be "0".

[0130] In non-strict mode, the source data tables do not need to be actually concatenated. The server can directly retrieve and store the updated set of source data tables based on their respective location information. In practice, the server can write each source data table back to any of the following: a data warehouse, a single-machine disk, a network cloud drive, a distributed storage system like Hadoop, etc.

[0131] In some embodiments, the server is further configured to perform the following steps: when the data persistence mode information is the second preset information, the server obtains each source data table in the updated source data table set according to the metadata of each source data table, and groups each source data table in the updated source data table set to obtain multiple data table groups; the server concatenates the data tables in each data table group to obtain the current concatenated data table for each data table group; the server groups each currently concatenated data table to obtain multiple data table groups; the server repeatedly enters the step of concatenating the data tables in each data table group until a target wide table containing all source data in the updated source data table set is obtained; the server stores the target wide table.

[0132] The second preset information is used to characterize the data landing mode as strict mode. The second preset information can be identification information used to characterize strict mode, for example, the identification information can be "1".

[0133] In strict mode, the source data tables need to be physically concatenated. The server can then retrieve the updated set of source data tables based on their respective metadata, and concatenate them to obtain the target wide table. In practice, the server can write each source data table back to any of the following: a data warehouse, a single-machine disk, a network cloud disk, a distributed storage system like Hadoop, etc.

[0134] Specifically, since the number of data tables in the source data table set is often very large, directly concatenating them one by one is very time-consuming. Therefore, in this embodiment, the server can first group the various source data tables in the updated source data table set to obtain multiple data table groups. For each data table group, the server concatenates the data tables in the data table group to obtain the concatenated data table corresponding to the data table group. The server further groups the concatenated data table to obtain multiple data table groups, and concatenates the data groups in these data table groups again, and so on, until only one data table remains. This data table contains the data of all the source data in the updated source data table set, and this data table is the target wide table.

[0135] For example, suppose there are 100 data tables that need to be concatenated. These 100 data tables can be divided into 10 groups, with each group containing 10 data tables. The 10 data tables in each group are concatenated separately to obtain 10 concatenated data tables. Then, these 10 concatenated data tables are divided into two groups, with five data tables in each group being concatenated separately to obtain 2 concatenated data tables. Finally, these 2 concatenated data tables are concatenated to obtain the target wide table.

[0136] In the above embodiments, the server can improve the efficiency of data table splicing by grouping and splicing the source data tables in the updated source data table set.

[0137] In some embodiments, such as Figure 8 As shown, a data table processing method is provided, which can be applied to the above-mentioned... Figure 1 Taking the terminal in the example, the method specifically includes the following steps:

[0138] Step 802: Obtain the generated data protocol.

[0139] The data protocol is generated based on the metadata and wide table view of each source data table to be concatenated. The metadata of each source data table includes at least the field names of each source data table. The wide table view is obtained by organizing the field names into a data table according to the acquired concatenation method information.

[0140] Step 804: Receive the initial data transformation information for the splicing result through the wide table view corresponding to the data protocol.

[0141] In some embodiments, the initial data transformation information can be a computation graph; the computation graph includes at least two graph nodes; the graph nodes are any one of row selection operators, column selection operators, and column generation operators.

[0142] Step 806: Determine the target data table corresponding to the initial data transformation information based on the field names of each source data table, and generate target data transformation information for the target data table based on the initial data transformation information.

[0143] The target data transformation information is used to characterize the corresponding data transformations performed on the target data table. Data transformation operations refer to operations that transform data in the data table, including but not limited to row selection operations, column selection operations, and column generation operations. Specifically, row selection operations involve selecting several rows from a data table, or filtering rows based on conditions, such as selecting users whose registration date is greater than 2020; column selection operations involve selecting several columns from a data table, such as selecting columns related to monetary amounts; and column generation operations involve creating new columns based on existing columns in a data table, such as calculating the average amount for the first quarter by averaging the amounts from January, February, and March, and using this average as the new column.

[0144] Specifically, the terminal can determine the field name of the data involved in the transformation operation based on the identification information carried in the initial data transformation information, and then match this field name with the field names in the metadata of each source data table. Based on the matching result, the terminal can determine the source data table where the data involved in the transformation operation resides, and then identify this source data table as the target data table. It can be understood that there can be one or more target data tables, with at least two. For example, the terminal can determine that the field name of the data involved in the transformation operation is "gender" based on the identification information carried in the initial data transformation information. If the metadata of source data table 3 includes the field name "gender," then the terminal can successfully match the metadata of source data table 3, and thus identify source data table 3 as the target data table.

[0145] The terminal can generate target data transformation information for the target data table based on the initial data transformation information. The generated target data transformation information carries the target data table identifier and is used to characterize the corresponding data transformation performed on the target data table.

[0146] Step 808: Record the target data transformation information through the data protocol.

[0147] In some embodiments, the data protocol includes a logical layer and a user layer, wherein the user layer provides a wide table view, and the logical layer records metadata about each source data table. After obtaining target data transformation information for the target data table, the terminal can record this target data transformation information through the logical layer of the data protocol to update the data protocol.

[0148] Step 810: When splicing the various source data tables, the metadata of each source data table and the target data transformation information recorded therein are sent to the server; the server is used to perform data transformation operations based on the metadata of each source data table and the target data transformation information, and splice the various source data tables to obtain the target wide table.

[0149] Specifically, when splicing the various source data tables, the terminal sends the metadata of each data table and all the target data transformation information to the server. After receiving this information, the server locates the corresponding source data table from the data warehouse based on the target data transformation information and the location information recorded in the metadata of each data table, and performs data transformation operations on these source data tables to update them. After the update is completed, the server splices the data tables to be spliced ​​to obtain the target wide table.

[0150] In some embodiments, the target data transformation information includes at least one of target row selection information, target column selection information, and target column generation information. When the server receives target row selection information, it performs a row selection operation on the target data table corresponding to the target row selection information to update the target data table. When the server receives target column selection information, it performs a column selection operation on the target data table corresponding to the target column selection information to update the target data table. When the server receives target column generation information, it performs a column generation operation on the target data table corresponding to the target column generation information to update the target data table. For details, please refer to the descriptions in the preceding embodiments; these will not be repeated here.

[0151] For further detailed descriptions of steps 802-810 in this embodiment, please refer to the descriptions in the preceding embodiments; these will not be repeated here.

[0152] The above data table processing method obtains a generated data protocol, which is generated based on the metadata and wide table view of each source data table to be concatenated. The metadata of each source data table includes at least its respective field names. The wide table view is obtained by organizing the field names into a data table according to the obtained concatenation method information. The method then receives initial data transformation information for the concatenation result through the wide table view corresponding to the data protocol. Based on the metadata of each source data table, it determines the target data table corresponding to the initial data transformation information and generates target data transformation information for the target data table based on the initial data transformation information. The information is used to characterize the corresponding data transformation performed on the target data table. The target data transformation information is recorded through the data protocol. When splicing the various source data tables, the metadata of each source data table and the recorded target data transformation information are sent to the server. The server performs data transformation operations based on the metadata of each source data table and the target data transformation information, and splices the various source data tables to obtain the target wide table. Thus, by using the data protocol to represent the actual spliced ​​wide table, the splicing of the wide table can be delayed as much as possible. By converting the data transformations applied to the wide table to those applied to the source data tables, the amount of data during wide table splicing can be reduced as much as possible, thereby improving the execution efficiency of wide table splicing.

[0153] In some embodiments, before step 802, i.e., obtaining the generated data protocol, the terminal generates the data protocol through the following steps: obtaining the metadata and splicing method information of each source data table to be spliced; the metadata of each source data table includes at least the field names of each source data table; organizing the field names into a data table according to the splicing method information to obtain the corresponding wide table view; constructing a logic layer based on the metadata of each source data table; constructing a user layer based on the wide table view; and generating the data protocol based on the logic layer and the user layer.

[0154] In some embodiments, the source data tables to be concatenated form a source data table set; the data processing method further includes: sending a data landing request to the server; the data landing request carries the metadata of each source data table, target data transformation information, and data landing mode information; the target data transformation information is generated by converting the initial data transformation information received by the wide table view of the data protocol for the concatenation result into source data tables in the source data table set; the metadata of each source data table includes the location information of each source data table; the server performs the following steps: according to the target data transformation information and the location information of the target source data table corresponding to the target data transformation information, performs a data transformation operation on the target source data table to update the source data table set; when the data landing mode information is the first preset information, the server obtains and stores each source data table in the updated source data table set according to the location information of each source data table.

[0155] In some embodiments, the server is further configured to perform the following steps: when the data persistence mode information is the second preset information, the server obtains each source data table in the updated source data table set according to the metadata of each source data table, and groups each source data table in the updated source data table set to obtain multiple data table groups; the server concatenates the data tables in each data table group to obtain the current concatenated data table for each data table group; the server groups each currently concatenated data table to obtain multiple data table groups; the server repeatedly enters the step of concatenating the data tables in each data table group until a target wide table containing all source data in the updated source data table set is obtained; the server stores the target wide table.

[0156] This application also provides an application scenario in which the above-mentioned data table processing method is applied. In this application scenario, user features need to be used as training samples to train the model. The user features are distributed across hundreds of subject tables (such as identity traits, payment behavior, credit history, etc.). Before training the model, the data from these data tables need to be aggregated into one table, i.e., wide table concatenation.

[0157] In related technologies, the splicing is based on the Apache Spark computing framework, such as... Figure 8AThe diagram shows a flowchart of the data concatenation process based on the Apache Spark computing framework. The Spark framework first reads the metadata of the source data table to be concatenated, generates a concatenation computation task in memory, performs computation on the concatenation task, and finally writes it back. However, since wide tables often have more than 100,000 dimensions, the Spark Driver is prone to memory overflow errors. Furthermore, for inexperienced developers, it may be necessary to first concatenate a wide table with billions of rows and then filter the feature data of millions of users. Concatenating a wide table with billions of rows may take several weeks. However, the data table processing method provided in this application embodiment can represent the wide table without memory overflow and automatically optimizes the process by first filtering millions of users and then concatenating the wide table, saving time.

[0158] Specifically, this data table processing method is applied in this application scenario as follows:

[0159] (a) Generate Data Protocol

[0160] 1.1 Obtain the metadata and splicing method information of each source data table to be spliced; the metadata of each source data table shall include at least the field names of each source data table.

[0161] 1.2. Organize the field names into a data table according to the concatenation method information to obtain the corresponding wide table view.

[0162] 1.3 Construct a logical layer based on the metadata of each source data table, construct a user layer based on a wide table view, and generate a data protocol based on the logical layer and the user layer.

[0163] (ii) Receive the computation graph and update the data protocol using the computation graph.

[0164] 2.1 Receive the computation graph for the splicing result through the wide table view corresponding to the data protocol; wherein, the computation graph includes at least two graph nodes; the graph node is any one of row selection operator, column selection operator and column generation operator.

[0165] 2.2 The optimizer transforms the data transformation process described by the operators declared by the user through the computation graph into optimal execution logic:

[0166] 2.2-1 When the graph node is a row selection operator; the target data table corresponding to the computation graph is determined based on the field names of each source data table, and target data transformation information for the target data table is generated based on the computation graph, including: comparing the column information corresponding to the row selection operator with the field names of each source data table, and determining the target data table based on the comparison result; generating target row selection information for the target data table based on the row selection operator; the target row selection information is used to characterize the corresponding row selection operation performed on the target data table; and the target data transformation information is recorded through the data protocol to update the data protocol, including: recording the target row selection information through the data protocol to update the data protocol.

[0167] 2.2-2. When the graph node is a column selection operator; the target data table corresponding to the computation graph is determined based on the field names of each source data table, and target data transformation information for the target data table is generated based on the computation graph, including: comparing the column information corresponding to the column selection operator with the field names of each source data table, and determining the target data table based on the comparison result; generating target column selection information for the target data table based on the column selection operator; the target column selection information is used to characterize the corresponding column selection operation performed on the target data table; recording the target data transformation information through the data protocol to update the data protocol includes: recording the target column selection information through the data protocol to update the data protocol; representing the splicing result obtained by splicing the various source data tables through the generated data protocol also includes: performing column selection operation on the wide table view.

[0168] 2.2-3. When the graph node is a column generation operator; the target data table corresponding to the computation graph is determined based on the field names of each source data table, and the target data transformation information for the target data table is generated based on the computation graph, including: comparing the column information corresponding to the column generation operator with the field names of each source data table, and determining the target data table based on the comparison result; generating target column generation information for the target data table based on the column generation operator; the target column generation information is used to characterize the corresponding column generation operation performed on the target data table; recording the target data transformation information through the data protocol to update the data protocol includes: recording the target column generation information through the data protocol to update the data protocol; the generated data protocol represents the splicing result obtained by splicing the various source data tables, and also includes: performing column generation operations on the wide table view.

[0169] When there are multiple target data tables, generating target column generation information for the target data tables based on the initial column generation information corresponding to the column generation operator includes: determining logical concatenation information corresponding to multiple target data tables; generating column generation information for the target concatenated data tables corresponding to the logical concatenation information based on the column generation operator; and determining the logical concatenation information and the column generation information of the target concatenated data tables as the target column generation information for multiple target data tables.

[0170] (iii) After the optimizer completes the optimization, the resulting data protocol is updated. For subsequent analysis, the data needs to be persisted. Depending on the requirements of the downstream tasks, different persistence methods can be used. The terminal can send a data persistence request to the server, carrying logical layer record information and data persistence mode information. The server then performs the data persistence operation based on this request.

[0171] 1.1 If the data persistence mode information indicates that the persistence mode is non-strict, that is, the source data table is allowed to be written back without concatenation, then the server can perform data transformation operations on the corresponding source data table according to the position information in the metadata of the logical layer of the data protocol and all the target data transformation information recorded, so as to update the corresponding source data table. After the execution is completed, all source data tables are written back directly.

[0172] 1.2 If the data landing mode is a strict mode, requiring actual table concatenation, the server can perform data transformation operations on the corresponding source data table based on the location information in the metadata of the logical layer of the data protocol and all recorded target data transformation information to update the corresponding source data table. After the execution is completed, the server retrieves all concatenated source data tables from the data warehouse based on the metadata of each source data table, groups and concatenates all source data tables to obtain the target wide table, and writes the obtained target wide table back.

[0173] Specifically, the server groups and concatenates the source data table using the following steps:

[0174] 1.2-1 The server groups the source data tables in the updated source data table set to obtain multiple data table groups.

[0175] 1.2-2. The server concatenates the data tables in each data table group to obtain the current concatenated data table for each data table group.

[0176] 1.2-3. The server groups the currently assembled data tables into multiple data table groups.

[0177] 1.2-4. The server repeats steps 1.2-2-1.2-3 until only one data table remains, resulting in a target wide table containing all source data from the updated source data table set.

[0178] In some embodiments, as shown in Table 5, a comparison of the performance of the data table processing method (computation graph) provided in this application embodiment and the traditional method (Spark) is presented. Table 5 shows that in a scenario involving the concatenation of 360 data tables, the traditional method fails to run, while the data table processing method provided in this application embodiment successfully represents the wide table through a data protocol and persists the concatenated wide table by writing it back. In a scenario involving the concatenation of multiple partitions of a monthly table, the data table processing method provided in this application embodiment also shows a significant efficiency advantage compared to the traditional method. Therefore, the data table processing method provided in this application embodiment expands the number of wide table columns supported by the data warehouse system and improves the execution efficiency of wide table concatenation.

[0179] Table 5

[0180]

[0181] It should be understood that, although Figure 2-7 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2-7 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0182] In some embodiments, such as Figure 9 As shown, a data table processing device 900 is provided. This device can be a software module, a hardware module, or a combination of both as part of a computer device. Specifically, the device includes:

[0183] The information acquisition module 902 is used to acquire the metadata and splicing method information of each source data table to be spliced; the metadata of each source data table includes at least the field names of each source data table.

[0184] The data splicing module 904 is used to organize the various field names into a data table according to the splicing method information to obtain the corresponding wide table view;

[0185] The data protocol generation module 906 is used to generate a data protocol based on the metadata and wide table view of each source data table.

[0186] The splicing result representation module 908 is used to represent the splicing result obtained by splicing the various source data tables through the generated data protocol.

[0187] The aforementioned data table processing device acquires the metadata and concatenation method information of each source data table to be concatenated. The metadata of each source data table includes at least the field names of each source data table. It then organizes the field names into a data table according to the concatenation method information to obtain a corresponding wide table view. Based on the metadata and wide table view of each source data table, a data protocol is generated. This generated data protocol represents the concatenation result obtained by concatenating the source data tables. Since the data protocol only needs to concatenate the field names of the source data tables during generation, rather than all the data in the source data tables, it greatly saves computer memory and effectively avoids memory overflow caused by directly concatenating all the data from each source data table.

[0188] In some embodiments, the data protocol generation module is further configured to construct a logical layer based on the metadata of each source data table; construct a user layer based on a wide table view; and generate a data protocol based on the logical layer and the user layer.

[0189] In some embodiments, the splicing result representation module is further configured to receive initial data transformation information for the splicing result through a wide table view corresponding to the data protocol; determine the target data table corresponding to the initial data transformation information based on the field names of each source data table, and generate target data transformation information for the target data table based on the initial data transformation information; the target data transformation information is used to characterize the corresponding data transformation performed on the target data table; and the target data transformation information is recorded through the data protocol to update the data protocol.

[0190] In some embodiments, the splicing result representation module is further configured to receive a computation graph for the splicing result through a wide table view corresponding to the data protocol; wherein, the computation graph includes at least two graph nodes; the graph node is any one of row selection operator, column selection operator, and column generation operator; determining the target data table corresponding to the initial data transformation information based on the field names of each source data table, and generating target data transformation information for the target data table based on the initial data transformation information includes: determining the target data table corresponding to the computation graph based on the field names of each source data table, and generating target data transformation information for the target data table based on the computation graph.

[0191] In some embodiments, the concatenation result representation module is further configured to compare the column information corresponding to the row selection operator with the field names of each source data table, determine the target data table based on the comparison result; generate target row selection information for the target data table based on the row selection operator; the target row selection information is used to characterize the corresponding row selection operation performed on the target data table; and record the target row selection information through the data protocol to update the data protocol.

[0192] In some embodiments, the concatenation result representation module is further configured to compare the column information corresponding to the column selection operator with the field names of each source data table, determine the target data table based on the comparison result; generate target column selection information for the target data table based on the column selection operator; the target column selection information is used to characterize the corresponding column selection operation performed on the target data table; record the target column selection information through the data protocol to update the data protocol; and perform column selection operation on the wide table view.

[0193] In some embodiments, the concatenation result representation module is further used to compare the column information corresponding to the column generation operator with the field names of each source data table, and determine the target data table based on the comparison result; generate target column generation information for the target data table based on the column generation operator; the target column generation information is used to characterize the corresponding column generation operation performed on the target data table; record the target column generation information through the data protocol to update the data protocol; and perform column generation operation on the wide table view.

[0194] In some embodiments, the splicing result representation module is further configured to determine logical splicing information corresponding to multiple target data tables; generate column generation information for the target splicing data table corresponding to the logical splicing information according to the column generation operator; and determine the logical splicing information and the column generation information of the target splicing data table as the target column generation information for multiple target data tables.

[0195] In some embodiments, the source data tables to be spliced ​​form a source data table set; the above apparatus further includes: a sending module, used to send a data landing request to the server; the data landing request carries the metadata of each source data table, target data transformation information, and data landing mode information; the target data transformation information is generated by converting the initial data transformation information received by the wide table view of the data protocol for the splicing result into source data tables in the source data table set; the metadata of each source data table includes the location information of each source data table; the server is used to perform data transformation operations on the target source data table according to the target data transformation information and the location information of the target source data table corresponding to the target data transformation information, so as to update the source data table set; when the data landing mode information is the first preset information, the server obtains each source data table in the updated source data table set according to the location information of each source data table and stores it.

[0196] In some embodiments, when the data persistence mode information is the second preset information, the server is further configured to: obtain each source data table in the updated source data table set according to the metadata of each source data table; group each source data table in the updated source data table set to obtain multiple data table groups; concatenate the data tables in each data table group to obtain the current concatenated data table for each data table group; group each currently concatenated data table to obtain multiple data table groups; repeat the step of concatenating the data tables in each data table group until a target wide table containing all source data in the updated source data table set is obtained; and store the target wide table.

[0197] In some embodiments, such as Figure 10 As shown, a data table processing device 1000 is provided. This device can be a software module, a hardware module, or a combination of both as part of a computer device. Specifically, the device includes:

[0198] The data protocol acquisition module 1002 is used to acquire the generated data protocol; wherein, the data protocol is generated based on the metadata and wide table view of each source data table to be concatenated; the metadata of each source data table includes at least the field names of each source data table; the wide table view is obtained by organizing the field names into a data table according to the acquired concatenation method information;

[0199] The information receiving module 1004 is used to receive initial data transformation information for the splicing result through the wide table view corresponding to the data protocol;

[0200] The information transformation module 1006 is used to determine the target data table corresponding to the initial data transformation information based on the field names of each source data table, and to generate target data transformation information for the target data table based on the initial data transformation information; the target data transformation information is used to characterize the corresponding data transformation performed on the target data table.

[0201] Information recording module 1008 is used to record target data transformation information through a data protocol;

[0202] The information sending module 1010 is used to send the metadata of each source data table and the target data transformation information recorded in each source data table to the server when splicing the source data tables. The server is used to perform data transformation operations according to the metadata of each source data table and the target data transformation information, and splice the source data tables to obtain the target wide table.

[0203] The aforementioned data table processing device acquires a generated data protocol, which is generated based on the metadata and wide table view of each source data table to be concatenated. The metadata of each source data table includes at least the field names of each source data table. The wide table view is obtained by organizing the field names into a data table according to the acquired concatenation method information. The device then receives initial data transformation information for the concatenation result through the wide table view corresponding to the data protocol. Based on the metadata of each source data table, it determines the target data table corresponding to the initial data transformation information and generates target data transformation information for the target data table based on the initial data transformation information. This is used to characterize the corresponding data transformation performed on the target data table. The target data transformation information is recorded through a data protocol. When splicing the various source data tables, the metadata of each source data table and the recorded target data transformation information are sent to the server. The server is used to perform data transformation operations based on the metadata of each source data table and the target data transformation information, and splices the various source data tables to obtain the target wide table. Thus, by using a data protocol to represent the actual spliced ​​wide table, the splicing of the wide table can be delayed as much as possible. By converting the data transformations applied to the wide table to those applied to the source data tables, the amount of data during wide table splicing can be reduced as much as possible, thereby improving the execution efficiency of wide table splicing.

[0204] For specific limitations regarding the data table processing device, please refer to the limitations on the data table processing method above, which will not be repeated here. Each module in the aforementioned data table processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0205] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 11As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a data table processing method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0206] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0207] In some embodiments, such as the data table processing method disclosed in this application, the various source data tables to be concatenated may be stored on a blockchain.

[0208] In some embodiments, such as the data table processing method or apparatus disclosed in this application, multiple servers may form a blockchain, and the servers involved in the embodiments of this application are nodes on the blockchain.

[0209] In some embodiments, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0210] In some embodiments, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0211] In some embodiments, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above-described method embodiments.

[0212] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0213] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0214] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A data table processing method, characterized in that, The method includes: Obtain the metadata and splicing method information of each source data table to be spliced; the metadata of each source data table shall include at least the field names of each source data table. Organize each of the field names into a data table according to the concatenation method information to obtain the corresponding wide table view; A logical layer is constructed based on the metadata of each of the aforementioned source data tables; Construct a user layer based on the wide table view; A data protocol is generated based on the logical layer and the user layer; The generated data protocol represents the concatenation result obtained by concatenating the various source data tables.

2. The method according to claim 1, characterized in that, The concatenation result obtained by concatenating the various source data tables through the generated data protocol includes: The initial data transformation information for the splicing result is received through the wide table view corresponding to the data protocol; The target data table corresponding to the initial data transformation information is determined based on the field names of each of the source data tables, and target data transformation information for the target data table is generated based on the initial data transformation information; the target data transformation information is used to characterize the corresponding data transformation performed on the target data table. The target data transformation information is recorded through the data protocol in order to update the data protocol.

3. The method according to claim 2, characterized in that, Receiving initial data transformation information for the splicing result through the wide table view corresponding to the data protocol includes: The calculation graph for the splicing result is received through the wide table view corresponding to the data protocol; The computation graph includes at least two graph nodes; each graph node is any one of a row selection operator, a column selection operator, and a column generation operator. The step of determining the target data table corresponding to the initial data transformation information based on the field names of each of the source data tables, and generating target data transformation information for the target data table based on the initial data transformation information, includes: The target data table corresponding to the computation graph is determined based on the field names of each of the source data tables, and target data transformation information for the target data table is generated based on the computation graph.

4. The method according to claim 3, characterized in that, When the graph node is a row selection operator; the step of determining the target data table corresponding to the computation graph based on the field names of each of the source data tables, and generating target data transformation information for the target data table based on the computation graph includes: The column information corresponding to the row selection operator is compared with the field names of each of the source data tables, and the target data table is determined based on the comparison results. Generate target row selection information for the target data table based on the row selection operator; The target row selection information is used to characterize the corresponding row selection operation performed on the target data table; The step of recording the target data transformation information through the data protocol to update the data protocol includes: The target row selection information is recorded through the data protocol to update the data protocol.

5. The method according to claim 3, characterized in that, When the graph node is a column selection operator; the step of determining the target data table corresponding to the computation graph based on the field names of each of the source data tables, and generating target data transformation information for the target data table based on the computation graph includes: The column information corresponding to the column selection operator is compared with the field names of each of the source data tables, and the target data table is determined based on the comparison results. Generate target column selection information for the target data table based on the column selection operator; The target column selection information is used to characterize the corresponding column selection operation performed on the target data table; The step of recording the target data transformation information through the data protocol to update the data protocol includes: The target column selection information is recorded through the data protocol in order to update the data protocol; The concatenation result obtained by concatenating the various source data tables through the generated data protocol representation also includes: Perform the column selection operation on the wide table view.

6. The method according to claim 3, characterized in that, When the graph node is a column generation operator; the step of determining the target data table corresponding to the computation graph based on the field names of each of the source data tables, and generating target data transformation information for the target data table based on the computation graph includes: The column information corresponding to the column generation operator is compared with the field names of each of the source data tables, and the target data table is determined based on the comparison results. Generate target column generation information for the target data table based on the column generation operator; The target column generation information is used to characterize the corresponding column generation operation performed on the target data table; The step of recording the target data transformation information through the data protocol to update the data protocol includes: The target column generation information is recorded through the data protocol to update the data protocol; The concatenation result obtained by concatenating the various source data tables through the generated data protocol representation also includes: Perform the column generation operation on the wide table view.

7. The method according to claim 6, characterized in that, The target data table includes multiple tables; the step of generating target column generation information for the target data table based on the column generation operator includes: Determine the logical concatenation information corresponding to the multiple target data tables; Generate column generation information for the target concatenated data table corresponding to the logical concatenation information based on the column generation operator; The logical concatenation information and the column generation information of the target concatenation data table are determined as the target column generation information of multiple target data tables.

8. The method according to claim 1, characterized in that, The various source data tables to be spliced ​​together form a source data table set; After the concatenation result obtained by concatenating the various source data tables through the generated data protocol representation, the method further includes: Send a data landing request to the server; the data landing request carries the metadata of each of the source data tables, the target data transformation information and the data landing mode information; the target data transformation information is generated by converting the initial data transformation information received by the wide table view of the data protocol for the splicing result into the source data tables in the set of source data tables. The metadata of each source data table includes the location information of each source data table; the server is used to perform the following steps: Based on the target data transformation information and the location information of the target source data table corresponding to the target data transformation information, a data transformation operation is performed on the target source data table to update the source data table set; When the data landing mode information is the first preset information, the server obtains and stores each source data table in the updated source data table set according to the location information of each source data table.

9. The method according to claim 8, characterized in that, The server is also used to perform the following steps: When the data landing mode information is the second preset information, the server obtains each source data table in the updated source data table set according to the metadata of each source data table, and groups each source data table in the updated source data table set to obtain multiple data table groups. The server concatenates the data tables in each data table group to obtain the current concatenated data table for each data table group. The server groups the currently assembled data tables into multiple data table groups; The server repeatedly enters the step of splicing the data tables in each data table group until a target wide table containing all source data in the updated source data table set is obtained. The server stores the target wide table.

10. A data table processing method, characterized in that, The method includes: Retrieve the generated data protocol; The data protocol is generated based on the metadata and wide table view of each source data table to be concatenated; the metadata of each source data table includes at least the field names of each source data table; the wide table view is obtained by organizing the field names into a data table according to the acquired concatenation method information. The initial data transformation information for the splicing result is received through the wide table view corresponding to the data protocol; the splicing result is obtained by splicing the various source data tables. The target data table corresponding to the initial data transformation information is determined based on the field names of each of the source data tables, and target data transformation information for the target data table is generated based on the initial data transformation information; the target data transformation information is used to characterize the corresponding data transformation performed on the target data table. The target data transformation information is recorded through the data protocol; When splicing the various source data tables, the metadata of each source data table and the recorded target data transformation information are sent to the server. The server is used to perform data transformation operations based on the metadata of each of the source data tables and the target data transformation information, and to concatenate the source data tables to obtain a target wide table.

11. A data table processing device, characterized in that, The device includes: The information acquisition module is used to acquire the metadata and concatenation method information of each source data table to be concatenated; the metadata of each source data table includes at least the field names of each source data table. The data splicing module is used to organize each of the field names into a data table according to the splicing method information to obtain the corresponding wide table view; The data protocol generation module is used to construct a logical layer based on the metadata of each of the source data tables; construct a user layer based on the wide table view; and generate a data protocol based on the logical layer and the user layer. The splicing result representation module is used to represent the splicing result obtained by splicing the various source data tables through the generated data protocol.

12. The apparatus according to claim 11, characterized in that, The splicing result representation module is further configured to receive initial data transformation information for the splicing result through a wide table view corresponding to the data protocol; determine the target data table corresponding to the initial data transformation information based on the field names of each of the source data tables; and generate target data transformation information for the target data table based on the initial data transformation information; the target data transformation information is used to characterize the corresponding data transformation performed on the target data table. The target data transformation information is recorded through the data protocol in order to update the data protocol.

13. The apparatus according to claim 12, characterized in that, The splicing result representation module is also used to receive a computation graph for the splicing result through a wide table view corresponding to the data protocol; wherein, the computation graph includes at least two graph nodes; the graph node is any one of row selection operator, column selection operator and column generation operator; the target data table corresponding to the computation graph is determined according to the field names of each of the source data tables, and target data transformation information for the target data table is generated according to the computation graph.

14. The apparatus according to claim 13, characterized in that, When the graph node is a row selection operator, the splicing result representation module is also used to compare the column information corresponding to the row selection operator with the field names of each of the source data tables, determine the target data table based on the comparison result, and generate target row selection information for the target data table based on the row selection operator. The target row selection information is used to characterize the corresponding row selection operation performed on the target data table; the target row selection information is recorded through the data protocol to update the data protocol.

15. The apparatus according to claim 13, characterized in that, When the graph node is a column selection operator, the splicing result representation module is also used to compare the column information corresponding to the column selection operator with the field names of each of the source data tables, determine the target data table based on the comparison result, and generate target column selection information for the target data table based on the column selection operator. The target column selection information is used to characterize the corresponding column selection operation performed on the target data table; the target column selection information is recorded through the data protocol to update the data protocol; Perform the column selection operation on the wide table view.

16. The apparatus according to claim 13, characterized in that, When the graph node is a column generation operator, the splicing result representation module is also used to compare the column information corresponding to the column generation operator with the field names of each of the source data tables, determine the target data table based on the comparison result, and generate target column generation information for the target data table based on the column generation operator. The target column generation information is used to characterize the corresponding column generation operation performed on the target data table; The target column generation information is recorded through the data protocol to update the data protocol; Perform the column generation operation on the wide table view.

17. The apparatus according to claim 16, characterized in that, The target data tables include multiple ones; the splicing result representation module is also used to determine logical splicing information corresponding to the multiple target data tables; generate column generation information for the target splicing data tables corresponding to the logical splicing information according to the column generation operator; and determine the logical splicing information and the column generation information of the target splicing data tables as the target column generation information of the multiple target data tables.

18. The apparatus according to claim 11, characterized in that, The various source data tables to be spliced ​​together form a source data table set; the device further includes a sending module, which is used to send a data landing request to the server; the data landing request carries the metadata of each of the source data tables, target data transformation information, and data landing mode information; the target data transformation information is generated by converting the initial data transformation information received by the wide table view of the data protocol for the splicing result into the source data tables in the source data table set; the metadata of each source data table includes the location information of each source data table; the server is used to perform a data transformation operation on the target source data table according to the target data transformation information and the location information of the target source data table corresponding to the target data transformation information, so as to update the source data table set; When the data landing mode information is the first preset information, the server obtains and stores each source data table in the updated source data table set according to the location information of each source data table.

19. The apparatus according to claim 18, characterized in that, The server is also used to perform the following steps: When the data landing mode information is the second preset information, the server obtains each source data table in the updated source data table set according to the metadata of each source data table, and groups each source data table in the updated source data table set to obtain multiple data table groups. The server concatenates the data tables in each data table group to obtain the current concatenated data table for each data table group. The server groups the currently assembled data tables into multiple data table groups; The server repeatedly enters the step of splicing the data tables in each data table group until a target wide table containing all source data in the updated source data table set is obtained. The server stores the target wide table.

20. A data table processing device, characterized in that, The device includes: The data protocol acquisition module is used to acquire the generated data protocol; wherein, the data protocol is generated based on the metadata and wide table view of each source data table to be concatenated; the metadata of each source data table includes at least the field names of each source data table; the wide table view is obtained by organizing the field names into a data table according to the acquired concatenation method information; The information receiving module is used to receive initial data transformation information for the splicing result through the wide table view corresponding to the data protocol; the splicing result is obtained by splicing the various source data tables. The information transformation module is used to determine the target data table corresponding to the initial data transformation information based on the field names of each of the source data tables, and to generate target data transformation information for the target data table based on the initial data transformation information; the target data transformation information is used to characterize the corresponding data transformation performed on the target data table; The information recording module is used to record the target data transformation information through the data protocol; The information sending module is used to send the metadata of each of the source data tables and the recorded target data transformation information to the server when splicing the source data tables; the server is used to perform data transformation operations according to the metadata of each of the source data tables and the target data transformation information, and splice the source data tables to obtain the target wide table.

21. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.

22. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

23. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 10.

Images