A data table processing method, device, computer equipment and storage medium
By automatically building target configuration scripts and using Spark for data conversion, the problem of insufficient flexibility and scalability of data table processing methods in the existing technology in different business scenarios is solved, and an efficient and flexible data table processing process is achieved.
Patent Information
- Application Number
- CN202110358384.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-01
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-04-01
AI Technical Summary
The existing data table processing methods lack flexibility and scalability in different business scenarios, resulting in a high threshold for use of customized scripts and affecting processing efficiency.
By obtaining multiple original data tables, data acquisition, conversion and storage are collected, converted and stored based on acquisition, connection, configuration and storage instructions, target configuration scripts are automatically built, and field-level conversion and storage are used to abstract data ETL processes to achieve reuse and flexibility sex.
It improves the efficiency and flexibility of data table processing, adapts to the needs of different business scenarios, reduces dependence on customized scripts, and simplifies operational processes.
Smart Images

Figure CN114675812B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data table processing method, device, computer equipment and storage medium. Background Art
[0002] With the advent of the big data era, data processing has emerged in various business scenarios. Common data processing involves data extraction, transformation, and loading (Extract-Transform-Load, ETL). This processing flow can organize the data of scattered original data tables into target data tables of data warehouses or data marts according to business needs, which serves as the basis for data analysis and mining.
[0003] At present, for data ETL, which is a data flow with dependencies, Airflow scheduling can be used to complete it when related jobs are running. In this process, the data in the original data table is converted into the data in the target data table by adding business customized scripts to each task flow, corresponding to customized collection scripts, customized conversion scripts, and customized storage scripts. Obviously, for non-software developers (such as data miners), the threshold for using customized scripts is high. When facing data table processing in different business scenarios, it lacks flexibility and scalability. When the business scenario changes, the customized script needs to be manually changed, which affects the processing efficiency. Summary of the invention
[0004] In view of the above problems, the embodiments of the present application provide a data table processing method, apparatus, computer equipment and storage medium, which can improve the flexibility and scalability of generating target data tables in different business scenarios and effectively improve the efficiency of data table processing.
[0005] On the one hand, an embodiment of the present application provides a data table processing method, including:
[0006] Acquire multiple original data tables, and acquire data sets in each original data table based on the received acquisition instruction to obtain multiple original data sets;
[0007] The plurality of original data sets are connected based on the received connection instruction, and each of the original data sets after the connection is established is transformed to obtain a target data set;
[0008] Receiving configuration instructions for each data item in the target data set, and constructing a target configuration script based on the configuration instructions for each data item;
[0009] The target data set is processed based on the target configuration script, and the processed target data set is stored in a target data table based on the received storage instruction.
[0010] An embodiment of the present application provides a data table processing device, including:
[0011] The acquisition module is used to obtain multiple original data tables, and based on the received acquisition instructions, acquire the data sets in each original data table to obtain multiple original data sets;
[0012] A connection module, configured to establish a connection between the plurality of original data sets based on the received connection instruction;
[0013] The conversion module is used to convert each original data set after the connection is established to obtain the target data set;
[0014] A configuration module, configured to receive configuration instructions for each data item in the target data set, and construct a target configuration script based on the configuration instructions for each data item;
[0015] A processing module, configured to process the target data set based on the target configuration script;
[0016] The storage module is used to store the processed target data set into the target data table based on the received storage instruction.
[0017] On the one hand, an embodiment of the present application provides a computer device, including: a network interface, a processor and a memory, wherein the network interface, the processor and the memory are connected, the network interface is used to provide a data communication function, the memory is used to store a computer program, and the processor is used to call the computer program to execute some or all of the steps described in the first aspect of the embodiment of the present application.
[0018] Accordingly, an embodiment of the present application provides a storage medium, which stores a computer program, and the computer program includes program instructions, which are loaded and executed by one or more processors to execute the data table processing method described in one aspect of the embodiment of the present application.
[0019] Accordingly, the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the data table processing method described in one aspect.
[0020] It can be seen that in the embodiment of the present application, multiple original data sets are obtained by collecting the original data table through the collection instruction, and the multiple original data sets are connected and converted according to the connection instruction to obtain the target data set. Then, according to the configuration instructions specified for each data item in the target data set, the target configuration script is automatically constructed, and finally the target configuration script is used to process the target data set, and the processed target data set is stored in the target data table using the storage instruction. The entire data table processing flow abstracts data collection, conversion, and storage (i.e., data ETL), and reuses the same logic of data ETL, thereby improving the processing efficiency of the original data table to obtain the required data set to the target data table. In addition, according to the received collection instructions, connection instructions, configuration instructions, and storage instructions, a rich set of functional instructions can be matched to indicate different data table processing logics, and it is easy to expand, and can also be adaptively adjusted according to business scenarios to achieve flexible and changeable data table processing functions in different business scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0022] Figure 1 It is a schematic diagram of the architecture of a data table processing system provided in an embodiment of the present application;
[0023] Figure 2 It is a flowchart of a data table processing provided by an embodiment of the present application;
[0024] Figure 3 This is a schematic diagram of an interface for connecting and configuring an original data set provided in an embodiment of the present application;
[0025] Figure 4 This is a schematic diagram of an interface for configuring conversion rules for a target data set provided in an embodiment of the present application;
[0026] Figure 5 It is a flowchart of the processing logic of a Spark processor provided in an embodiment of the present application;
[0027] Figure 6 It is a flowchart of a Scala dynamic executor processing logic provided in an embodiment of the present application;
[0028] Figure 7 It is a schematic diagram of an interface for storage policy configuration provided in an embodiment of the present application;
[0029] Figure 8 It is a flowchart of a data table processing provided by an embodiment of the present application;
[0030] Fig. 9 It is a schematic diagram of a data table processing flow in an incremental collection and deduplication append storage mode provided in an embodiment of the present application;
[0031] Fig.10 It is a schematic diagram of a data table processing flow in an incremental collection and additional storage mode provided in an embodiment of the present application;
[0032] Fig.11 It is a schematic diagram of a data table processing flow in a full collection and additional storage mode provided in an embodiment of the present application;
[0033] Fig.12 It is a schematic diagram of a data table processing flow in a full collection and overwriting storage mode provided in an embodiment of the present application;
[0034] Fig.13 It is a structural schematic diagram of a data table processing device provided in an embodiment of the present application;
[0035] Fig.14 It is a structural schematic diagram of a computer processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0037] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.
[0038] In order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts that may be involved in the embodiments of the present application are first introduced below.
[0039] Airflow: An open source distributed task scheduling framework.
[0040] Spark: An open source big data stream processing engine.
[0041] Processor: Each data table processing node on the task flow in Airflow.
[0042] Original data table: Operational Data Store (ODS) catalog table.
[0043] Target data tables: including catalog tables of DWD (detailed data layer), DWS (summary data layer), DIM (common dimension layer), and ADS (application data layer).
[0044] ETL: Extract-Transform-Load, data extraction, transformation, and loading.
[0045] DataFrame: A distributed data set composed of columns (column name, column type, column value), with different names given to each column.
[0046] Usually, when using Airflow to process data streams, it is necessary to add business customized scripts to each task flow, and perform data table processing tasks on the original data table to generate a target data table. In a specific business scenario, the three task flows of data collection, data conversion, and data storage are involved, and customized scripts need to be manually written to complete data processing. However, when the business scenario changes, the data table processing requirements are different, and the customized scripts need to be rewritten. It can be found that such operations lack flexibility and scalability, which will affect the efficiency of data table processing. In view of this, the present application provides an ETL method developed based on the Airflow task scheduling framework, which loads the data of the original data table through multi-table connections and different collection modes, and then performs field-level conversion through Spark, and finally writes the target data table in different storage modes. The method abstracts the loading and storage modes of data, completes the data reading and storage process with different mode combinations, uses Spark for data conversion in the middle layer, and takes into account flexibility and scalability through dynamic execution of code, which can meet the needs of various complex business scenarios for data table processing.
[0047] It can be found that, as disclosed in the data table processing method of the present application, these data (such as the data of the original data table and the data of the target data table) can be stored on the blockchain.
[0048] See also Figure 1 , is a schematic diagram of the architecture of a data table processing system provided in an embodiment of the present application. Figure 1As shown, the data table processing system includes a data table processing device 100 and a terminal device 101. The terminal device 101 may include multiple terminals. Each terminal device 101 may be connected to the data table processing device 100 through a network. The network connection may include a wired connection or a wireless connection, so that each terminal device may interact with the data table processing device 100 through the network connection, and the data table processing device 100 may receive a business execution instruction from the terminal device 101 and perform corresponding data table processing tasks according to these business execution instructions. The specific functions of the data table processing device 100 and the terminal device 101 are described in detail below.
[0049] The main function of the data table processing device 100 is to perform data table processing tasks according to the business execution instructions received from the terminal device 101, including collection instructions, connection instructions, configuration instructions, and storage instructions. Among them, the collection instruction can be obtained by the server according to its own business needs, without being specified by the terminal device 101. The data table processing device 100 collects the original data set from the original data table stored in the underlying database or other storage sources (such as local files) according to the received collection instruction, and connects the original data set according to the connection instruction, and then converts the original data set after the connection processing to obtain the target data set, and then generates a target configuration script according to the configuration instruction, processes the target data set, and finally stores the processed target data set in the target data table according to the storage method indicated by the storage instruction. It should be noted that the connection instruction, configuration instruction and storage instruction as computer languages can all be generated according to the corresponding rules or policies specified by the terminal device 101 on the corresponding functional client. The data table processing device 100 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms, without any limitation here.
[0050] The terminal device 101 can install and run a client for data collection, conversion and storage, such as a data ETL visualization management platform, and can manually configure the collection mode, connection relationship, conversion rules and storage mode of the data table on the configuration interface provided by the client, generate corresponding business execution instructions according to the results of the manual configuration, and send them to the data table processing device 100. Optionally, the terminal device 101 can also monitor whether the collected original data set is successfully written into the target data table according to the specified rules on the data ETL visualization management platform it runs, and can also view the data table processing process of the back-end data table processing device 100. The terminal device 101 can be one or more of a smart phone, a camera, a desktop computer, a tablet computer, an MP4 player and a laptop portable computer.
[0051] It can be found that the above-mentioned data table processing system can avoid manually writing customized scripts, but instead combines business execution instructions of different processing logics through the terminal device 101. After receiving these business execution instructions, the data table processing device 100 can obtain the corresponding configuration script and perform data table processing based on the configuration script. When the specific business scenario changes, it only needs to adjust the business execution instructions configured in the terminal device 101, so as to quickly respond to the conversion of the data table processing logic caused by the change of the business scenario, making the data table processing more flexible and efficient.
[0052] For further understanding, please see Figure 2 , is a flow chart of a data table method provided in an embodiment of the present application. The execution subject in this embodiment can be a computer device or a cluster composed of multiple computer devices. The computer device can be a terminal device or a server. Here, the execution subject in this embodiment is a server (such as Figure 1 The data table processing device 100 shown in FIG. 1 is used as an example for description. The data table processing method may at least include the following steps S201-S204:
[0053] S201, acquiring multiple original data tables, and acquiring data sets in each original data table based on the received acquisition instruction to obtain multiple original data sets.
[0054] In a possible embodiment, the original data table is a catalog table from different operation data layers, which is the most original data. It is a structured data table, that is, it includes field name, type and other information, such as a hive table. Therefore, multiple original data tables can be obtained from the underlying database (such as MySQL, hive, Redis...) or the data table stored locally offline by the terminal device, and there is no restriction on the way and method of acquisition. The collection instruction received by the server can be sent by the terminal device, or it can be a collection instruction automatically generated by the server based on the structural information of the multiple original data tables obtained and the specific business scenario, and there is no restriction on this. The collection instruction indicates the mode of obtaining the data in the original data table, such as obtaining all the data in the original data table or obtaining part of the data in the original data table, that is, including full collection and incremental collection. Correspondingly, the received collection instruction can be a full collection instruction or an incremental collection instruction for each original data table. Specifically, the data stored in the fixed table structure of each original data table is collected according to the collection instruction. The obtained data set may have the same or different table structure as the original data table, but a corresponding data set can be obtained by collecting the data of each original data table. For example, according to the collection instruction, all the data in the original data table 1 is collected to obtain data set 1. In this way, multiple original data sets corresponding to each original data table can be obtained.
[0055] Taking the received collection instruction as an incremental collection instruction as an example, based on the incremental collection instruction, the data sets in each original data table are collected, and the specific steps of obtaining multiple original data sets can be as follows: First, obtain the existing data set in the target data table. The existing data set of the target data table refers to the data stored in the target data table before the current data table processing operation, wherein the data set includes multiple data items, each of which is the data corresponding to a single field, and records the time node of data generation, which is used as a timestamp, or the time data corresponding to the time field in the target data table is used as a timestamp. For the convenience of comparison, the timestamps of each data item are counted according to a unified standard, such as seconds, minutes, hours, days, weeks, months, etc., which are not limited. Then, determine the target timestamp in the existing data set, which refers to the largest timestamp among the timestamps of each data item in the existing data set. For example, if the existing data set of the target data table is the daily order volume of the product from March 1 to March 23, then according to the date, the maximum timestamp of the existing data set is March 23. Next, at least one original data item in each of the original data tables whose timestamp is greater than the target timestamp is determined, that is, based on the target timestamp, at least one original data item in each of the original data tables whose timestamp is greater than the target timestamp can be determined. Similar to the target data table, the original data items included in the multiple original data tables obtained above also have timestamps. For any original data table, the data items in each original data item whose timestamp is greater than the target timestamp are selected, that is, the data that has not been loaded into the target data table before is determined. Furthermore, multiple original data sets corresponding to the multiple original data tables are constructed based on at least one original data item, that is, multiple original data sets can be constructed based on these original data items, and each original data set corresponds to each original data table. For example, the existing data set is the daily order volume of the product from March 1 to March 23. One of the multiple original data tables obtained records the daily order volume of the product from March 1 to April 1, and the maximum timestamp of the existing data set is March 23. At this time, the daily order volume of the product from March 24 to April 1 is used as the original data item greater than the maximum timestamp of the existing data set. The original data set corresponding to the original data table can be constructed. As an optional example, the daily order volume of the product from March 24 to April 1 can be directly used as the original data set, or the product identification and other description data of the product (such as daily page views, daily transaction volume, etc.) can be pulled out based on the daily order volume of the product from March 24 to April 1 as the original data set. It should be noted that an original data set can be obtained for each original data table, and the method of obtaining the original data set by the above incremental acquisition instruction is applicable to any original data table among the multiple original data tables obtained to obtain the original data set.
[0056] S202, establishing connections between multiple original data sets based on the received connection instruction, and performing transformation processing on each of the original data sets after the connections are established to obtain a target data set.
[0057] In a possible embodiment, since multiple original data tables come from different databases, and different databases store data tables in different formats, the table structures and specific data types of the multiple original data tables will be different, which will cause the inconsistency of multiple original data sets. Therefore, these original data sets need to be converted and centrally managed. The original data sets collected from the original data tables are used as the data source for data conversion. It is necessary to pre-associate multiple original data sets through table connections. Specifically, multiple original data sets can be connected based on the received connection instructions, wherein the received connection instructions are generated by the table connection relationship configured by the terminal device for multiple original data sets, including any one or more corresponding connection instructions among inner join, full join, Cartesian product join, left join, and right join. Please refer to Figure 3 , is a schematic diagram of an interface for connecting and configuring an original data set provided in an embodiment of the present application, such as Figure 3 As shown, the input catalog table (i.e., the original data table) includes the health code full data table in the health code screen of the data operation layer and the class punch-in activity test data table of the campus circle. The original data sets collected from these two original data tables are configured for inner connection in the configuration table connection interface. Specifically, the two original data sets are connected by a line with a single arrow pointing in a dragging manner (if the direction is not distinguished, the line may not have an arrow), and the connection relationship is specified as an inner connection, or the connection relationship is automatically generated as an inner connection after the line is connected. Users can also modify the automatically generated connection relationship according to their needs. Through the inner connection of the original data set, the health code data of the class punch-in activity can be filtered out from the original data set collected by the health code full data table.
[0058] In a possible embodiment, after the connection relationship of multiple original data sets is successfully established, in order to centralize the management of the original data sets, it is also necessary to convert these original data sets to obtain the target data set, so that the target data set can be directly processed accordingly later. The implementation method of converting each original data set after the connection is established to obtain the target data set can be specifically as follows: first, obtain the target original data set in each original data set after the connection is established, and the target original data set is connected with each original data set as the root node, that is, each original data set is regarded as an independent node. The original data set to establish the connection is a set of multiple nodes with an associated relationship. The root node can be determined from the multiple node sets with an associated relationship. The specific selection criteria for the root node are as follows: the in-degree is 0, and all nodes can be traversed. It should be noted that the root node selection for inner join, full join and Cartesian product join does not distinguish the direction, while the left join and right join must distinguish the direction. Then, a table connection tree corresponding to each original data set after the connection is established is constructed, and the table connection tree is used to indicate the connection relationship between each original data set. In other words, the connection relationship between each original data set will be converted into a tree representation structure, that is, the table connection tree is used to indicate the connection relationship between each original data set. Then, based on the table connection tree, a connection relationship query expression can be generated, and each original data set is processed based on the connection relationship query expression to obtain the target data set. The connection relationship query expression is a structured query language (SQL) expression for multi-table connection. According to the SQL expression, each original data set can be processed, and the data in the original data set can be integrated into the same data set as the target data set. The target data set here can be a Spark DataFrame object df. DataFrame is similar to a relational data table and belongs to the built-in type of Spark data table. The DataFrame object is generated by connecting to other databases as data sources through SQL expressions. After the DataFrame object is generated, related operations can be performed on DataFame, such as displaying data, obtaining statistical information of specified field data, performing conditional queries and connection operations, etc. Compared with ordinary distributed data sets, DataFrame has more optimizations at the bottom layer. Therefore, data table related processing based on this target data set can help improve the speed of data table processing and make the entire data table processing process smoother.
[0059] S203, receiving configuration instructions for each data item in the target data set, and constructing a target configuration script based on the configuration instructions for each data item.
[0060] In a possible embodiment, the target data table, as the output data of the task flow, needs to pre-define the table structure information (such as field name, type), and store the defined table structure in a structured database. The specific data corresponding to the table structure information of the target data table comes from the original data set collected by the original data table, but there are differences in fields, types, etc. between the original data table and the target data table. It is necessary to configure field conversion rules to indicate the processing logic for converting the original data table fields to the target data table fields. In a specific implementation, the field conversion rules can be configured in a client with data table processing function (such as the aforementioned data ETL visualization management platform running on the terminal device), generate configuration instructions, and send the configuration instructions to the server. Please refer to Figure 4 , is a schematic diagram of an interface for configuring conversion rules for a target data set provided in an embodiment of the present application, such as Figure 4 As shown, in the data ETL visualization management platform provided in the terminal device, you can independently select the target data table to be configured, such as Figure 4 In the output catalog table (i.e., target data table) selection box, specify the learning situation analysis table as the detailed data layer DWD. When the target data table is selected, the field name of the target data table, the type of each field name, whether it is the primary key, and the conversion input box will be displayed on the interface. By configuring the corresponding conversion rules in the conversion input box, the processing logic from the original data table to the target data table can be obtained, which involves the configuration of each data item in the target data set. The specific steps for operating on the interface for configuring the conversion rules of the target data set are: when the user's cursor is in the conversion input box, an associative drop-down list will appear, listing all the Spark functions and field names that can be entered at the current position, such as the function add to calculate the cumulative sum, the function concat to connect strings, and the new field name generated after the original data table is connected. After the user selects the input, the previous process is repeated until the expression is completed. It should be noted that the Spark function here can be a built-in function of Spark itself, or a user-defined function built into Spark. For example, to obtain the course identifier course_id in the above catalog table that needs to be output, that is, the learning situation analysis table, it is necessary to associate the original data set collected from the original data table of the school dimension and the original data set collected from the original data table of the class dimension, convert them to obtain the target data set, and then use the aggregation function agg in the Spark function to count and extract the total number of courses arranged by the school with the same course_id for each class as the data to be written into the course_id field in the target data table.
[0061] At this point, each data item in the target data set corresponds to a configuration instruction, indicating the processing logic of each data item. The processing logic indicated by the configuration instruction of each data item can correspond to a preset code module. The target configuration script can be generated by calling the specified code module according to the configuration instruction. The target configuration script can be Scala code or other types of code, such as Java, Python, etc., which are not limited here. In this step, automatically building the target configuration script through the configuration instruction makes user-oriented operations simpler and more convenient, reducing the difficulty of using the system.
[0062] S204: Process the target data set based on the target configuration script, and store the processed target data set into the target data table based on the received storage instruction.
[0063] In a possible embodiment, the target configuration script is a code file composed of code modules corresponding to the configuration instructions of each data item in the target data set. The code file can be run by a code processing tool to realize the processing of the target data set according to the processing logic of the target configuration script. The specific processing content is mainly to convert each data item in the target data set to obtain data suitable for storage in the target data table. For example, if the field in the target data table is the age of user A, the corresponding data is a numerical value such as 1-100, and the target field in the obtained target data set is the birth date of user A, and the corresponding data has a certain format, such as the record data in December 1997. At this time, it is necessary to convert the birth date of user A into the age field according to the processing logic indicated by the corresponding functional code module in the target configuration script, and use the data set with the age field as the processed target data set.
[0064] In a possible embodiment, the specific steps of processing the target data set based on the target configuration script may include: starting a computing engine process, and calling the computing engine to execute the target configuration script to generate processing instructions, and then processing the target data set based on the processing instructions to obtain a processing result. If the processing result meets a preset condition, the computing engine process is closed to stop processing the target data set.
[0065] Specifically, the computing engine can be a Spark processor, and the task scheduler Airflow can be used to schedule the Spark processor to execute the target configuration script to process the target data set. For specific steps, see Figure 5 , is a flowchart of the processing logic of a Spark processor provided in an embodiment of the present application. Figure 5As shown, first set the execution instruction, specifically set the custom class TxSparkOperator to inherit the base class BaseOperator, where the operator Operator can be understood as the abstract class of the task Task, and BaseOperator is also the source of all functional Operators. TxSparkOperator can realize the function of data conversion based on airflow.models.BaseOperator. The main parameters exposed by TxSparkOperator include the dynamic executor parameter execute_file, the execution main class parameter main_class, the execution parameter execute_args, the Spark directory parameter Spark_home, and the Spark execution mode parameter Spark_master. Then start the subprocess, that is, start the computing engine process, call the computing engine Spark to execute the target configuration script, and generate the processing instruction. The specific code description is sp=subprocess.Popen(Spark_cmd). Then wait for the Spark process (i.e., the subprocess) to exit. The specific code description is intcode=sp.wait(). Before the process exits, the computing engine processes the target data set according to the processing logic of the processing instruction, and obtains the processing result after the process exits. Finally, determine whether the process return code is 0, that is, determine whether the processing result meets the preset conditions. If the preset conditions are met, that is, the process return code is 0, it means success. At this time, you can shut down the computing engine and stop processing the target data set. On the contrary, it means failure, you can pause the computing engine, and give correction prompt information to help adjust the places that do not meet the process execution logic, so that the target data set can be successfully processed according to the target configuration script.
[0066] Among them, the specific process of calling the computing engine to execute the target configuration script to generate processing instructions can be: first use the programmable interface of the computing engine process to receive the target configuration script, then encapsulate the target configuration script to obtain an executable class, and then call the executable class based on a preset calling function to generate a processing instruction.
[0067] Specifically, the dynamic executor indicated by the dynamic executor parameter execute_file set in the computing engine executes the target configuration script and generates processing instructions. For example, the target configuration script is a Scala code file and is processed by the Scala executor. Figure 6, is a flow chart of the processing logic of a Scala dynamic executor provided in an embodiment of the present application. The implementation details of the processing logic steps of the dynamic executor are as follows: first, load the Scala code file, that is, receive the target configuration script, specifically through the custom class ScalaParser for dynamically executing Scala code, as the entry of the Spark task, and read the original Scala code from the ScalaParser entry. Then encapsulate the Scala code, specifically through a custom class for encapsulating the original Scala code, the Scala class is also the ScalaScript encapsulation, and provides a running context, such as SparkContext and SQLContext. The read original Scala code is encapsulated as a class WrapScript (that is, an executable class), and inherits ScalaScript. The SparkContext and SQLContext contexts provided by ScalaScript can be used in the Scala code for related operations. The encapsulated code is as follows:
[0068] class WrapScript extends ScalaScript{
[0069] override def run():Unit={
[0070] Original Scala code
[0071] }
[0072] }
[0073] new WrapScript().run()
[0074] Among them, the subclass WrapScript inherits the parent class ScalaScript using the keyword extends in the extension class. The original Scala code is specifically encapsulated in the rewritten method, and an object that can run the class WrapScript is created. Then the Scala interpreter is instantiated, that is, the Scala interpreter object scala.tools.nsc.interpreter.IMain is created. The specific code description is IMain interpreter = new IMain(). This instantiated Scala interpreter is a preset call function. The interpreter interpreter can be used to run the encapsulated Scala code. The specific code description is interpreter.interpret (encapsulated Scala code), where the encapsulated Scala code can run the call processing of the class. Finally, check whether the interpreter running result is abnormal. If it is abnormal, it means that the Scala code fails to run, otherwise it runs successfully. The code generated by the above steps can be regarded as a processing instruction, which can be used to process the target data set to obtain the processing result.
[0075] In the process of receiving the target configuration script by using the programmable interface of the computing engine process, the target configuration script can be read through the customized Airflow RestAPI interface, and the read target configuration script is passed to the Spark processor programmable interface for processing, specifically the ScalaParser entry where the dynamic executor indicated by the dynamic executor parameter passed to the Spark processor is located. In addition, the RestAPI interface can be used for backend operations such as deploying, starting, and stopping task flows.
[0076] Generally speaking, after the Scala code is run, a jar package similar to an executable file will be generated. The jar package corresponds to the properties, classes, and methods involved in the Scala code. Usually, the jar package obtained by the execution is passed to Spark for distributed computing. However, the development of the dynamic executor allows the Scala code to be directly passed to Spark for processing, and the Scala code is executed using the properties, classes, and methods defined in the Scala dynamic executor (equivalent to the built-in jar package in Spark) to achieve the function of processing the target data set. This approach avoids the generation of different jar packages for different Scala codes, reduces the redundancy of Spark processing jar packages, and thus improves the efficiency of data table processing.
[0077] Due to different data requirements at the destination end in different business scenarios, the data table processing from the data source end to the destination end requires specifying the storage strategy of the processed target data set, which mainly determines the mode in which data is written to the target data table. The storage instructions received by the server in this step are the instructions generated by the storage strategy specified by the user on the client with data table processing function. For details, see Figure 7 , is a schematic diagram of an interface for storage policy configuration provided by an embodiment of the present application. Figure 7 As shown, the storage strategy input box column displays three modes of overwrite, append and deduplication append through a drop-down module. In addition, there are some restrictions on the target data table, including time filtering, condition filtering, etc. to further filter the target data set after processing. For example, when the target data table is a log data table, which is a type of data that continues to grow, it is only necessary to add the latest data to the target data table without paying attention to the historical data already stored in the target data table. At this time, the storage strategy can be defined as an append mode, that is, appending new data to the target data table. It should be noted that the storage instructions generated according to different storage strategies are different. When the processed target data set is stored based on the received storage instructions, the processing logic executed according to different storage instructions is also different. For the process of storing the processed target data set to the target data table under specific storage instructions, please refer to the following embodiments.
[0078] In summary, the embodiments of the present application have at least the following advantages:
[0079] For the multiple original data tables obtained, full collection instructions or incremental collection instructions can be received to collect and process the data to obtain multiple original data sets. Based on the received connection instructions, connections are established for the multiple original data tables, and the multiple original data sets are associated to make data processing more efficient; the original data sets after the connection is established are converted into target data sets, and sufficient preparations are made for writing data into the target data table. The target data set uses the Spark DataFrame object to help improve the efficiency of data table processing. The target configuration script is automatically generated based on the configuration instructions of each data item of the target data set, avoiding manual writing of data conversion scripts. Among them, the Spark processor distributed computing realizes data conversion, and the dynamic executor dynamically executes the target configuration script, taking into account flexibility and scalability.
[0080] See also Figure 8 , is a flow chart of a data table method provided in an embodiment of the present application. The execution subject in this embodiment can be a computer device or a cluster composed of multiple computer devices. The computer device can be a terminal device or a server. Here, the execution subject in this embodiment is a server (such as Figure 1The data table processing device 100 shown in FIG. 1 is used as an example for description. The data table processing method may at least include the following steps S801-S806:
[0081] S801, taking multiple original data tables, and collecting data sets in each original data table based on the received collection instruction to obtain multiple original data sets.
[0082] S802: Establish connections between the multiple original data sets based on the received connection instruction, and perform transformation processing on each of the original data sets after the connections are established to obtain a target data set.
[0083] S803: Receive configuration instructions for each data item in the target data set.
[0084] The specific implementation of the above steps can be found in the above Figure 2 S201-S203 in the corresponding embodiment will not be described in detail here.
[0085] S804, obtaining the code module corresponding to the configuration instruction of each data item.
[0086] In a possible embodiment, each data item is a data item in the target data set, and the field conversion rule configured in the front-end visual interface for this data item represents a configuration instruction. The terminal device can use open source code tools to implement intelligent prompts and analysis of conversion rules, such as the online code editor CodeMirror, and the javascript code analysis engine Tern, to assist in enhancing CodeMirror's javascript code intelligent editing capabilities. The combination of the two can implement intelligent prompts for field conversion rules. The specific implementation content of the analysis of field conversion rules is as follows: First, the syntax format of Spark's built-in functions must be defined, such as the accumulation function add, which accumulates data between two different columns. Its syntax format is as follows:
[0087] "add":{
[0088] "!type":"fn(col1:Column,col2:Column)->Column",
[0089] "!doc":"add(col1,col2): Calculate the cumulative sum of col1 and col2",
[0090] "!Spark":"%c.plus(%c)",
[0091] "!SparkType":"column"
[0092] }
[0093] Then use the fast JavaScript parser Acorn to parse the field conversion rules entered by the user into an abstract syntax tree (Abstract Syntax Tree, AST), and then convert the AST into a valid rule code. Here, the valid rule code for a Spark built-in function conversion is a code module. For example, the data items involved in the above add function are the data corresponding to the columns col1 and col2. For these two data item sets, the configuration instruction instructs the add function to parse the valid rule code, which is the code module corresponding to the configuration instruction. From a broader perspective, the field can also be regarded as a data item, such as the column col1 is a data item. The specific configuration instruction is based on the field configuration, and the data contained in the field can be processed according to the valid rule code. For other custom functions or built-in functions, the code module can be obtained in the same way and stored in the database in advance, so that the corresponding code module can be obtained in time when the data item configuration instruction is received.
[0094] S805 , determining the combination logic corresponding to each code module based on the table connection tree, and combining the code modules based on the combination logic to obtain a target configuration script.
[0095] In a possible embodiment, the table connection tree represents the connection relationship between the original data sets. According to the connection relationship, there is also a combination logic between the data items. For example, if the original data set A is left-connected to the original data set B, then the original data set A is used as the main table, and the data of the original data set B should be associated with the original data set A, and the associated data is the data of the fields in the original data set A. Based on such combination logic, the code modules corresponding to the configuration instructions of each data item can be combined to obtain the target configuration script.
[0096] S806: Process the target data set based on the target configuration script, and store the processed target data set into the target data table based on the received storage instruction.
[0097] In a possible embodiment, the processing of the target data set based on the target configuration script can be referred to the detailed description of the aforementioned embodiment, which will not be repeated here. This step mainly describes how to store the processed target data set into a data table according to different storage instructions. According to the storage strategy mentioned above, the storage instructions corresponding to the storage strategy may include overwrite storage instructions, append storage instructions, and deduplication append storage instructions. The following takes the deduplication append storage instruction as an example to describe the specific steps of storing the target data set to be processed into the target data table based on the received deduplication append storage instruction. The specific contents are as follows:
[0098] First, the existing data set is left-connected to the target data set to obtain a merged data set. The left-connection process uses the existing data set as the main table and the target data set as the secondary table. When connecting, the primary key of the existing data set can be used as the connection standard. The data connected is the data in the target data set. The data that is not connected (that is, the data that the main table and the secondary table do not match) is filled with Null, and finally the merged data set can be obtained. For example, if the data in the score field of a student in the main table is 90 or Null, and the score field of a student in the secondary table is 92, the condition of the left connection is that the score field is the same, and the two data are not the same, then the score field will be recorded as Null, and then the data items with empty data in the merged data set can be filled based on the data in the target data set to obtain a filled data set. Still using the above example as an illustration, that is, replace Null with 92 in the secondary table to generate a new data set, and finally store the filled data set in the target data table in an overwritten manner to complete the data update or append. Of course, for the same data, it will not be updated, and this reflects the de-duplication process.
[0099] As another optional method, the specific steps of storing the target data set to be processed into the target data table based on the received deduplication append storage instruction can also be: first determine the data items with empty primary keys in the target data set, wherein the primary key can be one or more fields (or attribute groups) in the target data set, which are used to uniquely identify a record in the target data set. For example, the student table (student number, name, gender, class), in which each student's student number is unique, the student number is a primary key, the score table (student number, course number, score) a single attribute in the score table cannot uniquely identify a record, the combination of the student number and the course number can uniquely identify a record, so the attribute group of the student number and the course number is a primary key, and the corresponding data item with an empty primary key indicates that there is no data corresponding to the data set with the primary key of the existing data set in the target data set. Then, an empty data set is constructed based on the data items with empty primary keys, specifically, the existing data set of the acquired target data table and the target data set without a primary key are left-joined, that is, the existing data set is used as the main table, the same fields in the target data set and the existing data set are screened out, and merged into a data table, which is called an empty data set. Then, the empty data set and the target data set are combined to obtain a combined data set. Finally, the combined data set is written to the target table in an overwritten manner, so that if the new data primary key exists in the existing data set, it can be updated. If it does not exist, the new data primary key can be added to the existing data set according to the combination process.
[0100] For example, the following student information table and student score table are used as the existing data set and the target data set respectively. The primary key of the existing data set is S_id, and the primary key of the target data set is Sc_id:
[0101] Student Information Form
[0102] S_id S_name S_birth 1 Zhang San 1991-12 2 Li Si 1994-01 3 Wang Wu 1995-03 4 Zheng Er 1997-04
[0103] Student transcript
[0104] Sc_id c_id S_source 1 001 88 2 002 95 3 003 73 5 004 94
[0105] Left-joining these two tables yields the following table, which serves as an empty data set.
[0106] S_id S_name S_birth Sc_id c_id S_source 1 Zhang San 1991-12 1 001 88 2 Li Si 1994-01 2 002 95 3 Wang Wu 1995-03 3 003 73 4 Zheng Er 1997-04 Null Null Null
[0107] It can be found that the records in the student score table (as the target data set) in the above table that do not have data corresponding to the primary key in the student information table (as the existing data table) are empty Null. Then the target data set and the empty data set are processed and the data obtained are as follows. The data set is stored in the target data table in overwrite mode:
[0108] S_id S_name S_birth S_id c_id S_source 1 Zhang San 1991-12 1 001 88 2 Li Si 1994-01 2 002 95 3 Wang Wu 1995-03 3 003 73 4 Zheng Er 1997-04 Null Null Null Null Null Null 5 004 94
[0109] It is worth noting that although the storage instruction is deduplication and append storage, the mode of writing the data set into the target data table in the end is the overwrite mode. The overwrite mode here can reflect the principle of appending the data corresponding to the key field (or primary key) that does not exist in the existing data set of the target data table, and updating the existing primary key data. Deduplication can be understood as that in a data record corresponding to the same primary key, the data that is the same in both the existing data set and the target data set will not be updated.
[0110] Combined with the above Figure 2 The incremental collection instruction for the original data table in the embodiment and the deduplication and additional storage instruction in this embodiment can provide a data table processing solution. Fig. 9, is a schematic diagram of a data table processing flow in an incremental acquisition and deduplication append storage mode provided by an embodiment of the present application. This mode will process according to the primary key of the target data table each time a task is executed. If the new data primary key exists in the target data table, it will be updated. If it does not exist, it will be added. It is used for data that continues to grow and needs to be unique. The specific steps are: first read the target data table data set cdf (that is, the existing data set of the target data table). The target data table data set can be a set of catalog tables at any level of DWD, DWS, DIM, and ADS. After obtaining the target data table data set, calculate the maximum timestamp mt of the cdf (or other identifiers that can be incremented during update). The maximum timestamp is used as an identifier for update increment, which can provide a reference standard for the incremental acquisition data set. Therefore, the data in the original data table with a timestamp greater than the maximum timestamp can be taken as the collection object, and data collection is performed to generate a data set df (the table to be connected has been converted to obtain the target data set). Then, custom operations are performed on the data set (i.e., field conversion rules and storage strategies are configured). Next, cdf is calculated to be left-joined to df, and the data set edf (i.e., an empty data set) whose primary key is df is calculated. Then, the union udf of df and edf (i.e., the merged data set) is calculated. Finally, udf is written into the target data table in an overwrite manner.
[0111] In addition to the above incremental collection and deduplication append storage mode data table processing flow, there are other combination mode data table processing flows. Fig.10 , is a schematic diagram of a data table processing flow in an incremental collection and append storage mode provided by an embodiment of the present application. This mode appends new data to the target data table each time a task is executed, and is used for continuously growing data, such as logs. The specific steps are as follows Fig.10 As shown, first read the target data table data set cdf (that is, the existing data set of the target data table), then calculate the maximum timestamp mt of cdf or other identifiers that can be incremented when updating, then read the data sets with timestamps greater than mt in the main table of the original data table (that is, original data table 1), that is, incrementally collect the data in the main table to obtain the original data set, and at the same time read all the other original data tables (original data tables 2 to N) to generate N original data sets in total, perform table connection operations on the N original data sets, including inner join, left join, right join, full join, and cross join to obtain the data set df, and perform custom operations on df (same as Fig. 9 The target data set processed by the custom operation is written to the target data table in the append mode.
[0112] See also Fig.11, is a schematic diagram of a data table processing flow in a full collection and additional storage mode provided in an embodiment of the present application. This mode appends new data to the target data table each time a task is executed. It is often used for complete data backup. The specific steps are as follows Fig.11 As shown, first, all original data tables 1 to N are read in full to generate N original data sets, that is, each original data table read in full corresponds to an original data set, and then table join operations are performed on the original data sets, including inner join, left join, right join, full join, and cross join to obtain the data set df, and then custom operations are performed on df, and finally df is written to the target data table in append mode.
[0113] See also Fig.12 , is a schematic diagram of a data table processing flow in a full collection and overwriting storage mode provided by an embodiment of the present application. In this mode, each time a task is executed, the original data in the target data table will be cleared for overall data update. The specific steps are as follows Fig.12 As shown, first read all the original data tables 1 to N to generate N original data sets, then perform table join operations on the N original data sets, including inner join, left join, right join, full join, and cross join to obtain the data set df, then perform custom operations on df, and finally write df to the target data table in an overwrite manner, that is, write all the data included in df to the target data table to complete the update of the target data table. Optionally, in this mode, you do not need to clear the original data when executing the task. Instead, replace the original data in the target data table after obtaining the target data to achieve the update purpose. Fig.11 The difference between the full-volume acquisition and append storage modes shown is that the final writing method of the target data table is overwriting. Using this storage mode requires updating all data in the target data table, while the append storage mode only requires updating part of the data in the target data table.
[0114] From the data table processing flow schemes under different mode combinations in the above examples, it can be seen that in full mode, multiple tables can be used for connection. In incremental mode, in order to prevent data disorder caused by connecting multiple incremental tables, a single table is generally used. In order to meet a wider range of business scenarios, a single main table incremental mode and multiple secondary tables full mode can also be used for multi-table connection (such as Fig.10). By combining the abstract data collection and data storage modes of this application, a data table processing method that meets different business needs can be used. Different Spark scripts are synthesized for different modes. The Spark script here includes preset code modules obtained by various instructions. These code modules are combined according to specified logical relationships to form a Spark script. Although the processing flow of the script is different in each mode, it can flexibly adapt to the data processing needs of most current business scenarios. Of course, for the combination modes not shown (such as full collection and deduplication and additional storage modes), their applicable business scenarios have yet to be explored.
[0115] In summary, the embodiments of the present application have at least the following advantages:
[0116] The data table processing solution provided in the embodiment of the present application is a complete data ETL processing framework developed based on the task scheduling framework Airflow, which abstracts the functions of different stages of ETL. For example, in the data conversion stage, the configuration instructions are used to pull the corresponding preset code modules, synthesize the target configuration script, and realize field-level conversion, so that the data conversion of the data table is faster. In the data storage stage, the converted target data set is written into the target data table according to the storage mode indicated by the storage instruction. The acquisition instructions, configuration instructions, storage instructions, etc. are abstracted, and the corresponding code modules are set. For the corresponding scripts under the same instructions, code reuse can be achieved, the portability of the system can be improved, and the data table can be processed flexibly and variably by combining different instructions. It can handle data services in multiple modes, such as incremental acquisition and deduplication additional storage, incremental acquisition and additional storage, etc. It is compatible with the needs of various complex scenarios for large-scale data processing, and is easy and flexible to use, with good scalability.
[0117] See also Fig.13 , is a schematic diagram of the structure of a data table processing device provided in an embodiment of the present application. The data table processing device described in the embodiment of the present application corresponds to the computer device described above. The data table processing device may be running on Figure 1 A computer program (including program code) in the data table processing device 100 shown, for example, a data table processing device is an application software; the device can be used to execute the corresponding steps in the data table processing method provided in the embodiment of the present application. The data table processing device 130 includes: a collection module 1301, a connection module 1302, a conversion module 1303, a configuration module 1304, a processing module 1305, and a storage module 1306, wherein:
[0118] The acquisition module 1301 is used to acquire multiple original data tables, and acquire data sets in each original data table based on the received acquisition instruction to obtain multiple original data sets.
[0119] The connection module 1302 is used to establish a connection between multiple original data sets based on the received connection instruction.
[0120] The conversion module 1303 is used to convert each original data set after the connection is established to obtain a target data set.
[0121] The configuration module 1304 is used to receive configuration instructions for each data item in the target data set, and construct a target configuration script based on the configuration instructions for each data item.
[0122] The processing module 1305 is used to process the target data set based on the target configuration script.
[0123] The storage module 1306 is configured to store the processed target data set into a target data table based on the received storage instruction.
[0124] In one embodiment, the acquisition module 1301 is specifically used to: obtain an existing data set in a target data table, and determine a target timestamp in the existing data set, where the target timestamp is the largest timestamp among the timestamps of each data item in the existing data set; determine at least one original data item in each original data table whose timestamp is greater than the target timestamp; and construct multiple original data sets corresponding to multiple original data tables based on at least one original data item.
[0125] In one embodiment, the conversion module 1303 is specifically used to: obtain a target original data set from each original data set after the connection is established, and the target original data set is used as a root node to establish a connection with each original data set; starting from the target original data set, traverse each original data set to construct a table connection tree corresponding to each original data set after the connection is established, and the table connection tree is used to indicate the connection relationship between each original data set; generate a connection relationship query expression based on the table connection tree, and process each original data set based on the connection relationship query expression to obtain the target data set.
[0126] In one embodiment, the configuration module 1304 is specifically used to: obtain the code module corresponding to the configuration instruction of each data item; determine the combination logic corresponding to each code module based on the table connection tree, and combine each code module based on the combination logic to obtain a target configuration script.
[0127] In one embodiment, the storage module 1306 is specifically used to: left-join the existing data set to the target data set to obtain a merged data set; fill the data items in the merged data set that are empty based on the data in the target data set to obtain a filled data set; and store the filled data set in the target data table in an overwritten manner.
[0128] In one embodiment, the processing module 1305 is specifically used to: start the computing engine process, and call the computing engine to execute the target configuration script to generate processing instructions; process the target data set based on the processing instructions to obtain a processing result; if the processing result meets the preset conditions, close the computing engine process to stop processing the target data set.
[0129] In one embodiment, the processing module 1305 is further specifically used to: receive the target configuration script using the programmable interface of the computing engine process; encapsulate the target configuration script to obtain an executable class; call the executable class based on a preset calling function to generate a processing instruction.
[0130] It is understandable that the functions of the functional modules of the data table processing device described in the embodiment of the present application can be specifically implemented according to the relevant methods in the above-mentioned data table processing method embodiment, and the specific implementation process can refer to the relevant description of the above-mentioned method embodiment, which will not be repeated here. In addition, the description of the beneficial effects of using the same data table processing method will not be repeated here.
[0131] See also Fig.14 , is a structural diagram of a computer device provided in an embodiment of the present application, the computer device 140 may include a processor 1401, a memory 1402, a network interface 1403 and at least one communication bus 1404. Among them, the processor 1401 is used to schedule computer programs, and may include a central processing unit, a controller, and a microprocessor; the memory 1402 is used to store computer programs, and may include a high-speed random access memory, a non-volatile memory, such as a disk storage device, a flash memory device; the network interface 1403 provides a data communication function, and the communication bus 1404 is responsible for connecting various communication elements. The computer device 140 corresponds to the data table processing device 100 mentioned above.
[0132] The processor 1401 may be used to call a computer program in the memory to perform the following operations:
[0133] Acquire multiple original data tables, and based on the received acquisition instructions, collect data sets in each original data table to obtain multiple original data sets; establish connections between the multiple original data sets based on the received connection instructions, and transform each original data set after the connection is established to obtain a target data set; receive configuration instructions for each data item in the target data set, and construct a target configuration script based on the configuration instructions for each data item; process the target data set based on the target configuration script, and store the processed target data set in the target data table based on the received storage instructions.
[0134] In one embodiment, the processor 1401 is specifically used to: obtain an existing data set in a target data table, and determine a target timestamp in the existing data set, where the target timestamp is the largest timestamp among the timestamps of each data item in the existing data set; determine at least one original data item in each original data table whose timestamp is greater than the target timestamp; and construct multiple original data sets corresponding to multiple original data tables based on at least one original data item.
[0135] In one embodiment, the processor 1401 is specifically used to: obtain a target original data set from each original data set after the connection is established, and the target original data set is used as a root node to establish a connection with each original data set; starting from the target original data set, traverse each original data set to construct a table connection tree corresponding to each original data set after the connection is established, and the table connection tree is used to indicate the connection relationship between each original data set; generate a connection relationship query expression based on the table connection tree, and process each original data set based on the connection relationship query expression to obtain the target data set.
[0136] In one embodiment, the processor 1401 is specifically used to: obtain the code module corresponding to the configuration instruction of each data item; determine the combination logic corresponding to each code module based on the table connection tree, and combine each code module based on the combination logic to obtain a target configuration script.
[0137] In one embodiment, the processor 1401 is specifically used to: left-join the existing data set to the target data set to obtain a merged data set; fill the data items in the merged data set that are empty based on the data in the target data set to obtain a filled data set; and store the filled data set in the target data table in an overwritten manner.
[0138] In one embodiment, the processor 1401 is specifically used to: start a computing engine process, and call the computing engine to execute a target configuration script to generate processing instructions; process the target data set based on the processing instructions to obtain a processing result; if the processing result meets a preset condition, close the computing engine process to stop processing the target data set.
[0139] In one embodiment, the processor 1401 is specifically used to: receive a target configuration script using a programmable interface of a computing engine process; encapsulate the target configuration script to obtain an executable class; and call the executable class based on a preset calling function to generate a processing instruction.
[0140] In a specific implementation, the processor 1401, memory 1402 and network interface 1403 described in the embodiment of the present application can execute the implementation method of the computer device described in a data table processing method provided in an embodiment of the present application, and can also execute the implementation method and beneficial effects described in a data table processing device provided in an embodiment of the present application, which will not be repeated here.
[0141] The embodiment of the present application also provides a computer-readable storage medium, which stores instructions, and when the instructions are executed on a computer, the computer executes the data table processing method as described in the embodiment of the present application. The specific implementation method can be referred to the above description, which will not be repeated here.
[0142] The embodiment of the present application also provides a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the data table processing method of the embodiment of the present application. The specific implementation method can be referred to the above description, which will not be repeated here.
[0143] It should be noted that, for the above-mentioned various method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.
[0144] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0145] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A data table processing method, characterized in that: include: Acquire multiple original data tables, and acquire data sets in each original data table based on the received acquisition instruction to obtain multiple original data sets; The plurality of original data sets are connected based on the received connection instruction, and each of the original data sets after the connection is established is transformed to obtain a target data set; Receive configuration instructions for each data item in the target data set, and obtain a code module corresponding to the configuration instruction of each data item; Determine the combination logic corresponding to each code module based on the table connection tree, and combine the code modules based on the combination logic to obtain a target configuration script; wherein the table connection tree is generated during the conversion process, and the table connection tree is used to indicate the connection relationship between the original data sets; The target data set is processed based on the target configuration script, and the processed target data set is stored in a target data table based on the received storage instruction.
2. The method according to claim 1, characterized in that The acquisition instruction includes an incremental acquisition instruction, and the acquisition of data sets in each original data table based on the received acquisition instruction includes: Acquire an existing data set in the target data table, and determine a target timestamp in the existing data set, wherein the target timestamp is the largest timestamp among the timestamps of the data items in the existing data set; Determine at least one original data item in each of the original data tables whose timestamp is greater than the target timestamp; A plurality of original data sets corresponding to the plurality of original data tables are constructed based on the at least one original data item.
3. The method according to claim 1, characterized in that: The conversion process is performed on each original data set after the connection is established to obtain a target data set, including: Acquire a target original data set from among the original data sets after the connection is established, wherein the target original data set is used as a root node to establish a connection with each original data set; Taking the target original data set as a starting point, traversing each original data set, and constructing a table connection tree corresponding to each original data set after the connection is established; A connection relationship query expression is generated based on the table connection tree, and each of the original data sets is processed based on the connection relationship query expression to obtain a target data set.
4. The method according to claim 2, characterized in that: The storage instruction includes a deduplication and additional storage instruction, and storing the processed target data set into the target data table based on the received storage instruction includes: Left-connect the existing data set to the target data set to obtain a merged data set; Filling the data items in the merged data set that are empty based on the data in the target data set to obtain a filled data set; The filling data set is stored in the target data table in an overwritten manner.
5. The method according to claim 1, characterized in that The processing of the target data set based on the target configuration script includes: Starting a computing engine process, and calling the computing engine to execute the target configuration script to generate processing instructions; Processing the target data set based on the processing instruction to obtain a processing result; If the processing result meets the preset condition, the computing engine process is closed to stop processing the target data set.
6. The method according to claim 5, characterized in that The calling of the computing engine to execute the target configuration script to generate a processing instruction includes: Receiving the target configuration script using a programmable interface of the computing engine process; Encapsulating the target configuration script to obtain an executable class; The executable class is called based on a preset calling function to generate a processing instruction.
7. A data table processing device, characterized in that: include: The acquisition module is used to obtain multiple original data tables, and based on the received acquisition instructions, acquire the data sets in each original data table to obtain multiple original data sets; A connection module, configured to establish a connection between the plurality of original data sets based on the received connection instruction; The conversion module is used to convert each original data set after the connection is established to obtain the target data set; A configuration module, used to receive configuration instructions for each data item in the target data set, and obtain a code module corresponding to the configuration instruction of each data item; Determine the combination logic corresponding to each code module based on the table connection tree, and combine the code modules based on the combination logic to obtain a target configuration script; wherein the table connection tree is generated during the conversion process, and the table connection tree is used to indicate the connection relationship between the original data sets; A processing module, configured to process the target data set based on the target configuration script; The storage module is used to store the processed target data set into the target data table based on the received storage instruction.
8. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a network communication function, the memory is used to store program code, and the processor is used to call the program code to execute the data table processing method described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the data table processing method according to any one of claims 1 to 6 is executed.
Citation Information
Patent Citations
Method and device for linking data tables
CN106484740A
Relational database query method and device, electronic equipment and storage medium
CN111159316A
Data detection method and device and storage medium
CN111177176A
Data conversion method and device, and storage medium
CN111177247A
Method and device for generating ETL codes
CN111324647A
