Multi-source heterogeneous offline data processing method, system, device and medium

Abstract

The application relates to a multi-source heterogeneous offline data processing method, system, device and medium. The method reduces manual operation and reduces system development difficulty based on a visual interface configuration. Zero-code development is realized by adopting a DAG plug-in data table configuration, and flexibility is increased by dynamically arranging table relationships. A data table entity modeling model is designed to perform feature modeling on data tables in integrated plug-ins. A multi-source type table building statement template is designed to reduce manual coding, unify data source connections and increase system expansibility. A multi-source table building instance construction process is designed to automatically construct target table building statements. A multi-source type target table creation process is designed to autonomously complete efficient target table creation operations. Finally, data integration is realized in multiple links through offline data table processing, and the offline data processing complexity of multi-source heterogeneous type data tables is greatly reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, and relates to a multi-source heterogeneous offline data processing method, system, device and medium. BACKGROUND

[0002] At present, massive data synchronization from the edge to the cloud or a big data platform, offline data processing based on multiple heterogeneous data sources has become an important technical means, multi-source data table construction is a key link of data processing, and creating a unified and efficient multi-source system data table construction method and specification is an important prerequisite for realizing massive offline data processing. With the rapid development of the Internet of Things, industrial data types present diversity and complexity, data sources are various and difficult to manage and control uniformly. At present, for offline data processing of multi-source heterogeneous industrial systems, most traditional methods have defects such as low data processing efficiency and lack of flexibility and expansibility, especially when using data tables to process heterogeneous data, manual definition of data processing mapping rules is often required, and a plurality of external tools are connected to the database in sequence to perform effective target table data processing operations. With the complexity of business scenarios and the heavy manual operation, data processing becomes very difficult, and data security is difficult to guarantee.

[0003] At present, many scholars have conducted in-depth research on offline data processing methods based on heterogeneous type data tables and have achieved certain research results. For example, traditional technologies such as pre-processing data and matching multiple type preset templates, a method of establishing a data table by a script and referring to a preset template, a method of constructing a target data set according to SQL statement analysis of multiple different databases or data tables, a method of instructing a target data node to create a target table by sending a data table construction instruction to the target data node, and a method of constructing a target table by generating a data analysis algorithm according to user requirements and analyzing the data structure of multiple dimension tables using the data analysis algorithm. However, the foregoing traditional technologies still have the technical problem of high table construction complexity in the multi-source system data integration process. SUMMARY

[0004] In view of the problems in the foregoing traditional methods, the present application provides a multi-source heterogeneous offline data processing method, a multi-source heterogeneous offline data processing system, a computer device and a computer readable storage medium, which can greatly reduce the offline data processing complexity based on data tables.

[0005] In order to achieve the above purpose, the embodiments of the present application adopt the following technical solutions:

[0006] On the one hand, a multi-source heterogeneous offline data processing method is provided, comprising the steps of:

[0007] Adding each offline data source, configuring data source connection information and completing connectivity test in the data source management page of the heterogeneous data source; the offline data source includes a MySQL data source, a DB2 data source, a Hive data source, a Phoenix data source, a ClickHouse data source and a Doris data source;

[0008] Selecting a data table in the data source in the data set management page of the heterogeneous data source, synchronizing metadata information of the data table and saving the metadata information to a data set table; the metadata information includes a data table name, a field name, a field type, a primary key field, a partition field and an index field;

[0009] Creating and editing a DAG integrated canvas based on an interface operation of the DAG and saving to a canvas table;

[0010] Selecting an integrated plug-in based on the DAG in the DAG integrated canvas by using an offline integrated plug-in plug-in model; the integrated plug-in includes an input plug-in, a conversion plug-in and an output plug-in;

[0011] Selecting an input plug-in and configuring an input table in the DAG integrated canvas, selecting a conversion plug-in and configuring a conversion rule, and selecting an output plug-in and configuring an output table;

[0012] In the DAG integrated canvas, a data table relationship configuration model is used to build a conversion relationship between data tables in the integrated plug-in;

[0013] In the DAG integrated canvas, a data table entity modeling model is used to perform data table entity modeling on the selected input table and the output table, and a conversion rule entity modeling model is used to perform data modeling on the conversion rule;

[0014] In the DAG integrated canvas, an input table entity construction model is used to perform instance conversion on the input table configuration in the input plug-in, to obtain an input table entity and save to a data table entity table;

[0015] In the DAG integrated canvas, a conversion rule entity construction model is used to perform instance conversion on the conversion rule configuration in the conversion plug-in, to obtain a conversion rule entity and save to a conversion rule entity table;

[0016] In the DAG integrated canvas, an output table entity construction model, a data table relationship configuration model and a data table entity modeling model are used to build an output table entity and save to a data table entity table;

[0017] After obtaining a data source type of the output table entity, a multi-source table building template screening process model is used to perform type determination and template screening, to obtain a table building statement template of a target table;

[0018] In the DAG integration canvas, a model is constructed by matching the table creation statement template with the parameter configuration of the output table entity by using the multi-source target table table creation instance, so as to obtain the table creation statement instance of the target table;

[0019] A multi-source target table self-creation model is used to automatically connect the data source and execute the target table creation process according to the data source type of the output table entity, so as to obtain the created target table; wherein the creation of the target table and the whole process of offline data processing based on the heterogeneous type data table are realized by using the Spark offline computing engine.

[0020] On the other hand, a multi-source heterogeneous offline data processing system is also provided, comprising:

[0021] A source configuration module is configured to add each offline data source in the data source management page of the heterogeneous data source, configure the data source connection information and complete the connectivity test; the offline data source includes a MySQL data source, a DB2 data source, a Hive data source, a Phoenix data source, a ClickHouse data source and a Doris data source.

[0022] A meta-synchronization module is configured to select the data table in the data source in the data set management page of the heterogeneous data source, synchronize the metadata information of the data table and save the metadata information to the data set table; the metadata information includes the data table name, the field name, the field type, the primary key field, the partition field and the index field.

[0023] A canvas construction module is configured to create and edit the DAG integration canvas based on the DAG interface operation and save it to the canvas table.

[0024] A canvas plug-in module is configured to dynamically select the integration plug-in in the DAG integration canvas based on the DAG drag-and-drop by using the offline integration canvas plug-in model; the integration plug-in includes an input plug-in, a conversion plug-in and an output plug-in.

[0025] A data table configuration module is configured to select the input plug-in and configure the input table, select the conversion plug-in and configure the conversion rule, and select the output plug-in and configure the output table in the DAG integration canvas.

[0026] A table relationship configuration module is configured to construct the conversion relationship between the data tables in the integration plug-in by using the data table relationship configuration model in the DAG integration canvas.

[0027] An entity modeling module is configured to perform data table entity modeling on the selected input table and output table by using the data table entity modeling model, and perform data modeling on the conversion rule by using the conversion rule entity modeling model in the DAG integration canvas.

[0028] An input entity module is configured to, in the DAG integration canvas, utilize an input table entity construction model to perform instantiation conversion on input table configuration in an input plug-in, obtain an input table entity, and save the input table entity to a data table entity table;

[0029] A conversion entity module is configured to, in the DAG integration canvas, utilize a conversion rule entity construction model to perform instantiation conversion on conversion rule configuration in a conversion plug-in, obtain a conversion rule entity, and save the conversion rule entity to a conversion rule entity table;

[0030] An output entity module is configured to, in the DAG integration canvas, utilize an output table entity construction model, a data table relationship configuration model, and a data table entity modeling model to construct an output table entity, and save the output table entity to the data table entity table;

[0031] A template screening module is configured to utilize a multi-source table creation template screening process model to obtain a data source type of the output table entity, and then perform type determination and template screening to obtain a table creation statement template of a target table.

[0032] An instance matching module is configured to, in the DAG integration canvas, utilize a multi-source target table creation instance construction model to match the table creation statement template with parameter configuration of the output table entity, and obtain a table creation statement instance of the target table.

[0033] A table creation module is configured to utilize a multi-source target table autonomous creation model to automatically connect a data source and perform a target table creation process according to a data source type of the output table entity, and obtain a created target table. The table creation module is configured to utilize a Spark offline computing engine to implement the creation of the target table and implement an entire process of offline data processing based on a heterogeneous type data table.

[0034] In another aspect, a computer device is provided, which includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps of the multi-source heterogeneous offline data processing method are implemented.

[0035] In another aspect, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the multi-source heterogeneous offline data processing method are implemented.

[0036] One of the above technical solutions has the following advantages and beneficial effects:

[0037] The multi-source heterogeneous offline data processing method, system, device and medium reduce manual operation based on a visual interface configuration, reduce system development difficulty, realize zero-code development through a DAG plug-in data table configuration, increase flexibility through dynamic arrangement of table relationships, design a data table entity modeling model to perform feature modeling on data tables in integrated plug-ins, design multi-source type table creation statement templates to reduce manual coding, unify data source connections and increase system expandability, design a multi-source table creation instance construction process to automatically construct target table creation statements, design a multi-source type target table creation process to autonomously perform efficient target table creation operations, and finally, realize data integration in multiple aspects through offline data table processing, greatly reduce the offline data processing complexity of multi-source heterogeneous type data tables, solve the unified management and control problem of multi-source heterogeneous data, solve the problem of disordered data development due to numerous processing procedures, and also reduce labor costs and simplify system development procedures. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0039] Figure 1 A flowchart of a multi-source heterogeneous offline data processing method in an embodiment;

[0040] Figure 2 A schematic diagram of an offline integrated plug-in type in an embodiment;

[0041] Figure 3 A schematic diagram of offline data table relationship configuration in an embodiment;

[0042] Figure 4 A flowchart of table creation statement instance construction in an embodiment;

[0043] Figure 5 A flowchart of target table autonomous creation in an embodiment;

[0044] Figure 6 A module composition framework schematic diagram of a multi-source heterogeneous offline data processing system in an embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application.

[0047] It is to be appreciated that a reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another.

[0048] It will be understood by those within the art that, in some aspects of the application, subordinate features can be substituted for the features set forth and described herein while still providing a method in accordance with principles of the application. Thus, for example, those skilled in the art will understand that an element or component described herein as being tasked to perform a function can be implemented as software, hardware, firmware, or some combination of software, hardware, and firmware that operates to perform that function. As such, the embodiments described herein are to be considered in a descriptive sense only and not limiting. Therefore, no limitation is implied based on the description herein of one implementation or implementation in a certain embodiment.

[0049] In the offline data integration process, target table construction and processing are crucial. In the process of implementing the present application, the inventors found that although many scholars in the prior art have made some technical scheme explorations, they can achieve a certain degree of simple construction process, support multiple different databases, simplify data development by using preset templates, and automatically create target tables. However, there is still a lack of a unified standard model in the task of constructing a data table with a complex development process, and there are obvious limitations in one-stop offline data integration development. For example, through the matching of preprocessed data and multiple types of preset templates, and the method of establishing a data table by script and referring to a preset template, the manual script coding method is used, which increases the cost of manual development; the method of constructing a target data set by parsing multiple different databases or data tables according to SQL (Structured Query Language, which is a programming language used to manage relational databases. It is used to perform various database operations, such as querying data, inserting, updating, and deleting data, and creating and modifying database table structures) statements, needs to parse SQL statements, and the parsing process is complex in a complex processing process; the method of instructing the target data node to create a target table by sending a data table construction instruction to the target data node, the instruction construction customization degree is high, and the expansibility is poor in a multi-source system; the method of generating a data analysis algorithm according to user demand, and analyzing the data structure of multiple dimension tables by using the data analysis algorithm to construct a target table, there is a lot of repeated development of code in data table processing.

[0050] In view of the above technical defects of the prior art, the purpose of the present application is to create a simple and efficient off-line data processing standard scheme and system based on heterogeneous data tables, compatible with multiple heterogeneous data sources, solving the problem of data islands and depositing data assets; based on visual interface configuration to reduce manual operation and reduce system development difficulty; using DAG (Directed Acyclic Graph) plug-in data table configuration to realize zero-code development, and dynamically arranging table relationships to increase flexibility; designing a data table entity modeling model to model the data tables in the integrated plug-in; designing a multi-source type table creation statement template to reduce manual coding, unify data source connection, and increase system scalability; designing a multi-source table creation instance construction process to automatically construct the target table creation statement; designing a multi-source type target table creation process to automatically complete the target table creation operation; finally, through off-line data table processing, data integration is realized in multiple aspects, greatly reducing the complexity of off-line data processing of multi-source heterogeneous data tables, solving the problem of unified management and control of multi-source heterogeneous data, solving the problem of messy data development due to complex processing flow, and also reducing labor costs and simplifying system development process.

[0051] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0052] Please refer to Figure 1 In one embodiment, a multi-source heterogeneous off-line data processing method is provided, comprising the following processing steps S10 to S23:

[0053] S10, adding each off-line data source in the data source management page of the heterogeneous data source, configuring the data source connection information and completing the connectivity test; the off-line data source includes MySQL data source, DB2 data source, Hive data source, Phoenix data source, ClickHouse data source and Doris data source.

[0054] It can be understood that on the multi-source heterogeneous off-line data management platform, in the data set management page of each heterogeneous data source, multiple off-line data sources existing in the current application scenario are added, the connection information of these data sources is configured and tested for connectivity, and then the connection information is saved to the data source table configured by the system. These off-line data sources can include but are not limited to the existing ones in the art: MySQL data source, DB2 data source, Hive data source, Phoenix data source, ClickHouse data source and Doris data source, to determine the various heterogeneous off-line data sources involved in the current processing task.

[0055] S11, selecting the data table in the data source in the data set management page of the heterogeneous data source, synchronizing the metadata information of the data table and saving the metadata information to the data set table; the metadata information includes the data table name, the field name, the field type, the primary key field, the partition field and the index field.

[0056] It can be understood that next, in the dataset management page, the data table contained in the aforementioned offline data source is selected, the metadata information of the data table is synchronized, and the metadata information is saved to the dataset table corresponding to the dataset management page. These metadata information may include the data table name, field name, field type, primary key field, partition field and index field, thereby synchronizing the data table information of various offline data sources in the current application scenario to the dataset table for subsequent processing steps to call.

[0057] S12: Create and edit a DAG integration canvas based on the DAG interface operation and save it to the canvas table.

[0058] It can be understood that in this step, the DAG integration canvas for offline data integration is constructed. The integration canvas can be created and edited directly through the DAG interface and saved to the canvas table of the platform.

[0059] S13, using the offline integration canvas plug-in model to dynamically select an integration plug-in based on the DAG drag and drop in the DAG integration canvas; the integration plug-in includes an input plug-in, a conversion plug-in, and an output plug-in.

[0060] As you can understand, the offline integrated canvas plug-in is configured next. Specifically, the offline integrated canvas plug-in model is used to dynamically select integrated plug-ins based on the DAG drag-and-drop in the aforementioned DAG integrated canvas. These integrated plug-ins may include input plug-in I, transformation plug-in F, and output plug-in O. The offline integrated canvas plug-in model used can be represented by the following formula y1:

[0061]

[0062] Among them, DagPlugin() represents the integrated canvas plug-in model, n1 is the number of input plug-ins, n2 is the number of conversion plug-ins, n3 is the number of output plug-ins and n3∈[1,n2].

[0063] S14. In the DAG integration canvas, select an input plug-in and configure the input table, select a transformation plug-in and configure the transformation rules, and select an output plug-in and configure the output table.

[0064] It is understandable that Figure 2 As shown, the data table is then configured. In the aforementioned DAG integration canvas, select the input plug-in and configure the input table. These input plug-ins can include MySQL input plug-in, DB2 input plug-in, Hive input plug-in, Phoenix input plug-in, ClickHouse input plug-in and Doris input plug-in, and configure the data table of the corresponding plug-in type as the input table TI.

[0065] In the foregoing DAG integration canvas, a conversion plug-in is selected and a conversion rule is configured, the conversion plug-in can include an SQL multi-table association plug-in, a Filter data filtering plug-in, a Derive field derivation plug-in, a Concat field connection plug-in, a Substr character truncation plug-in and a Null null value processing plug-in, and the conversion rule of the corresponding plug-in type is configured as a conversion rule TF.

[0066] In the foregoing DAG integration canvas, an output plug-in is selected and an output table is configured, the output plug-in can include a MySQL output plug-in, a DB2 output plug-in, a Hive output plug-in, a Phoenix output plug-in, a ClickHouse output plug-in and a Doris output plug-in, and the data table of the corresponding plug-in type is configured as an output table TO. In this way, when data is input, the corresponding data table in the foregoing heterogeneous type data source is selected as an input table, and when data is output, the corresponding data table in the heterogeneous type data source is selected as an output table, and the conversion rule between the heterogeneous type data tables is configured, so as to ensure that the entire process can accurately implement the entire process of offline data processing based on the heterogeneous type data table, that is, the entire scheme is finally completed by using the data conversion processing between the data tables to complete the entire process of offline data processing based on the heterogeneous type data table.

[0067] S15, in the DAG integration canvas, the conversion relationship between the data tables in the integration plug-in is configured by using the data table relationship configuration model.

[0068] It can be understood that, as shown in Figure 3 Table relationship configuration is then performed, that is, in the foregoing DAG integration canvas, the conversion relationship between the tables in the integration plug-in is constructed by using the data table relationship configuration model, for constructing a target table. Specifically, the conversion relationship between the tables in the integration plug-in is constructed by dynamically arranging the relationships of the three integration plug-ins (I, F and O), and the conversion relationship is that a plurality of input tables TI and a plurality of conversion rules TF generate a plurality of output tables TO. The data table relationship configuration model used can be as shown in the following formula y2:

[0069]

[0070] Wherein, TRM() represents a data table relationship model, n1 is the number of input tables, n2 is the number of conversion rules, n3 is the number of output tables, r is the number of table relationships and r∈[n2, n1×n2], TI is an input table, TF is a conversion rule, TO is an output table, and an arrow represents a relationship conversion direction.

[0071] S16, in the DAG integration canvas, data table entity modeling is performed on the selected input table and output table by using a data table entity modeling model, and data modeling is performed on the conversion rule by using a conversion rule entity modeling model.

[0072] It can be understood that data table entity modeling can be performed subsequently, and the selected input table TI and output table TO are data modeled in the aforementioned DAG integration canvas by using a data table entity modeling model. Meanwhile, the entity modeling features can include a data source DS, a table name TN, a field name CN, a field type CT, a primary key field PK, a partition field PF, and an index field IF. The data table entity modeling model used can be as shown in the following formula y3:

[0073]

[0074] Wherein, TEM() represents a data table modeling model, m1 is the number of data fields, m2 is the number of primary key fields, m3 is the number of partition fields, and m4 is the number of index fields, and m2, m3, and m4 are all less than m1.

[0075] The conversion rule TF is data modeled in the aforementioned DAG integration canvas by using a conversion rule entity modeling model. Meanwhile, the entity modeling features can include a data source DS, a source table name TN, a multi-table relationship TR, a field name CN, and a field type CT. The conversion rule entity modeling model used can be as shown in the following formula y4:

[0076]

[0077] Wherein, TFM() represents a conversion rule modeling model, k1 is the number of source tables, k2 is the number of table relationships and k2∈[k1, k1×k1], and k3 is the number of fields.

[0078] S17, in the DAG integration canvas, the input table entity is obtained by using an input table entity construction model to perform instance conversion on the input table configuration in the input plug-in and saved to the data table entity table.

[0079] It can be understood that after the data table entity modeling is completed, the input table entity can be instantiated. In the aforementioned DAG integration canvas, the input table entity is obtained by using an input table entity construction model to perform instance conversion on the input table configuration in the input plug-in and saved to the data table entity table. Specifically, in the first step, the input table configuration of the input table TI in the input plug-in is obtained, and the input table configuration can include a data source DS, a table name TN, a field name CN, a field type CT, a primary key field PK, a partition field PF, and an index field IF. In the second step, the input table configuration and other modeling features are configured and instance-converted based on the data table entity modeling model, and finally, the input table entity TIE is constructed. The input table entity construction model used can be as shown in the following formula y5:

[0080]

[0081] Wherein, TIEM() represents an input table entity model, n1 is the number of input tables, TI is an input table, IC is an input table configuration, TIE is an input table entity, TEM is a data table modeling model, and the arrow represents an instantiation conversion direction.

[0082] S18, in the DAG integration canvas, the conversion rule configuration in the conversion plug-in is instantiated and converted by using the conversion rule entity construction model, to obtain a conversion rule entity and save it to a conversion rule entity table.

[0083] It can be understood that the conversion rule entity instantiation is performed, in the aforementioned DAG integration canvas, the conversion rule configuration in the conversion plug-in is instantiated and converted by using the conversion rule entity construction model, to obtain a conversion rule entity and save it to a conversion rule entity table. Specifically, the first step is to obtain the input table configuration of the input table TI in the input plug-in, which can include a data source DS and a table name TN; the second step is to obtain the conversion rule configuration of the conversion rule TF in the conversion plug-in, which can include a multi-table relationship TR, a field name CN, and a field type CT; the third step is to configure and instantiate the aforementioned input table configuration and conversion rule configuration, etc. based on the conversion rule entity modeling model, to finally construct the conversion rule entity TFE. The conversion rule entity construction model used can be as shown in the following formula y6:

[0084]

[0085] Wherein, TFEM() represents a conversion rule entity model, n1 is the number of input tables, n2 is the number of conversion rules, TI is an input table, TF is a conversion rule, IC is an input table configuration, FC is a conversion rule configuration, TFM is a conversion rule modeling model, TFE is a conversion rule entity, and the arrow represents an instantiation conversion direction.

[0086] S19, in the DAG integration canvas, the conversion rule configuration in the conversion plug-in is instantiated and converted by using the conversion rule entity construction model, to obtain a conversion rule entity and save it to a conversion rule entity table.

[0087] It can be understood that in the foregoing DAG integration canvas, the output table entity is constructed by using the output table entity construction model, and the conversion of the model and the data table entity modeling model based on the data table relationship configuration model is configured. Specifically, the first step is to obtain the input table entity TIE in the input plug-in and the conversion rule entity TFE in the conversion plug-in; the second step is to obtain the conversion relationship between the data tables in the integration plug-in based on the data table relationship configuration model; the third step is to convert the input table entity TIE and the conversion rule entity TFE based on the conversion relationship between the data tables in the integration plug-in to obtain the output table configuration; and the fourth step is to configure and instantiate the conversion of the modeling features such as the output table configuration based on the data table entity modeling model, and finally construct the output table entity TOE. The output table entity construction model used can be as shown in the following formula y7:

[0088]

[0089] Wherein, TOEM() represents the output table entity model, n1 is the number of input tables, n2 is the number of conversion rules, n3 is the number of output tables and n3∈[1,n2], TIE is the input table entity, TFE is the conversion rule entity, TRM is the data table relationship model, TEM is the data table modeling model, TOE is the output table entity, and the arrow represents the entity conversion direction.

[0090] S20, using a multi-source table building template screening process model, obtaining the data source type of the output table entity, and through type judgment and template screening, obtaining the table building statement template of the target table.

[0091] It can be understood that the data types can include MySQL data sources, DB2 data sources, Hive data sources, Phoenix data sources, ClickHouse data sources, and Doris data sources, and the table building statement templates corresponding to different data types of data sources can be obtained by a pre-defined manner.

[0092] In some embodiments, further, the above multi-source heterogeneous offline data processing method can further include the following steps:

[0093] According to the data source type of each offline data source, load each table building statement template defined in advance; the table building statement template includes a MySQL table building statement template, a DB2 table building statement template, a Hive table building statement template, a Phoenix table building statement template, a ClickHouse table building statement template, and a Doris table building statement template.

[0094] Specifically, a plurality of table creation statement templates of data source types are designed, which can include: a MySQL data source, a DB2 data source, a Hive data source, a Phoenix data source, a ClickHouse data source and a Doris data source. Wherein, a MySQL table creation statement template MT is defined, which can be as follows:

[0095] CREATE TABLE${tn}(${cn1}${ct1},${cn2}${ct2},${cn3}${ct3},...,PRIMARYKEY($pk))ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='${cm}'.

[0096] A DB2 table creation statement template BT is defined, which can be as follows:

[0097] CREATE TABLE${tn}(${cn1}${ct1},${cn2}${ct2},${cn3}${ct3},...)IN${tn}ORGANIZE BY ROW.

[0098] A Hive table creation statement template HT is defined, which can be as follows:

[0099] CREATE TABLE${tn}(${cn1}${ct1},${cn2}${ct2},${cn3}${ct3},...)PARTITIONED BY(${pf}string)ROW FORMAT SERDE${rf}STORED AS INPUTFORMAT${if}.

[0100] A Phoenix table creation statement template PT is defined, which can be as follows:

[0101] CREATE TABLE${tn}(${cn1}${ct1},${cn2}${ct2},${cn3}${ct3},...,CONSTRAINT my_pk PRIMARY KEY($pk)).

[0102] A ClickHouse table creation statement template CT is defined, which can be as follows:

[0103] CREATE TABLE${tn}(${cn1}${ct1},${cn2}${ct2},${cn3}${ct3},...)ENGINE=MergeTree()PARTITION BY toYYYYMM(${pf})ORDER BY${oc}。

[0104] The Doris table creation statement template DT can be defined as follows:

[0105] CREATE TABLE${tn}(${cn1}${ct1},${cn2}${ct2},${cn3}${ct3},...)DUPLICATE KEY($cn1)DISTRIBUTED BY HASH($cn2)BUCKETS1.

[0106] wherein ${} represents a template variable, tn is a table name, cn represents a field name, ct is a field type, pk is a primary key field, pf is a partition field, oc is an order field, rf is a line feed, if is a storage format, and cm is a table creation note. CREATE TABLE indicates table creation, PRIMARY KEY indicates a primary key, ENGINE=InnoDB DEFAULT CHARSET indicates that the database engine is set to the database default storage engine InnoDB, and the meanings of the remaining statements can be understood in the same way as the existing SQL table creation statements in the art.

[0107] Further, the multi-source heterogeneous offline data processing method described above can further include the following steps:

[0108] The multi-source template filter model is used to determine and filter the data source type, and obtain a table creation statement template corresponding to the data source type.

[0109] It can be understood that the multi-source template filter model is used to pre-design a template filtering rule based on the data source type, that is, to determine and filter the data source type, and obtain a table creation statement template corresponding to the data source type. The multi-source template filter model used can be as shown in the following formula y8:

[0110]

[0111] wherein Selector() is a template filtering rule, x is a data source type, and MT, BT, HI, PT, CT, and DT are multi-source table creation statement templates.

[0112] Further, as Figure 4As shown, the data source type of the output table entity is obtained by using the multi-source table building template screening process model, and the table building statement template of the target table is obtained through type determination and template screening. Specifically, the data source DS configuration is obtained in the output table entity TOE, and the data source type DT is parsed, and the table building statement template of the output table corresponding to the data source type is screened out according to the rule determination of the multi-source template selector model Selector. The multi-source table building template screening process model used can be as shown in the following formula y9:

[0113]

[0114] Wherein, TSPM{} is a template screening model, n3 is the number of output tables, TOE is an output table entity, DS is a data source configuration, DT is a data source type, Selector is a template selector, Template is a table building statement template, and the arrow indicates a template screening process.

[0115] S21, in the DAG integration canvas, the table building statement template is matched with the parameter configuration of the output table entity by using the multi-source target table building instance construction model, and the table building statement instance of the target table is obtained.

[0116] It can be understood that after the above processing steps are completed, the table building instance construction can be performed, and in the aforementioned DAG integration canvas, the table building statement template screened out is matched with the parameter configuration of the output table entity by using the multi-source target table building instance construction model, and the table building statement instance of the target table is obtained.

[0117] Specifically, the first step is to obtain the configuration characteristics (i.e., the output table entity parameter configuration) of the output table entity TOE, which can include the data source DS, the table name TN, the field name CN, the field type CT, the primary key field PK, the partition field PF, and the index field IF; the second step is to obtain the table building statement template Template of the target table based on the multi-source table building template screening process model and using the template selector rule determination; and the third step is to map and match the table building statement template screened out with the aforementioned configuration characteristics, and finally generate the table building statement instance TOCDL of the target table. The multi-source target table building instance construction model used can be as shown in the following formula y 10 As shown:

[0118]

[0119] Wherein, TEPM{} is a table building instance construction model, n3 is the number of output tables, TOE is an output table entity, TSPM is a template screening model, Template is a table building statement template, and TOCDL is a table building statement instance. The arrow indicates the table building instance construction direction.

[0120] S22, a multi-source target table is autonomously created, a data source is automatically connected and a target table creation process is performed according to a data source type of an output table entity, and a created target table is obtained; wherein the creation of the target table is implemented by using a Spark offline computing engine, and the whole process of offline data processing based on the heterogeneous type data table is implemented.

[0121] It can be understood that the designed multi-source data source connector model can construct a plurality of heterogeneous type Jdbc connection instances according to different types of data source configurations. The Jdbc connection is a Java API for executing SQL statements, and can provide unified access for a plurality of relational databases. The required data source configuration characteristics can include a data source type, a server IP, a port number, a database name, an instance name, a user name and a user password. The designed multi-source data source connector model is as shown in the following formula y 11

[0122]

[0123] Among them, MultiJdbc{} is a multi-source data source connector, q1 is the data volume of the data source, DS is the data source type, IP is the server IP, HP is the port number, DN is the database name, IN is the instance name, UN is the user name, and PP is the user password.

[0124] Specifically, the first step is to construct a model based on a multi-source target table table instance, and obtain a table creation statement instance TOCDL according to output table configuration; the second step is to set a plurality of types of data source connectors MultiJdbc, and the connector types can include MySQL, DB2, Hive, Phoenix, ClickHouse and Doris; the third step is to automatically connect the data source by using the MultiJdbc multi-source connector according to the data source configuration of the target table, and run the table creation statement instance TOCDL to execute the target table creation process; the fourth step is to obtain the final creation result and complete the creation operation. The multi-source target table autonomous creation model used can be as shown in the following formula y 12

[0125]

[0126] Among them, n3 is the number of table instances, TOE is the output table entity, TEPM is the table instance construction model, TOCDL is the table creation statement instance, MultiJdbc is the multi-source data source connector, Result is the table creation result, and the arrow represents the target table creation process.

[0127] ​​It can be understood that the target data table is created and the whole process of offline data processing based on the heterogeneous type data table is realized by using the existing Spark offline computing engine in the art. Specifically, the Spark offline computing engine is used to build the data table and the target table in each link of data processing, and to realize the relationship transformation between the input table, the conversion rule and the output table; through the data conversion processing between the data tables, the whole process of offline data processing is finally completed.

[0128] The above multi-source heterogeneous offline data processing method reduces manual operation and reduces system development difficulty based on visual interface configuration; zero-code development is realized by using DAG plug-in data table configuration, and flexibility is increased by dynamic arrangement of table relationship; a data table entity modeling model is designed to model the data table in the integrated plug-in; a multi-source type table building statement template is designed to reduce manual coding, unify data source connection, and increase system expansibility; a multi-source table building instance construction process is designed to automatically construct the target table building statement; a multi-source type target table creation process is designed to automatically complete efficient target table creation operation; finally, data integration is realized in multiple links through offline data table processing, which greatly reduces the complexity of offline data processing of multi-source heterogeneous type data table, solves the problems of unified management and control of multi-source heterogeneous data, solves the problem of messy data development due to numerous processing procedures, and also reduces labor cost and simplifies system development process.

[0129] In one embodiment, further, as shown in Figure 5 The above multi-source heterogeneous offline data processing method can further include the following processing steps:

[0130] In the target table creation process, if the target table creation is successful, a result indication of successful creation is returned;

[0131] If the target table creation fails, a result indication of failed creation is returned.

[0132] It can be understood that in the target table creation process, the platform also synchronously monitors the table building result and returns the corresponding creation result data to the foreground, so as to efficiently and intuitively indicate the result of automatic table building, facilitate monitoring and timely handling of the table building process, and improve the reliability of automatic table building.

[0133] It should be understood that although each step in the above processes Figure 1 and Figure 5 is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, the above processes Figure 1 and Figure 5At least one of the steps of the method can comprise a plurality of sub-steps or a plurality of stages, which sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the order of the sub-steps or stages is not necessarily sequential, but can be performed in rotation or alternation with other steps or sub-steps or stages of other steps.

[0134] In one embodiment, as shown in FIG. 1, a multi-source heterogeneous offline data processing system 100 is provided, comprising a source configuration module 11, a meta synchronization module 12, a canvas construction module 13, a canvas plug-in module 14, a data table configuration module 15, a table relationship configuration module 16, an entity modeling module 17, an input entity module 18, a conversion entity module 19, an output entity module 20, a template screening module 21, an instance matching module 22, and a table creation module 23. Among them: Figure 6 The source configuration module 11 is used to add each offline data source in the data source management page of the heterogeneous data source, configure the data source connection information and complete the connectivity test; the offline data source includes MySQL data source, DB2 data source, Hive data source, Phoenix data source, ClickHouse data source and Doris data source. The meta synchronization module 12 is used to select the data table in the data source in the data set management page of the heterogeneous data source, synchronize the metadata information of the data table and save the metadata information to the data set table; the metadata information includes the data table name, the field name, the field type, the primary key field, the partition field and the index field.

[0135]

[0136] ​The canvas building module 13 is configured to create and edit the DAG integrated canvas based on the DAG interface operation and save to the canvas table. The canvas plug-in module 14 is configured to utilize the offline integrated canvas plug-in model to dynamically select the integrated plug-in based on the DAG in the DAG integrated canvas; the integrated plug-in includes the input plug-in, the conversion plug-in and the output plug-in. The data table configuration module 15 is configured to select the input plug-in and configure the input table, select the conversion plug-in and configure the conversion rule, and select the output plug-in and configure the output table in the DAG integrated canvas. The table relationship configuration module 16 is configured to utilize the data table relationship configuration model to build the conversion relationship between the data tables in the integrated plug-in in the DAG integrated canvas. The entity modeling module 17 is configured to utilize the data table entity modeling model to perform the data table entity modeling on the selected input table and output table, and utilize the conversion rule entity modeling model to perform the data modeling on the conversion rule in the DAG integrated canvas. The input entity module 18 is configured to utilize the input table entity building model to perform the instantiation conversion on the input table configuration in the input plug-in, obtain the input table entity and save to the data table entity table in the DAG integrated canvas. The conversion entity module 19 is configured to utilize the conversion rule entity building model to perform the instantiation conversion on the conversion rule configuration in the conversion plug-in, obtain the conversion rule entity and save to the conversion rule entity table in the DAG integrated canvas. The output entity module 20 is configured to utilize the output table entity building model, the data table relationship configuration model and the data table entity modeling model to build the output table entity and save to the data table entity table in the DAG integrated canvas. The template screening module 21 is configured to utilize the multi-source table building template screening process model to obtain the data source type of the output table entity, and then perform the type determination and template screening to obtain the table building statement template of the target table. The instance matching module 22 is configured to utilize the multi-source target table building instance building model to match the table building statement template with the parameter configuration of the output table entity, and obtain the table building statement instance of the target table in the DAG integrated canvas. The table creation module 23 is configured to utilize the multi-source target table autonomous creation model to automatically connect the data source and perform the target table creation process according to the data source type of the output table entity, and obtain the created target table; wherein the creation of the target table and the whole process of the offline data processing based on the heterogeneous type data table are realized by utilizing the Spark offline computing engine.

[0137] The multi-source heterogeneous offline data processing system 100 reduces manual operation and system development difficulty based on visual interface configuration, realizes zero-code development through DAG plug-in data table configuration, increases flexibility through dynamic table relationship arrangement, designs a data table entity modeling model to perform feature modeling on data tables in integrated plug-ins, designs multi-source type table creation statement templates to reduce manual coding, unify data source connection, and increase system scalability, designs a multi-source table creation instance construction process to automatically construct target table creation statements, designs a multi-source type target table creation process to autonomously perform efficient target table creation operations, and finally, realizes data integration in multiple aspects through offline data table processing, greatly reduces the offline data processing complexity of multi-source heterogeneous type data tables, solves the unified management and control problem of multi-source heterogeneous data, solves the problem of disordered data development due to numerous processing procedures, reduces manual costs, and simplifies system development procedures.

[0138] In one embodiment, the multi-source heterogeneous offline data processing system 100 can further include a template loading module configured to load predefined table creation statement templates according to data source types of each offline data source; the table creation statement templates include MySQL table creation statement templates, DB2 table creation statement templates, Hive table creation statement templates, Phoenix table creation statement templates, ClickHouse table creation statement templates, and Doris table creation statement templates.

[0139] In one embodiment, the multi-source heterogeneous offline data processing system 100 can further include a type screening module configured to determine and screen templates for data source types using a multi-source template screening model to obtain table creation statement templates corresponding to the data source types.

[0140] In one embodiment, the table creation module, in the target table creation process, is further configured to return a creation success result indication when the target table creation is successful, and return a creation failure result indication when the target table creation fails.

[0141] For specific limitations of the multi-source heterogeneous offline data processing system 100, refer to the corresponding limitations of the multi-source heterogeneous offline data processing method in the foregoing description, which will not be repeated here. Each module in the multi-source heterogeneous offline data processing system 100 can be realized by software, hardware, or a combination thereof, in whole or in part. Each module can be embedded in or independent of a device with data processing function in hardware form, or stored in the memory of the device in software form, so that the processor can call and execute the operations corresponding to each module. The device can be, but is not limited to, various types of data computing and processing devices in the art.

[0142] In one embodiment, a computer device is also provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following processing steps when executing the computer program: adding each offline data source in a data source management page of the heterogeneous data source, configuring data source connection information and completing connectivity test; the offline data source includes a MySQL data source, a DB2 data source, a Hive data source, a Phoenix data source, a ClickHouse data source and a Doris data source; selecting a data table in the data source in a data set management page of the heterogeneous data source, synchronizing metadata information of the data table and saving the metadata information to a data set table; the metadata information includes a data table name, a field name, a field type, a primary key field, a partition field and an index field; creating and editing a DAG integrated canvas based on a DAG interface operation and saving to a canvas table; dynamically selecting an integrated plug-in based on DAG in the DAG integrated canvas by using an offline integrated plug-in plug-in model; the integrated plug-in includes an input plug-in, a conversion plug-in and an output plug-in; selecting the input plug-in in the DAG integrated canvas and configuring an input table, selecting the conversion plug-in and configuring a conversion rule, and selecting the output plug-in and configuring an output table; in the DAG integrated canvas, a conversion relationship between data tables in the integrated plug-in is constructed by using a data table relationship configuration model; in the DAG integrated canvas, data table entity modeling is performed on the selected input table and output table by using a data table entity modeling model, and data modeling is performed on the conversion rule by using a conversion rule entity modeling model; in the DAG integrated canvas, the input table configuration in the input plug-in is instantiated and converted by using an input table entity construction model, to obtain an input table entity and save to a data table entity table; in the DAG integrated canvas, the conversion rule configuration in the conversion plug-in is instantiated and converted by using a conversion rule entity construction model, to obtain a conversion rule entity and save to a conversion rule entity table; in the DAG integrated canvas, an output table entity is constructed by using an output table entity construction model, a data table relationship configuration model and a data table entity modeling model, and saved to the data table entity table; after obtaining the data source type of the output table entity, a table creation statement template of a target table is obtained through type judgment and template screening by using a multi-source table creation template screening process model; in the DAG integrated canvas, a table creation statement instance of the target table is obtained by matching the table creation statement template and the parameter configuration of the output table entity by using a multi-source target table table creation instance construction model; a created target table is obtained by automatically connecting the data source and executing the target table creation process according to the data source type of the output table entity by using a multi-source target table autonomous creation model; wherein the creation of the target table and the entire process of offline data processing based on the heterogeneous type data table are implemented by using a Spark offline computing engine.

[0143] It can be understood that the above computer device further includes other hardware and software components not listed in the description in addition to the above-mentioned memory and processor, and the specific computer device model can be determined according to different application scenarios, and the description will not be listed in detail.

[0144] In one embodiment, the processor also implements the steps or sub-steps added in each embodiment of the above multi-source heterogeneous offline data processing method when executing the computer program.

[0145] In one embodiment, a computer readable storage medium is also provided, and the computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to implement the following processing steps: adding each offline data source, configuring data source connection information and completing connectivity test in the data source management page of the heterogeneous data source; the offline data source includes a MySQL data source, a DB2 data source, a Hive data source, a Phoenix data source, a ClickHouse data source and a Doris data source; selecting a data table in the data source in the data set management page of the heterogeneous data source, synchronizing metadata information of the data table and saving the metadata information to a data set table; the metadata information includes a data table name, a field name, a field type, a primary key field, a partition field and an index field; creating and editing a DAG integrated canvas based on a DAG interface operation and saving to a canvas table; dynamically selecting an integrated plug-in based on DAG in the DAG integrated canvas by using an offline integrated plug-in plug-in model; the integrated plug-in includes an input plug-in, a conversion plug-in and an output plug-in; selecting the input plug-in in the DAG integrated canvas and configuring an input table, selecting the conversion plug-in and configuring a conversion rule, and selecting the output plug-in and configuring an output table; in the DAG integrated canvas, a conversion relationship between data tables in the integrated plug-in is constructed by using a data table relationship configuration model; in the DAG integrated canvas, data table entity modeling is performed on the selected input table and output table by using a data table entity modeling model, and data modeling is performed on the conversion rule by using a conversion rule entity modeling model; in the DAG integrated canvas, the input table configuration in the input plug-in is instantiated and converted by using an input table entity construction model, an input table entity is obtained and saved to a data table entity table; in the DAG integrated canvas, the conversion rule configuration in the conversion plug-in is instantiated and converted by using a conversion rule entity construction model, a conversion rule entity is obtained and saved to a conversion rule entity table; in the DAG integrated canvas, an output table entity is constructed by using an output table entity construction model, a data table relationship configuration model and a data table entity modeling model, and saved to the data table entity table; after obtaining the data source type of the output table entity, a table creation statement template of a target table is obtained through type determination and template screening by using a multi-source table creation template screening process model; in the DAG integrated canvas, the table creation statement template is matched with the parameter configuration of the output table entity by using a multi-source target table creation instance construction model, and a table creation statement instance of the target table is obtained; a target table is created according to the data source type of the output table entity by using a multi-source target table autonomous creation model, and a created target table is obtained; wherein the creation of the target table and the entire process of offline data processing based on the heterogeneous type data table are implemented by using a Spark offline computing engine.

[0146] In one embodiment, the computer program executed by the processor can also implement the steps or sub-steps added in each embodiment of the multi-source heterogeneous offline data processing method.

[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus dynamic random access memory (Rambus DRAM, RDRAM for short) and interface dynamic random access memory (DRDRAM).

[0148] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0149] The above embodiments only express several implementation manners of the present application, and the description is specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A multi-source heterogeneous offline data processing method, characterized in that: Including steps: In the data source management page of the heterogeneous data source, add each offline data source, configure the data source connection information and complete the connectivity test; the offline data sources include MySQL data source, DB2 data source, Hive data source, Phoenix data source, ClickHouse data source and Doris data source; On the dataset management page of a heterogeneous data source, select a data table in the data source, synchronize the metadata information of the data table, and save the metadata information to the dataset table; the metadata information includes the data table name, field name, field type, primary key field, partition field, and index field; Create and edit DAG integration canvas based on DAG interface operations and save it to the canvas table; Dynamically select an integration plug-in based on DAG drag and drop in the DAG integration canvas using an offline integration canvas plug-in model; the integration plug-in includes an input plug-in, a conversion plug-in, and an output plug-in; In the DAG integration canvas, select an input plugin and configure the input table, select a transformation plugin and configure the transformation rules, select an output plugin and configure the output table; In the DAG integration canvas, a data table relationship configuration model is used to construct a conversion relationship between data tables in the integration plug-in; In the DAG integration canvas, the selected input table and output table are modeled using a data table entity modeling model, and the transformation rule is modeled using a transformation rule entity modeling model; In the DAG integration canvas, the input table entity is used to build a model to instantiate and convert the input table configuration in the input plug-in, obtain the input table entity and save it to the data table entity table; In the DAG integration canvas, the conversion rule configuration in the conversion plug-in is instantiated and converted using the conversion rule entity construction model to obtain a conversion rule entity and save it to the conversion rule entity table; In the DAG integration canvas, the output table entity is constructed using the output table entity construction model, the data table relationship configuration model, and the data table entity modeling model, and the output table entity is constructed and saved to the data table entity table; Using a multi-source table creation template screening process model, the data source type of the output table entity is obtained, and then the table creation statement template of the target table is obtained through type determination and template screening; In the DAG integration canvas, a table creation statement template is matched with the parameter configuration of the output table entity using a multi-source target table creation instance construction model to obtain a table creation statement instance of the target table; A multi-source target table autonomously creates a model, automatically connects to the data source according to the data source type of the output table entity, and executes the target table creation process to obtain the created target table; wherein, the creation of the target table is achieved by adopting the Spark offline computing engine, and the entire offline data processing process based on heterogeneous data tables is realized.

2. The multi-source heterogeneous offline data processing method according to claim 1, characterized in that: Also includes the steps: Loading predefined table creation statement templates according to the data source type of each offline data source; The table creation statement templates include MySQL table creation statement templates, DB2 table creation statement templates, Hive table creation statement templates, Phoenix table creation statement templates, ClickHouse table creation statement templates and Doris table creation statement templates.

3. The multi-source heterogeneous offline data processing method according to claim 2, characterized in that: Before the step of obtaining the target table creation statement template by using the multi-source table creation template screening process model, after obtaining the data source type of the output table entity through type determination and template screening, the step further includes: The multi-source template filter model is used to determine the data source type and filter the template to obtain the table creation statement template corresponding to the data source type.

4. The multi-source heterogeneous offline data processing method according to claim 1, characterized in that: Also includes the steps: During the target table creation process, if the target table is created successfully, a result indication of successful creation is returned; If the target table fails to be created, a result indication indicating the creation failure is returned.

5. A multi-source heterogeneous offline data processing system, characterized in that: include: The source configuration module is used to add each offline data source in the data source management page of the heterogeneous data source, configure the data source connection information and complete the connectivity test; the offline data sources include MySQL data source, DB2 data source, Hive data source, Phoenix data source, ClickHouse data source and Doris data source; The metadata synchronization module is used to select a data table in the data source on the data set management page of the heterogeneous data source, synchronize the metadata information of the data table, and save the metadata information to the data set table; the metadata information includes the data table name, field name, field type, primary key field, partition field, and index field; The canvas construction module is used to create and edit the DAG integration canvas based on the DAG interface operation and save it to the canvas table; A canvas plug-in module, configured to dynamically select an integration plug-in based on DAG drag-and-drop in the DAG integration canvas using an offline integration canvas plug-in model; the integration plug-in includes an input plug-in, a conversion plug-in, and an output plug-in; A data table configuration module is used to select an input plug-in and configure an input table, select a transformation plug-in and configure a transformation rule, and select an output plug-in and configure an output table in the DAG integration canvas; A table relationship configuration module is used to construct the conversion relationship between the data tables in the integration plug-in using the data table relationship configuration model in the DAG integration canvas; An entity modeling module is used to perform data table entity modeling on the selected input table and output table using a data table entity modeling model in the DAG integrated canvas, and to perform data modeling on the transformation rule using a transformation rule entity modeling model; An input entity module is used to instantiate and convert the input table configuration in the input plug-in using the input table entity construction model in the DAG integration canvas, obtain the input table entity, and save it to the data table entity table; A conversion entity module is used to instantiate and convert the conversion rule configuration in the conversion plug-in using the conversion rule entity construction model in the DAG integration canvas, obtain the conversion rule entity, and save it to the conversion rule entity table; An output entity module, configured to construct an output table entity in the DAG integrated canvas using the output table entity construction model, the data table relationship configuration model, and the data table entity modeling model, and save the output table entity to the data table entity table; The template screening module is used to use the multi-source table creation template screening process model to obtain the data source type of the output table entity and then perform type determination and template screening to obtain the table creation statement template of the target table; An instance matching module is configured to match the table creation statement template with the parameter configuration of the output table entity in the DAG integration canvas using a multi-source target table creation instance construction model to obtain a table creation statement instance for the target table; The table creation module is used to autonomously create a model using a multi-source target table, automatically connect to the data source according to the data source type of the output table entity, and execute the target table creation process to obtain the created target table; wherein, the creation of the target table is achieved by adopting the Spark offline computing engine, and the entire offline data processing process based on heterogeneous data tables is realized.

6. The multi-source heterogeneous offline data processing system according to claim 5, characterized in that: Also includes: A template loading module, configured to load predefined table creation statement templates according to the data source type of each offline data source; The table creation statement templates include MySQL table creation statement templates, DB2 table creation statement templates, Hive table creation statement templates, Phoenix table creation statement templates, ClickHouse table creation statement templates and Doris table creation statement templates.

7. The multi-source heterogeneous offline data processing system according to claim 6, characterized in that: It also includes a type screening module for using a multi-source template filter model to determine the data source type and screen the template to obtain a table creation statement template corresponding to the data source type.

8. The multi-source heterogeneous offline data processing system according to claim 5, characterized in that: During the target table creation process, the table creation module is further configured to return a result indication of successful creation when the target table is created successfully, and return a result indication of failed creation when the target table is created unsuccessfully.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the multi-source heterogeneous offline data processing method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-source heterogeneous offline data processing method according to any one of claims 1 to 4 are implemented.

Images