One-stop visual lake and warehouse development system based on whole process

Abstract

The invention relates to a one-stop visual lake and warehouse development system based on a whole process, and the system comprises a lake and warehouse planning and processing module which carries out the planning and processing of a lake and warehouse in a mode of defining a data layer; the data source configuration module is used for configuring related information of a source end database and a destination end database which need to be synchronized before the data integration synchronization task is configured; the data development module is used for performing job development, program development, job flow development and operation strategy configuration processing; the job scheduling module is used for supporting a task, a job flow and a whole-flow data processing process; and the operation and maintenance monitoring module is connected with the job scheduling module and is used for monitoring the corresponding task according to the execution state of the task in the current system. According to the one-stop visual lake and warehouse development system based on the whole process, multiple sets of processing engines and deployment modes are provided, switching among multiple tools is not needed, and the lake and warehouse data development efficiency can be improved in a large range.

Description

Technical Field

[0001] The present invention relates to the field of computer software, and in particular to the field of data lake technology, and specifically refers to a one-stop visual lake warehouse development system based on the entire process. Background Art

[0002] With the acceleration of digital construction of enterprises, the application value of data has become increasingly prominent in the process of enterprise development. With the expansion of business and the rapid growth of data, the defects of the traditional single and centralized data construction model have become more and more obvious. The data development chain is long, the degree of integration is low, and the tools are scattered; there are many data types and the requirements change quickly, and batch collaboration is required to shield the inconsistency of the underlying data architecture; the environment configuration is diverse and complicated, and data resources affect each other; there is a large amount of repeated development work between the business lines of various departments, and the business indicators are inconsistent and difficult to trace. There are a series of problems such as data quality and data security. How to fully mine and process data more efficiently, release greater value, and achieve cost reduction and efficiency improvement has become a problem that enterprises are increasingly concerned about and urgently need to solve.

[0003] With the increasing number of data application scenarios, the integrated development architecture of data lake and data warehouse has become an inevitable trend in the future. The integrated development architecture of data lake and data warehouse is a new paradigm that combines the advantages of data lake and data warehouse. It implements data structure and data management functions similar to those in data warehouse on the low-cost storage used for data lake. This ensures the consistency of processing process and results, and provides systematic support for enterprises to build real-time data lake and warehouse applications, realize data integration and aggregation, support the construction of data middle platform, and realize real-time monitoring and business perception.

[0004] The core technology of the full-process one-stop visual lake warehouse development system adopts a hybrid processing architecture that integrates streaming computing capabilities with batch processing technology, which can support large-scale batch data processing and real-time lake warehouse applications at the same time.

[0005] Batch processing is a data processing technology that has been widely used since the data warehouse era. However, as data production scenarios become more and more abundant, data formats become more and more diverse, and data application scenarios have higher and higher requirements for real-time performance, batch processing technologies represented by MapReduce have gradually shown many inadaptabilities.

[0006] Stream computing technology has the characteristics of low latency, high throughput, continuous computing, event triggering, etc. It is good at real-time processing of large amounts of continuously generated data, and can prevent data backlog and loss. Flink, Spark, etc. are the representatives of stream computing that are currently highly praised, and SeaTunnel can shield the differences in underlying data processing technologies, compile a set of codes into programs that can be recognized and executed by multiple engines, and is fully compatible with batch processing technology. The system is based on the SeaTunnel architecture and integrates batch stream engines such as Flink, Spark, and Kettle. It provides three methods of local, cluster, and YARN for flexible deployment on demand, while providing complete job scheduling and operation monitoring capabilities, providing customers with a complete set of enterprise-level lake warehouse big data development, operation, deployment, and operation and maintenance capabilities. On this basis, the existing technology mainly faces the following problems:

[0007] 1. The data development chain is long, the degree of integration is low, and the tools are scattered;

[0008] 2. Data development is cumbersome and complex, with high technical barriers and great learning difficulty;

[0009] 3. There are many data types and the requirements change quickly, so batch and stream collaboration is needed to shield the inconsistency of the underlying data architecture;

[0010] 4. The environment configuration is diverse and complex, and data resources affect each other;

[0011] 5. There is a lot of duplication of development work between business lines of various departments;

[0012] 6. Inconsistent business indicators are difficult to trace, data quality, data security, etc.;

[0013] 7. There is no unified team collaboration platform, and there is a lack of effective management of permissions, resources, tasks, models, tools, etc. Silo-style development causes varying degrees of resource waste and is difficult to reuse effectively;

[0014] 8. High maintenance cost: The types of data sources are constantly increasing, and the number of tools increases linearly with each additional storage type, resulting in a large amount of subsequent maintenance work. Summary of the invention

[0015] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a one-stop visual lake warehouse development system based on the whole process.

[0016] In order to achieve the above objectives, the one-stop visual lake warehouse development system based on the whole process of the present invention is as follows:

[0017] The main features of the one-stop visual lake warehouse development system based on the whole process are that the system includes:

[0018] The lake warehouse planning processing module uses the method of defining the lake warehouse data layer to perform lake warehouse planning processing;

[0019] A data source configuration module is connected to the lake warehouse planning processing module and is used to access heterogeneous data sources, read raw data and write it into the lake warehouse;

[0020] The data development module is connected to the data source configuration module and performs job development, program development, job flow development and operation strategy configuration processing by providing a visual drag-and-drop method;

[0021] A job scheduling module, connected to the data development module, for assembling and executing corresponding tasks in a DAG streaming manner; and

[0022] The operation and maintenance monitoring module is connected to the job scheduling module and is used to perform multi-faceted graphical operation and maintenance monitoring processing on the task instance according to the execution status of the task in the current system.

[0023] Preferably, the lake warehouse planning processing module defines the following data layers in a top-down hierarchical manner:

[0024] The data reference layer is the data preparation area of ​​the data warehouse, which is used to store data that has been extracted, cleaned, and converted;

[0025] The detailed data layer is used to build the most fine-grained detailed fact table based on each specific business feature and store wide table data;

[0026] The summary data layer is used to build summary indicator tables of common granularity based on the indicator requirements of upper-layer applications and products, provide common indicators for the upper layer, and establish summary wide tables and detailed fact tables; and

[0027] The application data layer generates statistical indicator data for storing personalized data products based on upper-layer data processing.

[0028] Preferably, the data source configuration module includes supporting the following multiple data sources:

[0029] Relational databases: MySQL, SQL Server, Oracle, PostgreSQL;

[0030] Big data: Hive, HBase, ClickHouse, HDFS Hadoop, StarRocks;

[0031] NoSQL: Elasticsearch, MongoDB;

[0032] Message queue: Kafka.

[0033] Preferably, the data development module performs job development processing in the following manner:

[0034] Based on DAG technology, it can visually design and develop complex data integration services, process batch data in the entire link, support the development of various integration operations, and provide a variety of input, output, and conversion components for component expansion.

[0035] Input components: including relational database input, Hive input, HBase input, text file input, CSV file input, Excel input, XML input, JSON input, and record generation;

[0036] Output components: including relational database output, Hive output, Hbase output, text file output, Excel output, XML output, and JSON output;

[0037] Conversion components: including field selection, adding constants, Switch / Case, string replacement, adding sequences, string truncation, removing duplicate records, sorting records, sorting and merging, value mapping, calculator, splitting fields, splitting columns into multiple rows, running SSH commands, and data desensitization;

[0038] Other components: including Rest Client, merged records, and data connections.

[0039] Preferably, the data development module performs workflow development processing in the following manner:

[0040] Supports workflow development by dragging task nodes and establishing associations between task nodes to form a visual DAG;

[0041] The program development supports SQL programs, Shell programs, Spark programs, and Flink programs;

[0042] The data development module also supports configuration of timing strategies, timing mechanisms, failure strategies, task priorities, worker groups, named parameters, local parameters, and timeout alarms; and

[0043] It also supports online trial operation and debugging, running from the source end to any operator node and outputting real-time calculation result logs.

[0044] Preferably, the job scheduling module is a distributed and easily extensible job flow task scheduling module, which assembles tasks in a DAG streaming manner, can timely monitor the execution status of tasks, and supports operations such as retry, failure recovery of designated nodes, suspension, resumption, and termination of tasks.

[0045] Preferably, the operation and maintenance monitoring module includes the following processing:

[0046] Task instance: used to define an instance of a task node in a job flow and identify the specific task execution status;

[0047] Task monitoring: used to run, pause, stop, rerun, and delete workflow instances, as well as view workflow details, task running history, upstream and downstream relationships between tasks, and task execution instances; and

[0048] Job scheduling and execution: supports scheduled scheduling and manual scheduling based on Cron expressions, including: starting a job flow, starting execution from the current node, resuming a fault-tolerant workflow, resuming a paused job flow, starting execution from a failed node, making up numbers, timing, rerunning, pausing, stopping, and resuming waiting threads.

[0049] The one-stop visual lake warehouse development system based on the whole process of the present invention is adopted. Through the online IDE tool provided, offline batch processing jobs and real-time stream processing jobs can be developed and completed through drag-and-drop, template wizard, program and other methods. It is completely based on visualization and wizard-style development methods, and provides full synchronization, incremental synchronization, whole database migration and other solutions, which greatly improves development efficiency and lowers the development threshold. At the same time, lake warehouse planning and stratification, task development, scheduling configuration, testing, task operation and maintenance and other tasks can be completed, and the whole link covers all aspects of lake warehouse data development. Provide multiple sets of processing engines and deployment methods, without switching between multiple tools, one-stop solution to data development problems, can greatly improve data development efficiency. Through personalized processes, data is connected from various technical activities of demand analysis, design, development, testing, deployment, and operation and maintenance, realizing the mutual coordination and unified management of data activities in multiple environments of development, testing and production; on the other hand, facing the multiple perspectives of business, development and operation, and combining the data model and data asset management in the product system, a closed-loop interactive system is formed to identify data assets from the business and then feed back the business from the data assets. This will truly combine Hucang data development with the enterprise organizational structure, data assets, and business scenarios, discover data value from business needs, empower business development with data, and transform data elements into enterprise productivity, which has certain application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a schematic diagram of the overall structure of the one-stop visual lake warehouse development system based on the entire process of the present invention.

[0051] Figure 2 It is a structural diagram of the data source configuration module of the present invention.

[0052] Figure 3 It is a structural schematic diagram of the job scheduling module of the present invention. DETAILED DESCRIPTION

[0053] In order to more clearly describe the technical content of the present invention, further description is given below in conjunction with specific embodiments.

[0054] Before describing in detail embodiments according to the present invention, it should be noted that, hereinafter, the terms "comprises", "includes" or any other variations are intended to cover non-exclusive inclusion, whereby a process, method, article or apparatus comprising a series of elements includes not only these elements, but also other elements not explicitly listed or inherent to such process, method, article or apparatus.

[0055] This technical solution aims to complete offline batch processing jobs and real-time stream processing jobs through online IDE tools, visual drag and drop, and wizard-based development. At the same time, it can complete lake warehouse planning and layering, data modeling, query, SQL development, scheduling configuration, job testing, task operation and maintenance, etc., and the whole link covers all aspects of lake warehouse data development. This system mainly includes five parts: lake warehouse planning, data source configuration, scheduling engine configuration, data development, and operation and maintenance monitoring.

[0056] See also Figure 1 As shown, the one-stop visual lake warehouse development system based on the whole process includes:

[0057] The lake warehouse planning processing module uses the method of defining the lake warehouse data layer to perform lake warehouse planning processing;

[0058] A data source configuration module is connected to the lake warehouse planning processing module and is used to access heterogeneous data sources, read raw data and write it into the lake warehouse;

[0059] The data development module is connected to the data source configuration module and performs job development, program development, job flow development and operation strategy configuration processing by providing a visual drag-and-drop method;

[0060] A job scheduling module, connected to the data development module, for assembling and executing corresponding tasks in a DAG streaming manner; and

[0061] The operation and maintenance monitoring module is connected to the job scheduling module and is used to perform multi-faceted graphical operation and maintenance monitoring processing on the task instance according to the execution status of the task in the current system.

[0062] As a preferred embodiment of the present invention, the lake warehouse planning processing module defines the following data layers in a top-down hierarchical manner:

[0063] The data reference layer is the data preparation area of ​​the data warehouse, which is used to store data that has been extracted, cleaned, and converted;

[0064] The detailed data layer is used to build the most fine-grained detailed fact table based on each specific business feature and store wide table data;

[0065] The summary data layer is used to build summary indicator tables of common granularity based on the indicator requirements of upper-layer applications and products, provide common indicators for the upper layer, and establish summary wide tables and detailed fact tables; and

[0066] The application data layer generates statistical indicator data for storing personalized data products based on upper-layer data processing.

[0067] Lake warehouse planning:

[0068] Lake warehouse planning is a step in the top-level design of the data system. Before starting lake warehouse data development, you need to complete the planning of the lake warehouse, mainly the definition of the data layer. The data layer adopts a top-down layering, and the default built-in data layer of the platform is divided into:

[0069] Data Reference Layer (ODS, Operational Data Store): Stores raw data in the data warehouse system with almost no processing. The structure is basically consistent with the source system and is the data preparation area of ​​the data warehouse. The main responsibility of this layer is to synchronize basic data.

[0070] Detailed data layer (DWD, Data Warehouse Detail): Based on the characteristics of each specific business, the most fine-grained detailed fact table is constructed. In combination with the data usage characteristics of the enterprise, some important attribute fields of the detailed fact table can be appropriately redundant, that is, wide table processing.

[0071] Summary data layer (DWS, Data Warehouse Summary): Based on the indicator requirements of the upper-layer applications and products, a summary indicator table of public granularity is constructed. The model is physicalized by means of wide table, and statistical indicators with naming standards and consistent caliber are constructed to provide public indicators for the upper layer, and to establish summary wide tables and detailed fact tables.

[0072] Application Data Store (ADS): stores personalized statistical indicator data of data products, which is generated based on upper-layer data processing.

[0073] In actual use, the data layer information configured in the platform can be added, deleted, modified, checked, moved up, and moved down as needed.

[0074] API interface:

[0075] Add data layer: http: / / 127.0.0.1:11110 / api / public-resource / datalevel / create

[0076] {

[0077] "code": "string",

[0078] "createTime": "2024-01-29T09:03:37.917Z",

[0079] "createUser": "string",

[0080] "id": "string",

[0081] "name": "string",

[0082] "orders": 0,

[0083] "remark": "string",

[0084] "tenantId": "string",

[0085] "updateTime": "2024-01-29T09:03:37.917Z",

[0086] "updateUser": "string"

[0087] }

[0088] Modify the data layer: http: / / 127.0.0.1:11110 / api / public-resource / datalevel

[0089] {

[0090] "code": "string",

[0091] "createTime": "2024-01-29T09:05:04.498Z",

[0092] "createUser": "string",

[0093] "id": "string",

[0094] "name": "string",

[0095] "orders": 0,

[0096] "remark": "string",

[0097] "tenantId":"string",

[0098] "updateTime":"2024-01-29T09:05:04.498Z",

[0099] "updateUser":"string"

[0100] }

[0101] Delete the data level: http: / / 127.0.0.1:11110 / api / public-resource / datalevel [

[0103] "string" ]

[0105] Query data level: http: / / 127.0.0.1:11110 / api / public-resource / datalevel [

[0107] {

[0108] "code":"string",

[0109] "createTime":"2024-01-29T09:07:18.573Z",

[0110] "createUser":"string",

[0111] "id":"string",

[0112] "name":"string",

[0113] "orders":0,

[0114] "remark":"string",

[0115] "tenantId":"string",

[0116] "updateTime":"2024-01-29T09:07:18.573Z",

[0117] "updateUser":"string"

[0118] } ]

[0120] See also Figure 2 As shown, the data source configuration

[0121] Before configuring a data integration synchronization task, you need to configure the source and destination database information to be synchronized so that when configuring the synchronization task, you can select the data source name to determine the database that the synchronization task reads and writes. Various data sources are supported, including structured, semi-structured, unstructured, binary, and other data sources.

[0122] As a preferred embodiment of the present invention, the data source configuration module includes supporting the following multiple data sources:

[0123] Relational databases: MySQL, SQL Server, Oracle, PostgreSQL;

[0124] Big data: Hive, HBase, ClickHouse, HDFS Hadoop, StarRocks;

[0125] NoSQL: Elasticsearch, MongoDB;

[0126] Message queue: Kafka.

[0127] In actual application, the API interface:

[0128] Add a new data source: http: / / 127.0.0.1:11110 / api / public-resource / datasources / create

[0129] {

[0130] "code":"string",

[0131] "createTime":"2024-01-29T09:07:50.033Z",

[0132] "createUser":"string",

[0133] "id":"string",

[0134] "levels":"string",

[0135] "name":"string",

[0136] "owningOrgId":"string",

[0137] "owningOrgName":"string",

[0138] "owningSystem": "string",

[0139] "remark": "string",

[0140] "status": "string",

[0141] "tenantId": "string",

[0142] "types": "string",

[0143] "updateTime": "2024-01-29T09:07:50.033Z",

[0144] "updateUser": "string",

[0145] "useTo": "string"

[0146] }

[0147] Modify data source: http: / / 127.0.0.1:11110 / api / public-resource / datasources / update{

[0148] "code": "string",

[0149] "createTime": "2024-01-29T09:09:10.390Z",

[0150] "createUser": "string",

[0151] "id": "string",

[0152] "levels": "string",

[0153] "name": "string",

[0154] "owningOrgId": "string",

[0155] "owningOrgName": "string",

[0156] "owningSystem": "string",

[0157] "remark": "string",

[0158] "status": "string",

[0159] "tenantId":"string",

[0160] "types":"string",

[0161] "updateTime":"2024-01-29T09:09:10.390Z",

[0162] "updateUser":"string",

[0163] "useTo":"string"

[0164] }

[0165] Delete data source: http: / / 127.0.0.1:11110 / api / public-resource / datasources / delete [

[0167] "string" ]

[0169] Data source connection test: http: / / 127.0.0.1:11110 / api / public-resource / datasources / ping.

[0170] As a preferred embodiment of the present invention, the data development module performs job development processing in the following manner:

[0171] Based on DAG technology, it can visually design and develop complex data integration services, process batch data in the entire link, support the development of various integration operations, and provide a variety of input, output, and conversion components for component expansion.

[0172] Input components: including relational database input, Hive input, HBase input, text file input, CSV file input, Excel input, XML input, JSON input, and record generation;

[0173] Output components: including relational database output, Hive output, Hbase output, text file output, Excel output, XML output, and JSON output;

[0174] Conversion components: including field selection, adding constants, Switch / Case, string replacement, adding sequences, string truncation, removing duplicate records, sorting records, sorting and merging, value mapping, calculator, splitting fields, splitting columns into multiple rows, running SSH commands, and data desensitization;

[0175] Other components: including Rest Client, merged records, and data connections.

[0176] As a preferred embodiment of the present invention, the data development module performs workflow development processing in the following manner:

[0177] Supports workflow development by dragging task nodes and establishing associations between task nodes to form a visual DAG;

[0178] The program development supports SQL programs, Shell programs, Spark programs, and Flink programs;

[0179] The data development module also supports configuration of timing strategies, timing mechanisms, failure strategies, task priorities, worker groups, named parameters, local parameters, and timeout alarms; and

[0180] It also supports online trial operation and debugging, running from the source end to any operator node and outputting real-time calculation result logs.

[0181] Job Development

[0182] Based on DAG technology, it can visually design and develop complex data integration services and process batch data in the entire link. It supports the development of various integration jobs. It provides a variety of input, output, and conversion components, which can be easily expanded.

[0183] Input components: relational database input, Hive input, HBase input, text file input, CSV file input, Excel input, XML input, JSON input, generate records.

[0184] Output components: relational database output, Hive output, Hbase output, text file output, Excel output, XML output, and JSON output.

[0185] Conversion components: field selection, adding constants, Switch / Case, string replacement, adding sequences, string truncation, removing duplicate records, sorting records, sorting and merging, value mapping, calculator, splitting fields, splitting columns into multiple rows, running SSH commands, and data desensitization.

[0186] Other components: Rest Client, merge records, data connection.

[0187] API interface:

[0188] Get database information: http: / / 127.0.0.1:11110 / api / dws / prjdev / di / getDataBaseInfo

[0189] {

[0190] "dataBaseType":"string",

[0191] "datasourceId":"string",

[0192] "envType":"string",

[0193] "limit":0,

[0194] "paramMap":{

[0195] "additionalProp1":"string",

[0196] "additionalProp2":"string",

[0197] "additionalProp3":"string"

[0198] },

[0199] "pluginType":"string",

[0200] "schema":"string",

[0201] "sql":"string",

[0202] "stepName":"string",

[0203] "tableName":"string",

[0204] "targetDataBaseType":"string",

[0205] "targetDatasourceId":"string",

[0206] "targetEnvType":"string",

[0207] "targetFieldList":[

[0208] "string"

[0209] ],

[0210] "targetSchema":"string",

[0211] "targetTableName": "string"

[0212] }

[0213] Get database name: http: / / 127.0.0.1:11110 / api / dws / prjdev / di / getDBName

[0214] {

[0215] "dataBaseType": "string",

[0216]

[0217] Get table name: http: / / 127.0.0.1:11110 / api / dws / prjdev / di / getTables

[0218]

[0219] "schema": "string",

[0220] "sql": "string",

[0221] "stepName": "string",

[0222] "tableName": "string",

[0223] "targetDataBaseType": "string",

[0224] "targetDatasourceId": "string",

[0225] "targetEnvType": "string",

[0226] "targetFieldList":

[0227] "string"

[0228] ,

[0229] "targetSchema": "string",

[0230] "targetTableName": "string"

[0231] }

[0232] Get table field names: http: / / 127.0.0.1:11110 / api / dws / prjdev / di / getFieldInfo{

[0233] "dataBaseType": "string",

[0234] "datasourceId": "string",

[0235] "envType": "string",

[0236] "limit": 0,

[0237] "paramMap": {

[0238] "additionalProp1": "string",

[0239] "additionalProp2": "string",

[0240] "additionalProp3": "string"

[0241] },

[0242] "pluginType": "string",

[0243] "schema": "string",

[0244] "sql": "string",

[0245] "stepName": "string",

[0246] "tableName": "string",

[0247] "targetDataBaseType": "string",

[0248] "targetDatasourceId": "string",

[0249] "targetEnvType": "string",

[0250] "targetFieldList":

[0251] "string"

[0252] ,

[0253] "targetSchema":"string",

[0254] "targetTableName":"string"

[0255] }

[0256] Program Development

[0257] Supports multiple programs, including SQL programs, Shell programs, Spark programs, and Flink programs.

[0258] Workflow Development

[0259] Supports visual DAG formed by dragging task nodes and establishing associations between task nodes.

[0260] API interface: http: / / 127.0.0.1:11110 / api / dws / jobsch / processDefinition / list

[0261] {

[0262] "code":0,

[0263] "data":{},

[0264] "msg":"string"

[0265] }

[0266] Run policy configuration

[0267] Supports configuration of timing strategies, timing mechanisms, failure strategies, task priorities, Worker groups, named parameters, local parameters, and timeout alarms.

[0268] API interface: http: / / 127.0.0.1:11110 / api / dws / jobsch / alertGroup / register

[0269] Online operation / debugging

[0270] Supports online trial operation and debugging, running from the source to any operator node and outputting real-time calculation result logs. With debugging, users can easily check the debugging results, fine-grainedly check the result sets of each operator and accurately locate operator anomalies.

[0271] API interface: http: / / 127.0.0.1:11110 / api / dws / jobsch / executors / execute

[0272] See also Figure 3As shown, as a preferred embodiment of the present invention, the job scheduling module is a distributed and easily extensible job flow task scheduling module, which assembles tasks in a DAG streaming manner, can monitor the execution status of tasks in a timely manner, and supports retry, specifying node recovery failure, pausing, resuming, and terminating tasks.

[0273] The job scheduling module is a distributed and easily scalable job flow task scheduling system that supports tasks, job flows, and full-process data processing. It assembles tasks in a DAG (Directed Acyclic Graph, DAG) streaming manner, can monitor the execution status of tasks in a timely manner, and supports operations such as retry, failure to restore a specified node, pause, resume, and terminate tasks. The decentralized design ensures stability and provides overload fault tolerance.

[0274] The Master adopts a distributed and centerless design concept, and is mainly responsible for DAG task segmentation, task submission monitoring, and monitoring the health status of other Masters and Workers. When the Master service starts, it registers a temporary node with Zookeeper and performs fault tolerance by monitoring the changes of Zookeeper temporary nodes. The Master provides monitoring services based on Netty.

[0275] API interface:

[0276] Register / update scheduling engine: http: / / 127.0.0.1:11110 / api / public-resource / engine / register

[0277] {

[0278] "code":"string",

[0279] "connectionParams":"string",

[0280] "createTime":"2024-01-29T09:11:43.398Z",

[0281] "createUser":"string",

[0282] "defaultQueue":"string",

[0283] "defaultTenant":"string",

[0284] "engineUsage":"string",

[0285] "envType":"string",

[0286] "hdfsBasePath": "string",

[0287] "hdfsCode": "string",

[0288] "hdfsUserName": "string",

[0289] "id": "string",

[0290] "name": "string",

[0291] "owningOrgId": "string",

[0292] "owningOrgName": "string",

[0293] "owningSystem": "string",

[0294] "registryType": "string",

[0295] "remark": "string",

[0296] "serialNumber": "string",

[0297] "status": "string",

[0298] "tenantId": "string",

[0299] "updateTime": "2024-01-29T09:11:43.398Z",

[0300] "updateUser": "string"

[0301] }

[0302] Delete Scheduling Engine: http: / / 127.0.0.1:11110 / api / public-resource / engine / unregister

[0303] Master and Worker Health Check:

[0304] curl --request GET 'http: / / 127.0.0.1:11110 / api / public-resource / engine / monitor'

[0305] {

[0306] "code":"string",

[0307] "connectionParams":"string",

[0308] "createTime":"2024-01-29T08:31:01.594Z",

[0309] "createUser":"string",

[0310] "defaultQueue":"string",

[0311] "defaultTenant":"string",

[0312] "engineUsage":"string",

[0313] "envType":"string",

[0314] "hdfsBasePath":"string",

[0315] "hdfsCode":"string",

[0316] "hdfsUserName":"string",

[0317] "id":"string",

[0318] "name":"string",

[0319] "owningOrgId":"string",

[0320] "owningOrgName":"string",

[0321] "owningSystem":"string",

[0322] "registryType":"string",

[0323] "remark":"string",

[0324] "serialNumber":"string",

[0325] "status":"string",

[0326] "tenantId":"string",

[0327] "updateTime":"2024-01-29T08:31:01.594Z",

[0328] "updateUser":"string"

[0329] }

[0330] View the scheduling engine details: http: / / 127.0.0.1:11110 / api / public-resource / engine / engineDetail

[0331] Worker also adopts the distributed and decentralized design concept. Worker is mainly responsible for task execution and log service provision. When the Worker service starts, it registers a temporary node with Zookeeper and maintains the heartbeat. Worker provides monitoring services based on Netty. TaskExecutor is responsible for the task execution process and performs actual task processing according to different task types.

[0332] Both the Master and Worker in the system use ZooKeeper for cluster management and fault tolerance.

[0333] Alert provides alarm services and implements a variety of alarm methods through alarm plug-ins.

[0334] When executing tasks, the differences between the underlying processing engines are shielded, and the stream-batch computing engine can be switched as needed. The integrated stream-batch engine combines batch processing and real-time stream processing to give full play to the advantages of both. It can quickly respond to real-time data changes and process large-scale offline data, providing enterprises with powerful data processing capabilities.

[0335] The real-time stream processing component is responsible for receiving real-time data streams from data sources and performing real-time calculations and processing. It has the characteristics of low latency and high throughput, and can quickly respond to changes in real-time data.

[0336] The batch processing component is responsible for processing offline batch data, which can provide more powerful computing power and more complex analysis models, and is suitable for processing large-scale historical data.

[0337] As a preferred embodiment of the present invention, the operation and maintenance monitoring module includes the following processing:

[0338] Task instance: used to define an instance of a task node in a job flow and identify the specific task execution status;

[0339] Task monitoring: used to run, pause, stop, rerun, and delete workflow instances, as well as view workflow details, task running history, upstream and downstream relationships between tasks, and task execution instances; and

[0340] Job scheduling and execution: supports scheduled scheduling and manual scheduling based on Cron expressions, including: starting a job flow, starting execution from the current node, resuming a fault-tolerant workflow, resuming a paused job flow, starting execution from a failed node, making up numbers, timing, rerunning, pausing, stopping, and resuming waiting threads.

[0341] Task Examples

[0342] A task instance is an instantiation of a task node in a job flow definition, identifying a specific task execution status.

[0343] API interface: http: / / 127.0.0.1:11110 / api / dws / jobsch / taskInstance / list

[0344] Task monitoring

[0345] You can run, pause, stop, rerun, delete, and other operations on the job flow instance. Other functions include viewing workflow details, task running history, upstream and downstream relationships between tasks, task execution instances, and other functions.

[0346] API interface: http: / / 127.0.0.1:11110 / api / dws / jobsch / alert / detail

[0347] Job scheduling execution

[0348] Supports scheduled scheduling and manual scheduling based on Cron expressions. Supports: starting a workflow, starting execution from the current node, resuming a fault-tolerant workflow, resuming a paused workflow, starting execution from a failed node, supplementing numbers, timing, rerunning, pausing, stopping, and resuming waiting threads.

[0349] Cron expression is a string format used to describe the execution time of a scheduled task. It is widely used in operating systems such as Unix, Linux, and Mac OS X. Cron expression can define the triggering conditions and time points of a scheduled task through a series of time fields. These time fields include seconds (s), minutes (m), hours (h), days (d), months (M), and weeks (w). In addition, there are asterisks (*), commas (,), minus signs (-), forward slashes ( / ), and question marks (?), which are used to represent different matching rules and time intervals.

[0350] API interface: http: / / 127.0.0.1:11110 / api / dws / jobsch / projects / schedules

[0351] The following uses the development of batch jobs as an example to illustrate the Hucang data development process. This example implements the synchronization of database tables, that is, synchronizing the source table data to the target table:

[0352] 1. Configure the lake warehouse data layer

[0353] Create four data layers in the data layer management interface: ODS, DWD, DWS, and ADS. When creating data layers, standardize data layers, reduce repeated calculations, shield original data anomalies, and facilitate data lineage tracking.

[0354] The data stored in the ODS layer is extracted, cleaned, and converted, with the aim of simplifying subsequent data processing.

[0355] The DWS layer stores wide table data.

[0356] The DWS layer stores summary indicator table data.

[0357] ADS layer personalized statistical indicator table data.

[0358] 2. Configure data source

[0359] Before developing a data synchronization task, you need to configure the source and destination database information to be synchronized to determine the databases that the data synchronization task reads and writes.

[0360] Create a data source. The specific configuration properties are as follows:

[0361] Configuration parameters Parameter Value Data source encoding MySQL-test Data Source Name Data Source - Test Data layer to which the data source belongs ODS Data source type MySQL Data database URL jdbc:mysql: / / 127.0.0.1:3306 / test username root password 000000 Schema test

[0362] The generated configuration is as follows:

[0363]

[0364]

[0365] 3. Configure the scheduling engine

[0366] Create a scheduling engine. The specific configuration properties are as follows:

[0367] Configuration parameters Parameter Value Scheduling engine code Jobctrl-local Scheduling engine name Scheduling - Local Scheduling engine address localhost:2188 Yarn Queue default

[0368] 4. Prepare sample data

[0369] Create tables user and user_miggration as the source table and the target table respectively, and insert some data into the source table. CREATE TABLE user(

[0370] ID int NOT NULL,

[0371] NAME varchar(10) DEFAULT NULL,

[0372] AGE decimal(3,0) DEFAULT NULL,

[0373] PRIMARY KEY(ID)

[0374] ) ENGINE=InnoDB DEFAULT CHARSET=utf8;

[0375] CREATE TABLE user_miggration(

[0376] ID int NOT NULL,

[0377] NAME varchar(10) DEFAULT NULL,

[0378] AGE decimal(3,0) DEFAULT NULL,

[0379] PRIMARY KEY(ID)

[0380] ) ENGINE=InnoDB DEFAULT CHARSET=utf8;

[0381] INSERT INTO user(ID,NAME,AGE) VALUES(1, “Zhang San”, 20); INSERT INTO user(ID,NAME,AGE) VALUES(2, “Li Si”, 30); INSERT INTO user(ID,NAME,AGE) VALUES(3, “Wang Wu”, 40); 5. Develop a database synchronization job

[0382] Create a data synchronization job with the following configuration items:

[0383] Configuration parameters Parameter Value Source Data Source Name Data Source - Test Source database name test Source table name user Target data source name Data Source - Test Target database name test Target table name user_miggration

[0384] The final generated model is as follows:

[0385]

[0386]

[0387] 6. Configure operation parameters

[0388] The system supports three types of parameters: global parameters, local parameters, and named parameters.

[0389] Global parameters: Parameters that are valid for all task nodes in the entire job flow, configured on the job flow definition page.

[0390] Local parameters: Parameters configured on the batch job and general template job definition page. By default, the scope is limited to the integration job. If parameter transfer is configured, the parameter can be applied to downstream integration jobs.

[0391] Named parameters: parameters configured on the batch job and general template job definition page. The parameters here only define the function and declare the key and default value. In actual use, global parameters need to be used for parameter transmission.

[0392] This example configures "named parameters", target_table_name = user

[0393] 7. Configure the timing strategy

[0394] Create a cron timing policy and configure the parameters as follows:

[0395] Configuration parameters Parameter Value Start and end time 2024-02-03 to 2025-02-02 Timing mechanism 0 0***？* Failure Strategy continue Alarm Group Mail Notification strategy None Process Priority Highest Worker Grouping Default Whether to enable timing yes

[0396] When the cron timing rules are met, the scheduling component will automatically start the task.

[0397] 8. Save and run

[0398] After saving, running the database synchronization job results in the data in the source table user being synchronized to the target table user_miggration, as follows:

[0399] [INFO]2024-01-29 11:35:12.054+0800 --> Jan 29, 2024 11:35:11 AM org.apache.cxf.endpoint.ServerImpl initDestination

[0400] Information:Setting the server's publish address to be / lineage

[0401] Jan 29, 2024 11:35:11 AM org.apache.cxf.endpoint.ServerImplinitDestination

[0402] Information:Setting the server's publish address to be / i18n

[0403] [INFO]2024-01-29 11:35:15.063+0800-->2024 / 01 / 29 11:35:14-Pan-Started running.

[0404] [INFO]2024-01-29 11:35:16.064+0800-->2024 / 01 / 29 11:35:15-Carte-Installing timer to purge stale objects after 1440minutes.

[0405] 2024 / 01 / 29 11:35:15-Database Sync_1706499287077-Patch started for conversion [Database Sync_1706499287077]

[0406] 2024 / 01 / 29 11:35:16 - Table output.0 - Connected to database [test] (commit = 1000)

[0407] [INFO]2024-01-29 11:35:17.065+0800-->2024 / 01 / 29 11:35:16-Table input.0-Finished reading query, closing connection.

[0408] 2024 / 01 / 29 11:35:16-Filter records.0-Completed processing (I=0, O=0, R=16, W=1, U=0, E=0)

[0409] 2024 / 01 / 29 11:35:16 - Table input.0 - Completed processing (I=0, O=0, R=16, W=16, U=0, E=0)

[0410] 2024 / 01 / 29 11:35:16-TableMappingDDLStep.0-Finished reading query, closing connection.

[0411] 2024 / 01 / 29 11:35:16-TableMappingDDLStep.0-Processing completed (I=0, O=0, R=12, W=11, U=0, E=0)

[0412] [INFO]2024-01-29 11:35:18.073+0800-->2024 / 01 / 29 11:35:17-Table output.0-Processing completed (I=11, O=0, R=11, W=11, U=0, E=0)

[0413] 2024 / 01 / 29 11:35:17 - Pan - Done!

[0414] 2024 / 01 / 29 11:35:17-Pan-Start=2024 / 01 / 29 11:35:14.885, Stop=2024 / 01 / 29 11:35:17.448

[0415] 2024 / 01 / 29 11:35:17-Pan-Processing completed in 2 seconds.

[0416] 2024 / 01 / 29 11:35:17-Database synchronization_1706499287077-

[0417] 2024 / 01 / 29 11:35:17-Database synchronization_1706499287077-Process TableMappingDDLStep.0 successfully completed, processing 12 rows. (6 rows / second)

[0418] 2024 / 01 / 29 11:35:17-Database synchronization_1706499287077-Process table output.0 Successfully completed, 11 rows processed. (5 rows / second)

[0419] 2024 / 01 / 29 11:35:17-Database Sync_1706499287077-Process table input.0 successfully completed, 16 rows processed. (8 rows / second)

[0420] 2024 / 01 / 29 11:35:17-Database Sync_1706499287077-Process filtering records.0 successfully completed, 16 rows processed. (8 rows / second).

[0421] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0422] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device.

[0423] A person of ordinary skill in the art may understand that all or part of the steps of the method for implementing the above-mentioned embodiment may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0424] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0425] In the description of this specification, the description with reference to the terms "an embodiment", "some embodiments", "example", "specific example", or "embodiment" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0426] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

[0427] The one-stop visual lake warehouse development system based on the whole process of the present invention is adopted. Through the online IDE tool provided, offline batch processing jobs and real-time stream processing jobs can be developed and completed through drag-and-drop, template wizard, program and other methods. It is completely based on visualization and wizard-style development methods, and provides full synchronization, incremental synchronization, whole database migration and other solutions, which greatly improves development efficiency and lowers the development threshold. At the same time, lake warehouse planning and stratification, task development, scheduling configuration, testing, task operation and maintenance and other tasks can be completed, and the whole link covers all aspects of lake warehouse data development. Provide multiple sets of processing engines and deployment methods, without switching between multiple tools, one-stop solution to data development problems, can greatly improve data development efficiency. Through personalized processes, data is connected from various technical activities of demand analysis, design, development, testing, deployment, and operation and maintenance, realizing the mutual coordination and unified management of data activities in multiple environments of development, testing and production; on the other hand, facing the multiple perspectives of business, development and operation, and combining the data model and data asset management in the product system, a closed-loop interactive system is formed to identify data assets from the business and then feed back the business from the data assets. This will truly combine Hucang data development with the enterprise organizational structure, data assets, and business scenarios, discover data value from business needs, empower business development with data, and transform data elements into enterprise productivity, which has certain application value.

[0428] In this specification, the present invention has been described with reference to specific embodiments thereof. However, it is apparent that various modifications and variations may be made without departing from the spirit and scope of the present invention. Therefore, the specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A one-stop visual lake warehouse development system based on the whole process, characterized by: The system comprises: The lake warehouse planning processing module uses the method of defining the lake warehouse data layer to perform lake warehouse planning processing; A data source configuration module is connected to the lake warehouse planning and processing module and is used to access heterogeneous data sources, read raw data and write it into the lake warehouse; The data development module is connected to the data source configuration module and performs job development, program development, job flow development and operation strategy configuration processing by providing a visual drag-and-drop method; A job scheduling module, connected to the data development module, for assembling and executing corresponding tasks in a DAG streaming manner; and The operation and maintenance monitoring module is connected to the job scheduling module and is used to perform multi-faceted graphical operation and maintenance monitoring processing on the task instance according to the execution status of the task in the current system.

2. The one-stop visual lake warehouse development system based on the whole process according to claim 1 is characterized in that: The lake warehouse planning processing module defines the following data layers in a top-down hierarchical manner: The data reference layer is the data preparation area of ​​the data warehouse, which is used to store data that has been extracted, cleaned, and converted; The detailed data layer is used to build the most fine-grained detailed fact table based on each specific business feature and store wide table data; The summary data layer is used to build summary indicator tables of public granularity based on the indicator requirements of upper-layer applications and products, provide public indicators for the upper layer, and establish summary wide tables and detailed fact tables; as well as The application data layer generates statistical indicator data for storing personalized data products based on upper-layer data processing.

3. The one-stop visual lake warehouse development system based on the whole process according to claim 1 is characterized in that: The data source configuration module includes support for the following multiple data sources: Relational databases: MySQL, SQL Server, Oracle, PostgreSQL; Big data: Hive, HBase, ClickHouse, HDFS Hadoop, StarRocks; NoSQL: Elasticsearch, MongoDB; Message queue: Kafka.

4. The one-stop visual lake warehouse development system based on the whole process according to claim 1 is characterized in that: The data development module performs job development processing in the following manner: Based on DAG technology, it can visually design and develop complex data integration services, process batch data in the entire link, support the development of various integration operations, and provide a variety of input, output, and conversion components for component expansion. Input components: including relational database input, Hive input, HBase input, text file input, CSV file input, Excel input, XML input, JSON input, and record generation; Output components: including relational database output, Hive output, Hbase output, text file output, Excel output, XML output, and JSON output; Conversion components: including field selection, adding constants, Switch / Case, string replacement, adding sequences, string truncation, removing duplicate records, sorting records, sorting and merging, value mapping, calculator, splitting fields, splitting columns into multiple rows, running SSH commands, and data desensitization; Other components: including Rest Client, merged records, and data connections.

5. The one-stop visual lake warehouse development system based on the whole process according to claim 1 is characterized in that: The data development module performs workflow development processing in the following manner: Supports workflow development by dragging task nodes and establishing associations between task nodes to form a visual DAG; The program development supports SQL programs, Shell programs, Spark programs, and Flink programs; The data development module also supports configuration of timing strategies, timing mechanisms, failure strategies, task priorities, worker groups, named parameters, local parameters, and timeout alarms; and It also supports online trial operation and debugging, running from the source end to any operator node and outputting real-time calculation result logs.

6. The one-stop visual lake warehouse development system based on the whole process according to claim 1 is characterized in that: The job scheduling module is a distributed and easily extensible job flow task scheduling module, which assembles tasks in a DAG streaming manner, can monitor the execution status of tasks in a timely manner, and supports operations such as retry, failure recovery of designated nodes, suspension, resumption, and termination of tasks.

7. The one-stop visual lake warehouse development system based on the whole process according to claim 1 is characterized in that: The operation and maintenance monitoring module includes the following processing: Task instance: used to define an instance of a task node in a job flow and identify the specific task execution status; Task monitoring: used to run, pause, stop, rerun, and delete job flow instances, as well as view workflow details, task running history, upstream and downstream relationships between tasks, and task execution instances; as well as Job scheduling and execution: supports scheduled scheduling and manual scheduling based on Cron expressions, including: starting a job flow, starting execution from the current node, resuming a fault-tolerant workflow, resuming a paused job flow, starting execution from a failed node, making up numbers, timing, rerunning, pausing, stopping, and resuming waiting threads.

Images