Multi-source heterogeneous real-time data processing method, system, device and medium

Abstract

The application relates to a multi-source heterogeneous real-time data processing method, system, device and medium. The method realizes integration of data processing in multiple links through real-time data table processing, finally completes the whole process of real-time data processing, solves the unified management and control problem of multi-source heterogeneous real-time data, solves the problem of disordered data development due to numerous processing procedures, and also reduces artificial cost and simplifies the system development process. Compared with the traditional technology, the above scheme formulates a unified data processing standard model, simplifies the data processing logic, and reduces the system architecture complexity; the in-memory data processing of the temporary table greatly improves the real-time data processing speed and development efficiency; the module plug-in data development has flexibility and expansibility for different data types and rule processing requirements.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing and relates to a multi-source heterogeneous real-time data processing method, system, device and medium. Background Art

[0002] With the rapid adoption of IoT devices, real-time data processing is becoming increasingly widespread within the IoT landscape. However, IoT data also exhibits significant structural heterogeneity, comes from diverse sources, and is difficult to uniformly manage and control. Currently, most traditional approaches to real-time data processing for multi-source, heterogeneous IoT are custom-developed based on predefined data models and processing logic. These approaches lack flexibility and scalability for new data types and processing rules. They also require manual management of data flows, which can be labor-intensive and resource-intensive and prone to errors. Furthermore, these approaches suffer from numerous drawbacks that severely limit their application in large-scale, real-time data processing scenarios.

[0003] With the advent of the big data era, real-time data processing has become a requirement across all industries. Real-time data processing and analysis based on modular development is a current technical requirement. Different types of real-time tables and customized data processing processes are often created to meet different data processing requirements. Currently, supporting real-time processing of multi-source heterogeneous data is a critical prerequisite for technical implementation. Currently, many researchers have conducted in-depth research on real-time data processing methods for multi-source heterogeneous systems and have achieved considerable research results. These include methods that use a single parsing program to perform data specification parsing and processing on power grid data from different devices, methods that implement real-time data processing based on multi-engine support for plug-in process orchestration, methods that manage and process data based on sensor data items in high-speed data queues, methods that use at least one data parsing plug-in to parse and process real-time server data in message queues, and methods that determine whether to enable manual or automatic mode for real-time data processing tasks. However, these traditional technologies still suffer from complex and inefficient technical issues when processing multi-source heterogeneous real-time data. Summary of the Invention

[0004] In response to the problems existing in the above-mentioned traditional methods, the present invention proposes a multi-source heterogeneous real-time data processing method, a multi-source heterogeneous real-time data processing system, a computer device and a computer-readable storage medium, which can significantly reduce the complexity of multi-source heterogeneous real-time data processing and improve efficiency.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0006] In one aspect, a method for processing multi-source heterogeneous real-time data is provided, comprising the steps of:

[0007] In the data source management page of the heterogeneous data source, add each real-time data source, configure the data source connection information, and complete the connectivity test;

[0008] Add data processing rules in the rule configuration management page and save them to the rule configuration table; data processing rules include protocol parsing rules, SQL processing rules, data filtering rules, dimension association rules, primary key deduplication rules, indicator aggregation rules, and data structure rules;

[0009] Create and edit the Flink DAG real-time processing canvas based on the DAG interface and save it to the canvas table. In the DAG real-time processing canvas, dynamically select data processing plug-ins based on the DAG drag-and-drop function. Data processing plug-ins include Flink input plug-ins, Flink transformation plug-ins, and Flink output plug-ins.

[0010] In the DAG real-time processing canvas, the Flink real-time data processing model is used to divide the entire real-time data processing process into modules. These modules include the Flink input table processing module, the Flink conversion table processing module, the Flink output table processing module, and the Flink data storage processing module.

[0011] In the DAG real-time processing canvas, select the Flink input plugin and configure the input table, select the Flink transformation plugin and configure the transformation rules, select the Flink output plugin and configure the output table and storage table.

[0012] During real-time data processing, the data tables in the module are registered as temporary table objects based on Flink's internal temporary table registration model.

[0013] During real-time data processing, the Flink real-time data stream conversion model is used to obtain real-time data streams from the temporary table objects registered by each module plug-in;

[0014] In the real-time data processing process, the data table configuration modeling model is used to perform data modeling on the data table configuration in each module plug-in and define the modeling features;

[0015] In the Flink input table processing module, set up the Flink input temporary table construction process and register the input table based on the temporary table registration model;

[0016] In the Flink conversion table processing module, set up the Flink conversion temporary table construction process and register the conversion table based on the temporary table registration model;

[0017] In the Flink output table processing module, set up the Flink output temporary table construction process and register the output table based on the temporary table registration model;

[0018] In the Flink data storage processing module, set up the Flink storage temporary table construction process and register the storage table based on the temporary table registration model;

[0019] In the Flink data storage and processing module, the Flink real-time data synchronization model is used to execute the real-time data synchronization process of the external storage medium, and finally complete the entire real-time data processing process.

[0020] On the other hand, a multi-source heterogeneous real-time data processing system is also provided, comprising:

[0021] The source configuration module is used to add real-time data sources, configure data source connection information, and complete connectivity testing in the data source management page of heterogeneous data sources;

[0022] The rule configuration module is used to add data processing rules in the rule configuration management page and save them to the rule configuration table; data processing rules include protocol parsing rules, SQL processing rules, data filtering rules, dimension association rules, primary key deduplication rules, indicator aggregation rules, and data structure rules;

[0023] The canvas plugin module is used to create and edit Flink's DAG real-time processing canvas based on the DAG interface and save it to the canvas table. In the DAG real-time processing canvas, data processing plugins can be dynamically selected by dragging and dropping the DAG. Data processing plugins include Flink input plugins, Flink transformation plugins, and Flink output plugins.

[0024] The process partitioning module is used to divide the entire real-time data processing process into modules in the DAG real-time processing canvas using the Flink real-time data processing model. The modules include the Flink input table processing module, the Flink conversion table processing module, the Flink output table processing module, and the Flink data storage processing module.

[0025] The data table configuration module is used to select the Flink input plugin and configure the input table, select the Flink transformation plugin and configure the transformation rules, select the Flink output plugin and configure the output table and storage table in the DAG real-time processing canvas.

[0026] The temporary table registration module is used to register data tables in the module as temporary table objects based on Flink's internal temporary table registration model during real-time data processing.

[0027] The real-time streaming module is used to obtain real-time data streams from temporary table objects registered by each module plug-in during real-time data processing using Flink's real-time data stream conversion model.

[0028] The configuration modeling module is used to perform data modeling on the data table configuration in each module plug-in and define modeling features using the data table configuration modeling model during real-time data processing;

[0029] The input construction module is used to set up the Flink input temporary table construction process in the Flink input table processing module and register the input table based on the temporary table registration model;

[0030] The conversion construction module is used to set up the Flink conversion temporary table construction process in the Flink conversion table processing module and register the conversion table based on the temporary table registration model;

[0031] The output conversion module is used to set up the Flink output temporary table construction process in the Flink output table processing module and register the output table based on the temporary table registration model;

[0032] The storage construction module is used to set up the Flink storage temporary table construction process in the Flink data storage processing module and register the storage table based on the temporary table registration model;

[0033] The data synchronization module is used in the Flink data storage and processing module to perform real-time data synchronization of external storage media using the Flink real-time data synchronization model, ultimately completing the entire real-time data processing process.

[0034] On the other hand, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned multi-source heterogeneous real-time data processing method when executing the computer program.

[0035] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multi-source heterogeneous real-time data processing method described above are implemented.

[0036] One of the above technical solutions has the following advantages and beneficial effects:

[0037] The above-mentioned multi-source heterogeneous real-time data processing method, system, device, and medium reduce manual operations and ease system development difficulty by configuring through a visual interface. A real-time data processing construction model is designed to divide the entire real-time data processing process. An internal temporary table registration model is used to register data tables in each module plug-in as temporary table objects, and a real-time data stream conversion model is used to obtain real-time data streams from registered temporary tables. A data table configuration modeling model is used to model the data table configurations in each module plug-in. A temporary table construction process is designed and registered using a temporary table registration model. A real-time data synchronization model is used to synchronize real-time data with external storage media. Ultimately, through real-time data table processing, data processing is integrated across multiple links, ultimately completing the entire real-time data processing process. This solves the problem of unified management and control of multi-source heterogeneous real-time data and the problem of complex data development due to numerous processing flows. It also reduces labor costs and simplifies the system development process.

[0038] Compared with traditional technologies, the above solution establishes a unified data processing standard model, simplifies data processing logic, and reduces the complexity of system architecture; adopts in-memory data processing of temporary tables, greatly improving real-time data processing speed and development efficiency; is compatible with multiple heterogeneous data types and multiple source systems, solving data silos and consolidating data assets; and uses module plug-in data development with flexibility and scalability for different data types and rule processing requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 1 is a flow chart of a method for processing multi-source heterogeneous real-time data in one embodiment;

[0041] Figure 2 A schematic diagram of a real-time data processing plug-in type in one embodiment;

[0042] Figure 3 A schematic diagram of the configuration of plug-in data table relationships in one embodiment;

[0043] Figure 4 A schematic diagram of a process for constructing a temporary table registration instance in one embodiment;

[0044] Figure 5 A schematic diagram of a process for real-time data synchronization in one embodiment;

[0045] Figure 6 Schematic diagram of a module framework of a multi-source heterogeneous real-time data processing system in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application pertains. The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this application.

[0048] It should be noted that reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of such a phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments.

[0049] Those skilled in the art will appreciate that the embodiments described herein may be combined with other embodiments. The term "and / or" used in the present specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0050] Zero-code and modular data development are crucial for processing multi-source, heterogeneous, real-time data. While many researchers have explored technologies that allow for customized development based on predefined data models and processing logic, these approaches lack flexibility and scalability for different data types and processing rules. This is particularly true for real-time processing, which involves complex workflows and complex development. The lack of a unified standard model means real-time data processing still faces certain limitations.

[0051] At present, for example, the proposed real-time data processing system and method for the Internet of Things adopts a set of parsing programs and has a low degree of modular development; a proposed real-time data processing system and method that supports multiple engines adopts plug-in process orchestration, but the data processing logic is complex; the proposed smart city real-time data processing method, system and storage medium have an overly complex system architecture design; the proposed server real-time data processing method and system based on server clusters have poor heterogeneous compatibility and poor configuration flexibility; a proposed real-time data processing method and device have a high degree of system customization and problems with the degree of automation.

[0052] Therefore, in response to the above-mentioned shortcomings, the purpose of the present invention is to create a set of simple and efficient real-time data processing standard solutions and systems based on heterogeneous data tables, which are compatible with multiple heterogeneous data sources, eliminate data structure differences, and enable agile and efficient data development; reduce manual operations based on visual interface configuration and reduce the difficulty of system development; design a real-time data processing construction model to divide the entire real-time data processing process; adopt an internal temporary table registration model to register the data tables in each module plug-in as temporary table objects, and adopt a real-time data stream conversion model to obtain real-time data streams from the registered temporary tables; adopt a data table configuration modeling model to model the data table configuration in each module plug-in; design a temporary table construction process and adopt a temporary table registration model to register the data tables; and adopt a real-time data synchronization model to realize the real-time data synchronization process of external storage media. Ultimately, through real-time data table processing, data processing is integrated in multiple links, and the entire process of real-time data processing is finally completed, solving the problem of unified management and control of multi-source heterogeneous real-time data, solving the problem of data development being messy due to numerous processing processes, and also reducing labor costs and simplifying the system development process.

[0053] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings.

[0054] See also Figure 1 In one embodiment, a multi-source heterogeneous real-time data processing method is provided, comprising the following processing steps S10 to S22:

[0055] S10, in the data source management page of the heterogeneous data source, add each real-time data source, configure data source connection information and complete the connectivity test.

[0056] As you can understand, on the heterogeneous data source management page, you can pre-add or add the various heterogeneous real-time data sources needed for the current application scenario, configure their connection information, and complete the required connectivity tests. Real-time input data sources can include Kafka message middleware, file systems, FlinkCDC data sources, and custom data sources. Real-time output data sources can include Kafka message middleware, Redis data sources, HBase data sources, Elasticsearch data sources, Mongodb data sources, Hudi data sources, Iceberg data sources, and Paimon data sources.

[0057] S11. Add various data processing rules in the rule configuration management page and save them to the rule configuration table; data processing rules include protocol parsing rules, SQL processing rules, data filtering rules, dimension association rules, primary key deduplication rules, indicator aggregation rules and data structure rules.

[0058] It can be understood that in the rule configuration management page of the heterogeneous data source, multiple data processing rules can be added in advance or online and saved to the rule configuration table. These data processing rules can include protocol parsing rules, SQL processing rules, data filtering rules, dimension association rules, primary key deduplication rules, indicator aggregation rules and data structure rules.

[0059] S12: Create and edit the Flink DAG real-time processing canvas based on the DAG interface and save it to the canvas table. In the DAG real-time processing canvas, dynamically select data processing plug-ins based on the DAG drag-and-drop function. Data processing plug-ins include Flink input plug-ins, Flink transformation plug-ins, and Flink output plug-ins.

[0060] As you can understand, you can create and edit Flink's DAG real-time processing canvas based on the visual DAG interface and save it to the canvas table. In the DAG real-time processing canvas, you can dynamically select data processing plug-ins based on the DAG drag-and-drop. These data processing plug-ins include Flink input plug-ins, Flink transformation plug-ins, and Flink output plug-ins.

[0061] In S13, in the DAG real-time processing canvas, the Flink real-time data processing model is used to divide the entire real-time data processing process into modules. The modules include the Flink input table processing module, the Flink conversion table processing module, the Flink output table processing module, and the Flink data storage processing module.

[0062] It can be understood that in the DAG real-time processing canvas, the Flink real-time data processing construction model can be used to divide the entire real-time data processing process into four modules based on heterogeneous data table processing. The module composition may include: Flink input table processing module, Flink conversion table processing module, Flink output table processing module, and Flink data storage processing module. At the same time, two data processing relationships can be set: multiple input processing modules are associated with multiple conversion processing modules to construct multiple output processing modules, and the output processing modules are constructed with data storage modules in a one-to-one correspondence. Among them, the Flink real-time data processing construction model used can be shown as the following formula y1:

[0063]

[0064] Among them, FlinkTM() represents the real-time processing construction model, n1 is the number of input tables, n2 is the number of conversion tables, n3 is the number of output tables, and n4 is the number of storage media (i.e., data storage modules). The setting relationship is n3∈[n2,n1×n2] and n3=n4. TI is the Flink input table processing module, TF is the Flink conversion table processing module, TO is the Flink output table processing module, TS is the Flink data storage processing module, and the arrows represent the real-time data processing process.

[0065] In S14, in the DAG real-time processing canvas, select the Flink input plugin and configure the input table, select the Flink transformation plugin and configure the transformation rules, select the Flink output plugin and configure the output table and storage table.

[0066] It is understandable that Figure 2 and Figure 3 As shown, in the input table processing module of the DAG real-time processing canvas, select the Flink input plug-in and configure the input table. The Flink input plug-in can include the Kafka input plug-in, the file system input plug-in, the FlinkCDC input plug-in, and the custom data source input plug-in. The input table configuration may include the input source address, consumption topic, consumption group ID, file address, protocol parsing rules, and input table name.

[0067] In the Conversion Table Processing module of the DAG real-time processing canvas, select a Flink conversion plug-in and configure a conversion table. Flink conversion plug-ins can include SQL processing plug-ins, data filtering plug-ins, dimension association plug-ins, primary key deduplication plug-ins, and indicator aggregation plug-ins. Conversion table configuration can include the source table name, SQL query statement, filter fields, dimension fields, deduplication fields, aggregation fields, and conversion table name.

[0068] In the Output Table Processing module of the DAG real-time processing canvas, select a Flink output plug-in and configure the output table. Flink output plug-ins include Kafka, Redis, HBase, Elasticsearch, Mongodb, Hudi, Iceberg, and Paimon. Output table configuration includes the conversion table name, storage source address, production topic, data structure, primary key field, partition field, storage path, and output table name.

[0069] S15: During real-time data processing, the data table in the module is registered as a temporary table object based on Flink's internal temporary table registration model.

[0070] It can be understood that in the process of real-time data processing, the FlinkAPI internal temporary table registration model can be used to register the data tables in each module plug-in as temporary table objects. The registration objects may include Flink input tables, Flink conversion tables, and Flink output tables. Specifically, an internal registration method based on Flink temporary tables (Temporal Table) can be used. The execution engine based on the Flink Table API encapsulates the real-time data stream and registration features and completes the internal registration. The registration features may include the Flink execution environment, registry name, registration fields, and field types; finally, the data table in the module plug-in is registered as a temporary table object FTE through the Flink API internal temporary table registration model. Among them, the adopted Flink API internal temporary table registration model can be shown as the following formula y2:

[0071]

[0072] FlinkZM1() represents the Flink API registration model, d1 represents the number of real-time data streams, e1 represents the number of registration tables, c1 represents the number of table fields, FEnvi represents the Flink execution environment, Data represents the real-time data stream, TN represents the registration table name, CN represents the registration field, CT represents the field type, FTE represents the registered temporary table object, and the arrow indicates the temporary table registration process.

[0073] During real-time data processing, you can use the Flink SQL internal temporary table registration model to register the storage table in the data storage module as a temporary table object. Specifically, a temporary table creation statement based on SQL statements is constructed through data table configuration, and the aforementioned table creation statement is run and internal registration is completed based on the Flink SQL execution engine. Finally, the storage table in the storage module is registered as a temporary table object FTE through the Flink SQL internal temporary table registration model. The Flink SQL internal temporary table registration model used is shown in the following formula y3:

[0074]

[0075] FlinkZM2() represents the SQL registration model, e2 represents the number of registration tables, Config represents the data table configuration, SQL represents the temporary table creation statement, FlinkSQL represents the execution engine, FTE represents the registered temporary table object, and the arrow represents the temporary table registration process.

[0076] S16, during the real-time data processing process, uses the Flink real-time data stream conversion model to obtain the real-time data stream from the temporary table objects registered by each module plug-in.

[0077] It can be understood that in the real-time data processing process, the Flink real-time data stream conversion model can be used to obtain real-time data streams from the temporary table objects registered by each module plug-in. The temporary table objects can include Flink input temporary tables, Flink conversion temporary tables, and Flink output temporary tables.

[0078] Specifically, the first step is to extract the temporary table name and table structure from the data table configuration of each module plug-in and construct a Select statement based on the data query. The second step is to use the Flink SQL-based execution engine to run the Select statement and obtain the data table object Table. The third step is to obtain the field names and field types in the table structure and convert the data table object Table into a real-time data stream based on the FlinkTable API execution engine. Finally, the real-time data stream Stream is obtained from the registered temporary table object FTE through the Flink real-time data stream conversion model. The Flink real-time data stream conversion model used can be shown in the following formula y4:

[0079]

[0080] Among them, FlinkSM() represents the real-time stream conversion model, e3 is the number of registered temporary tables, d2 is the number of real-time data streams, FEnvi is the Flink execution environment, FTE is the registered temporary table object, Select is the query statement, Table is the data table object, Stream is the real-time data stream, and the arrow represents the real-time stream conversion process.

[0081] S17, during the real-time data processing process, a data table configuration modeling model is used to perform data modeling on the data table configuration in each module plug-in and define modeling features.

[0082] It is understood that the data table configuration modeling model can include the Flink input table configuration modeling model, the Flink transformation table configuration modeling model, the Flink output table configuration modeling model, and the Flink storage table configuration modeling model. During real-time data processing, the Flink input table configuration modeling model is used to perform data modeling for the input table configuration in the input module plug-in.

[0083] Specifically, configure the input table and modeling features in the Flink input plugin. Modeling features can include the input source address, consumption topic, consumption group ID, file address, protocol parsing rules, and input table name. Protocol parsing rules can include protocol field name, protocol field location, and protocol field type. These modeling features are encapsulated in the JSON data format. The Flink input table configuration modeling model used can be shown in the following formula y5:

[0084]

[0085] Among them, FTIM() represents the input table configuration model, v1 is the number of protocol fields, DS is the input source address, Topic is the consumption topic, GId is the consumption group ID, AD is the file address, CN is the protocol field name, CL is the protocol field location, CT is the protocol field type, and TNI is the input table name.

[0086] In some implementations, a sample Flink input table configuration based on the JSON data format may be as follows:

[0087] [{"bootstrap":"localhost:9092","topic":"metricm","groupid":"group001","schem ax":"{\"student\":\"String\",\"score\":\"int\",\"serveTime\":\"String\",\"itemName\":\"String\",\"ts\":\"long\"}","table_name":"student_001"}].

[0088] Among them, bootstrap is the input source address, topic is the consumption topic, groupid is the consumption group, schemax is the protocol parsing rule, table_name is the input table name, and [] represents multiple input modules.

[0089] In the real-time data processing module, the Flink conversion table configuration modeling model is used to model the conversion table configuration in the conversion module plug-in. Specifically, the Flink conversion plug-in configures the conversion table, conversion rules, and modeling features. These modeling features can include the source table name, SQL query statement, filter fields, dimension fields, deduplication fields, aggregation fields, and the conversion table name. These modeling features are also encapsulated in the JSON data format. The Flink conversion table configuration modeling model used can be shown in the following equation y6:

[0090]

[0091] Where FTFM() represents the conversion table configuration model, k1 is the number of SQL statements, k2 is the number of filter fields, k3 is the number of dimension fields, k4 is the number of deduplication fields, k5 is the number of aggregation fields, TNS is the source table name, SQL is the query statement, FF is the filter field, WF is the dimension field, DF is the deduplication field, AF is the aggregation field, TNF is the conversion table name, and a, b, c, d, and e are adjustment constants with values ​​of 0 or 1.

[0092] In some implementations, a sample Flink conversion table configuration based on the JSON data format may be as follows:

[0093] [{"sql":"select student,score,serveTime from student_score_001","table_name":"student_002"},{"from_table":"student_002","filter_field":"student","filter_value":"100,101,102","table_name":"student_003"}].

[0094] Among them, sql is the SQL query statement, from_table is the source table name, filter_field is the filter field, filter_value is the filter content, table_name is the conversion table name, and [] represents multiple conversion modules.

[0095] In the real-time data processing module, the Flink output table configuration modeling model is used to model the output table configuration in the output module plugin. Specifically, the Flink output plugin configures the output table and modeling features. These modeling features can include the conversion table name, table creation fields, field types, primary key fields, partition fields, storage path, and output table name. These modeling features are also encapsulated in the JSON data format. The Flink output table configuration modeling model used can be shown in the following equation y7:

[0096]

[0097] Where FTOM() represents the output table configuration model, z1 is the number of table creation fields, z2 is the number of primary key fields, z3 is the number of partition fields, TNF is the conversion table name, CN is the table creation field, CT is the field type, PK is the primary key field, PF is the partition field, PH is the storage path, TNO is the output table name, and f, g, and h are all adjustment constants with values ​​of 0 or 1.

[0098] In some implementations, a sample Flink output table configuration based on the JSON data format may be as follows:

[0099] [{"bootstrap":"localhost:9092","topic":"outputtopic","schema":"{\"score\":\"int\",\"student\":\"String\"}"},{"hudiserver":"l ocalhost:8080","primary_filed":"student","schemac":"{\"score\":\"int\",\"student\":\"String\"}","table_name":"student_004"}].

[0100] Among them, bootstrap is the Kafka storage source address, topic is the production topic, hudiserver is the Hudi storage source address, primary_filed is the primary key field, schemac is the table structure, table_name is the output table name, and [] represents multiple output modules.

[0101] In the real-time data processing module, the Flink storage table configuration modeling model is used to model the storage table configuration in the output module plug-in. Specifically, the storage table modeling features are extracted from the Flink output plug-in configuration. These modeling features may include the storage source address, production topic, table name, table fields, field types, primary key fields, partition fields, and storage path. These modeling features are then encapsulated in the JSON data format. The Flink storage table configuration modeling model used can be shown in the following equation y8:

[0102]

[0103] Among them, FTSM() represents the storage table configuration model, z1 is the number of table creation fields, z2 is the number of primary key fields, z3 is the number of partition fields, DS is the storage source address, Topic is the production topic, CN is the table creation field, CT is the field type, PK is the primary key field, PF is the partition field, PH is the storage path, TNS is the storage table name, q, f, g, and h are all adjustment constants with values ​​of 0 or 1.

[0104] S18: In the Flink input table processing module, a Flink input temporary table construction process is set and a temporary table registration model is used to register the input table as a temporary table.

[0105] As you can understand, the Flink input table processing module implements a Flink input temporary table construction process and registers the input table using the FlinkZM1 temporary table registration model. Specifically, the Flink input plugin and input table configuration are dynamically selected in the DAG real-time processing canvas. Input plugins can include Kafka input plugins, file system input plugins, FlinkCDC input plugins, and custom data source input plugins. Then, based on the existing FTIM input table configuration model, the required configuration features are retrieved from the input table configuration. These features may include the input source address, consumption topic, consumption group ID, protocol parsing rules, file address, and input table name. The table structure required for temporary table registration is then retrieved based on the protocol parsing rules. This table structure may include table fields and field types. Based on the data source address and consumption topic, the Flink real-time engine pulls the raw data stream IData from the input source in the input plugin. This real-time raw data stream is then parsed based on the protocol parsing rules to generate the real-time input stream IStream. Finally, the FlinkZM1 temporary table registration model is used to encapsulate the real-time input stream and the table structure, complete the internal registration, and finally build the input temporary table entity FTIE. The designed Flink input temporary table construction model can be shown as y9:

[0106]

[0107] Where n1 is the number of input tables, IData is the original data stream, IStream is the real-time input stream, FTIM is the input table configuration model, FlinkZM1 is the temporary table registration model, FTIE is the input temporary table entity, and the arrow indicates the temporary table registration process.

[0108] S19: In the Flink conversion table processing module, a Flink conversion temporary table construction process is set and a temporary table registration model is used to register the conversion table.

[0109] As can be understood, the Flink conversion table processing module implements a Flink conversion temporary table construction process and registers the conversion table using the FlinkZM1 temporary table registration model. Specifically, the Flink conversion plugin and conversion table configuration are dynamically selected in the DAG real-time processing canvas. Conversion plugins can include SQL processing plugins, data filtering plugins, dimension association plugins, primary key deduplication plugins, and metric aggregation plugins. The required configuration features are then retrieved from the conversion table configuration based on the FTFM conversion table configuration model. These features may include the source table name, SQL query statement, filter fields, dimension fields, deduplication fields, aggregation fields, and the conversion table name. The table structure required for temporary table registration is then retrieved based on the conversion configuration features. This table structure may include table fields and field types. The source table name is then specified to obtain a registered temporary table object. Based on the existing FlinkSM real-time stream conversion model, the real-time data stream is retrieved from the temporary table object as the data source IStream for the conversion module. This temporary table object may include a registered input temporary table or other conversion temporary tables. Then, different data conversion strategies are formulated based on the conversion rule type, and the real-time data stream is processed based on the conversion strategy to obtain the real-time conversion stream FStream. Finally, the FlinkZM1 temporary table registration model is used to encapsulate the real-time conversion stream and the table building structure and complete the internal registration, and finally build the conversion temporary table entity FTFE. Among them, the Flink conversion temporary table construction process can be set as follows: 10 As shown:

[0110]

[0111] Where n1 is the number of real-time input streams, n2 is the number of conversion tables, IStream is the real-time input stream, FStream is the real-time conversion stream, FTFM is the conversion table configuration model, FlinkZM1 is the temporary table registration model, FTFE is the conversion temporary table entity, and the arrow indicates the temporary table registration process.

[0112] S20: In the Flink output table processing module, a Flink output temporary table construction process is set and a temporary table registration model is used to register the output table.

[0113] As you can understand, the Flink output table processing module designs a Flink output temporary table construction process and registers the output table using the FlinkZM1 temporary table registration model. Specifically, the Flink output plugin and output table configuration are dynamically selected in the DAG real-time processing canvas. These output plugins can include Kafka, Redis, HBase, Elasticsearch, Mongodb, Hudi, Iceberg, and Paimon. Then, based on the existing FTOM output table configuration model, the required configuration features are retrieved from the output table configuration. These features include the conversion table name, data structure, primary key fields, partition fields, storage path, and output table name. The table structure required for temporary table registration is then retrieved based on the output configuration features. The table structure includes the table fields and field types. The conversion table name is then specified to obtain the registered temporary table object. Based on the FlinkSM real-time stream transformation model, the real-time data stream is extracted from the temporary table object and used as the real-time output stream OStream. Finally, the FlinkZM1 temporary table registration model is used to encapsulate the real-time output stream and the table structure and complete the internal registration, and finally build the output temporary table entity FTFE. Among them, the Flink output temporary table construction process can be set as follows: 11 As shown:

[0114]

[0115] In this example, n3 is the number of output tables, FTOM is the output table configuration model, FlinkSM is the real-time stream transformation model, OStream is the real-time output stream, FlinkZM1 is the temporary table registration model, FTOE is the output temporary table entity, and the arrow indicates the temporary table registration process.

[0116] S21: In the Flink data storage processing module, a Flink storage temporary table construction process is set and a temporary table registration model is used to register the storage table.

[0117] It can be understood that in the Flink data storage processing module, a Flink storage temporary table construction process is set up, and the storage table is registered as a temporary table based on the existing FlinkZM2 temporary table registration model. Specifically, the storage table configuration is first extracted in the Flink output plug-in selected in the DAG real-time processing canvas. Then, based on the existing FTSM storage table configuration model, the required configuration features are obtained from the storage table configuration. These configuration features may include storage source address, production topic, table name, table field, field type, primary key field, partition field and storage path, etc. Then, the corresponding storage connector template is pre-acquired according to the storage source type, and the storage connector template is instantiated based on the configuration features to obtain the temporary table registration table statement DCL. Finally, based on the FlinkZM2 temporary table registration model, the execution engine based on Flink SQL runs the table creation statement and completes the internal registration, and finally constructs the storage temporary table entity FTSE; wherein, the set Flink storage temporary table construction process can be as shown in y 12 As shown:

[0118]

[0119] In this example, n4 represents the number of storage tables, FTSM represents the storage table configuration model, Template represents the storage connector template, DCL represents the temporary table registration statement, FlinkZM2 represents the temporary table registration model, FTSE represents the storage temporary table entity, and the arrow indicates the temporary table registration process.

[0120] In some implementations, a Flink Connector-based data export method is used to design a multi-source heterogeneous output data source storage connector template; the storage connector templates corresponding to data sources of different data types can be obtained in a predefined manner.

[0121] Specifically, you can load predefined storage connector templates based on the data source type of each real-time output data source. These storage connector templates can include Kafka connector templates, Redis connector templates, HBase connector templates, Elasticsearch connector templates, Mongodb connector templates, Hudi connector templates, Iceberg connector templates, and Paimon connector templates. The definition of the Kafka storage connector template KT can be as follows:

[0122] CREATE TABLE${tn}(${cn1}${ct1},${cn2}${ct2},...,ts TIMESTAMP(3)METADATA FROM'timestamp')WITH('connector'='kafka','topic'='${topic}','properties.bootstrap.servers'='${host}',...).

[0123] The definition of the Redis storage connector template RT can be as follows:

[0124] CREATE TABLE${tn}(${cn1}${ct1},${cn2}${ct2},...)WITH('connector'='redis','host'='${host}').

[0125] The definition of the HBase storage connector template BT can be as follows:

[0126] CREATE TABLE${tn}(rowkey$pk,${fm1},${fm2},...)PRIMARY KEY(rowkey)NOTENFORCED WITH('connector'='hbase-1.4','table-name'='${tn}','zookeeper.quorum'='${host}').

[0127] The definition of the Elasticsearch storage connector template ET can be as follows:

[0128] CREATE TABLE${tn}(${cn1}${ct1},${cn2}${ct2},...,PRIMARY KEY(${pk})NOTENFORCED)WITH('connector'='elasticsearch-7','hosts'='${host}','in dex'='${tb}').

[0129] The definition of the Mongodb storage connector template MT can be as follows:

[0130] CREATE TABLE${tn}(${cn1}${ct1},${cn2}${ct2},...,PRIMARY KEY(${pk})NOTENFORCED)WITH('connector'='mongodb','uri'='${uri}','database'='${db}','collection'='${ctn}',...).

[0131] Among them, define the Hudi storage connector template HT as follows:

[0132] CREATE TABLE${tn}(${cn1}${ct1},${cn2}${ct2},...,PRIMARY KEY($pk)NOTENFORCED)PARTITIONED BY(${pf})WITH('connector'='hudi','path'='${path}','table.type'='MERGE_ON_READ',...).

[0133] Among them, define the Iceberg storage connector template IT as follows:

[0134] CREATE TABLE${tn}(${cn1}${ct1},${cn2}${ct2},...)PARTITIONED BY(${pf})WITH('connector'='iceberg','uri'='${uri}','catalog-database'='${db}','catalog-table'='${tb}','warehouse'='${path}',...).

[0135] The Paimon storage connector template PT is defined as follows:

[0136] CREATE TABLE paimon.${db}.${tn}(${cn1}${ct1},${cn2}${ct2},...,PRIMARY KEY($pk)NOT ENFORCED)PARTITIONED BY(${pf})WITH('buc ket'='8','bucket-key'='id','merge-engine'='partial-update',...).

[0137] Among them, ${} represents the template variable, uri is the connection address, host is the access address, topic is the production topic, db is the database name, tn is the table name, ctn is the document name, cn is the field name, ct is the field type, fm is the column family name, pk is the primary key field, pf is the partition field, and path is the storage path.

[0138] S22: In the Flink data storage processing module, the Flink real-time data synchronization model is used to execute the real-time data synchronization process of the external storage medium, and finally the entire real-time data processing process is completed.

[0139] It can be understood that in the Flink data storage processing module, the Flink real-time data synchronization model is used to implement the real-time data synchronization process of the external storage medium. Specifically, first extract the configuration features of the output table and the storage table in the Flink output plug-in, and register the output table as a temporary table based on the FlinkZM1 registration model, and register the storage table as a temporary table based on the FlinkZM2 registration model. Then set the association relationship between the output table and the storage table to be one-to-one; obtain the output table name and the storage table name according to the aforementioned configuration features, and design the data synchronization statement DIL based on InsertInto. Then use the FlinkSQL execution engine to run the data synchronization statement DIL to synchronize the real-time output stream from the output temporary table entity to the storage temporary table entity. Finally, obtain the external storage connector type in the storage temporary table configuration, and export the real-time stream in the storage temporary table to the external storage medium target table based on the Flink multi-source connector. Among them, the Flink real-time data synchronization model adopted can be as follows: 13 As shown:

[0140]

[0141] Where n3 is the number of output tables, n4 is the number of storage tables, and the relationship is n3=n4. FEnvi is the Flink execution environment, FTOE is the output registry entity, DIL is the data synchronization statement, FTSE is the storage temporary table entity, Connector is the Flink multi-source connector, TSE is the external media target table, and the arrows represent the data synchronization process.

[0142] In some implementations, the designed data synchronization statement template based on InsertInto may be as follows:

[0143] insert into${tableOut}select*from${tableIn}.

[0144] Among them, tableIn is the input table name, and tableOut is the output table name.

[0145] The existing Flink real-time computing engine in this field is used to implement the entire process of real-time data processing based on heterogeneous data tables. Specifically, the Flink real-time computing engine is used to build data table models in each link of data processing and implement the relationship transformation between input tables, conversion tables, output tables, and storage tables. By performing data conversion processing between data tables, the entire process of real-time data processing is finally completed.

[0146] The above-mentioned multi-source heterogeneous real-time data processing method reduces manual operations and eases system development difficulty by configuring based on a visual interface. It also designs a real-time data processing construction model to divide the entire real-time data processing process. It uses an internal temporary table registration model to register data tables in each module plug-in as temporary table objects, and uses a real-time data stream conversion model to obtain real-time data streams from registered temporary tables. It uses a data table configuration modeling model to model the data table configurations in each module plug-in. It designs a temporary table construction process and registers data tables based on a temporary table registration model. It also uses a real-time data synchronization model to implement real-time data synchronization with external storage media. Ultimately, through real-time data table processing, data processing is integrated across multiple links, ultimately completing the entire real-time data processing process. This solves the problem of unified management and control of multi-source heterogeneous real-time data and the problem of complex data development due to numerous processing processes. It also reduces labor costs and simplifies the system development process.

[0147] Compared with traditional technologies, the above solution establishes a unified data processing standard model, simplifies data processing logic, and reduces the complexity of system architecture; adopts in-memory data processing of temporary tables, greatly improving real-time data processing speed and development efficiency; is compatible with multiple heterogeneous data types and multiple source systems, solving data silos and consolidating data assets; and uses module plug-in data development with flexibility and scalability for different data types and rule processing requirements.

[0148] In one embodiment, Figure 4 As shown, further, the process of temporary table registration includes the following steps:

[0149] Get data table configuration;

[0150] Parse registration features according to data table configuration;

[0151] If the current data table is not a storage table, the Flink API internal temporary table registration model is used to register the temporary table and return the registration result.

[0152] Perform registration result verification;

[0153] Returns the registered entities of the database table.

[0154] As you can understand, if the current data table is not a storage table, a temporary table is registered using the Flink API. The Flink API then checks whether the temporary table registration for the data table (such as the input table, conversion table, and output table) is successful. If successful, a successful registration result is returned; otherwise, a failed registration result is returned. When verifying the registration result, if the registration is successful, the registered entity is returned normally; otherwise, a registration failure prompt is returned.

[0155] In one embodiment, Figure 4 As shown, further, the temporary table registration process also includes the following steps:

[0156] If the current data table is a storage table, the FlinkSQL internal temporary table registration model is used to register the temporary table and return the registration result;

[0157] Perform registration result verification;

[0158] Returns the registered entity of the storage table.

[0159] As you can understand, if the current data table is a storage table, the FlinkSQL registration method is used to register a temporary table. The success of the temporary table registration is then checked. If successful, a successful registration result is returned; otherwise, a failed registration result is returned. When verifying the registration result, if the registration is successful, the registered entity is returned normally; otherwise, a registration failure prompt is returned.

[0160] In one embodiment, Figure 5 As shown, further, the real-time data synchronization process includes:

[0161] Get output plugin configuration;

[0162] Parse configuration features based on output plugin configuration;

[0163] Execute the output table configuration and materialization process, call the synchronization template and obtain the data synchronization statement, and execute the storage table configuration and materialization process;

[0164] Synchronize the real-time output stream from the output temporary table entity to the storage temporary table entity;

[0165] Export the real-time stream from the temporary table entity to the target table in the external storage medium based on the Flink multi-source connector.

[0166] As you can understand, first obtain the output plugin configuration, then parse the configuration features, and then execute the corresponding materialization processes: output table configuration, use FlinkAPI registration to obtain the output temporary table entity, call the InsertInto data synchronization statement template to obtain the data synchronization statement DIL, and register the temporary table for the storage table configuration using FlinkSQL registration to obtain the storage temporary table entity, and establish data synchronization from the output temporary table entity to the storage temporary table entity. Then, perform data synchronization for the real-time storage stream: Based on the Flink Connector multi-source connector, the real-time stream in the storage temporary table is exported to the external storage media target table, and the success or failure of the data synchronization is output. After the entire external media synchronization process is completed, the synchronization result is returned to the front end.

[0167] It should be understood that although the above process Figure 1 、 Figure 4 and Figure 5 The steps in the flowchart are shown in the order indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 1 、 Figure 4 and Figure 5 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0168] In one embodiment, Figure 6As shown, a multi-source heterogeneous real-time data processing system 100 is provided, including a source configuration module 11, a rule configuration module 12, a canvas plug-in module 13, a process partitioning module 14, a data table configuration module 15, a temporary table registration module 16, a real-time flow module 17, a configuration modeling module 18, an input construction module 19, a conversion construction module 20, an output conversion module 21, a storage construction module 22 and a data synchronization module 23. Among them: the source configuration module 11 is used to add each real-time data source in the data source management page of the heterogeneous data source, configure the data source connection information and complete the connectivity test. The rule configuration module 12 is used to add each data processing rule in the rule configuration management page and save it to the rule configuration table; the data processing rules include protocol parsing rules, SQL processing rules, data filtering rules, dimension association rules, primary key deduplication rules, indicator aggregation rules and data structure rules. The canvas plug-in module 13 is used to create and edit Flink's DAG real-time processing canvas based on DAG interface operations and save it to the canvas table; in the DAG real-time processing canvas, data processing plug-ins are dynamically selected based on DAG dragging and dropping. Data processing plug-ins include Flink input plug-ins, Flink transformation plug-ins, and Flink output plug-ins. The process division module 14 is used to divide the entire real-time data processing process into modules in the DAG real-time processing canvas using the Flink real-time data processing construction model; the divided modules include the Flink input table processing module, the Flink transformation table processing module, the Flink output table processing module, and the Flink data storage processing module. The data table configuration module 15 is used to select the Flink input plug-in and configure the input table, select the Flink transformation plug-in and configure the transformation rules, select the Flink output plug-in and configure the output table and storage table in the DAG real-time processing canvas. The temporary table registration module 16 is used to register the data table in the module as a temporary table object based on the Flink internal temporary table registration model during the real-time data processing process.

[0169] The real-time streaming module 17 is used to obtain real-time data streams from the registered temporary table objects of each module plug-in using the Flink real-time data stream transformation model during real-time data processing. The configuration modeling module 18 is used to perform data modeling and define modeling features for the data table configurations in each module plug-in using the data table configuration modeling model during real-time data processing. The input construction module 19 is used to configure the Flink input temporary table construction process in the Flink input table processing module and register the input table using the temporary table registration model. The transformation construction module 20 is used to configure the Flink transformation temporary table construction process in the Flink transformation table processing module and register the transformation table using the temporary table registration model. The output transformation module 21 is used to configure the Flink output temporary table construction process in the Flink output table processing module and register the output table using the temporary table registration model. The storage construction module 22 is used to configure the Flink storage temporary table construction process in the Flink data storage processing module and register the storage table using the temporary table registration model. The data synchronization module 23 is used in the Flink data storage processing module to execute the real-time data synchronization process of the external storage medium using the Flink real-time data synchronization model, and finally complete the entire real-time data processing process.

[0170] The multi-source heterogeneous real-time data processing system 100 reduces manual operations and system development difficulty by configuring based on a visual interface; designs a real-time data processing construction model to divide the entire real-time data processing process; uses an internal temporary table registration model to register data tables in each module plug-in as temporary table objects, and uses a real-time data stream conversion model to obtain real-time data streams from registered temporary tables; uses a data table configuration modeling model to model the data table configuration in each module plug-in; designs a temporary table construction process and uses a temporary table registration model to register data tables; and uses a real-time data synchronization model to implement real-time data synchronization with external storage media. Ultimately, through real-time data table processing, data processing is integrated at multiple levels, ultimately completing the entire real-time data processing process. This solves the problem of unified management and control of multi-source heterogeneous real-time data and the problem of complex data development due to numerous processing processes. It also reduces labor costs and simplifies the system development process.

[0171] Compared with traditional technologies, the above solution establishes a unified data processing standard model, simplifies data processing logic, and reduces the complexity of system architecture; adopts in-memory data processing of temporary tables, greatly improving real-time data processing speed and development efficiency; is compatible with multiple heterogeneous data types and multiple source systems, solving data silos and consolidating data assets; and uses module plug-in data development with flexibility and scalability for different data types and rule processing requirements.

[0172] In one embodiment, the process of registering a temporary table includes the steps of: obtaining a data table configuration;

[0173] Parse registration features based on the data table configuration. If the current data table is not a storage table, use the Flink API internal temporary table registration model to register the temporary table and return the registration result. Perform registration result validation and return the registered entity of the data table.

[0174] In one embodiment, the temporary table registration process further includes the following steps: if the current data table is a storage table, registering the temporary table using the FlinkSQL internal temporary table registration model and returning the registration result; performing registration result verification; and returning the registration entity of the storage table.

[0175] In one embodiment, the real-time data synchronization process includes: obtaining output plug-in configuration; parsing configuration features based on the output plug-in configuration; executing output table configuration and instantiation processes, calling synchronization templates and obtaining data synchronization statements, executing storage table configuration and instantiation processes; synchronizing the real-time output stream from the output temporary table entity to the storage temporary table entity; and exporting the real-time stream in the storage temporary table entity to a target table on an external storage medium based on a Flink multi-source connector.

[0176] For the specific definition of the multi-source heterogeneous real-time data processing system 100, please refer to the corresponding definition of the multi-source heterogeneous real-time data processing method above, which will not be repeated here. Each module in the above-mentioned multi-source heterogeneous real-time data processing system 100 can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of a device with data processing functions in the form of hardware, or can be stored in the memory of the aforementioned device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. The aforementioned device can be, but is not limited to, various types of data computing and processing devices already available in the art.

[0177] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the following processing steps are implemented: adding each real-time data source, configuring data source connection information, and completing a connectivity test in the data source management page of the heterogeneous data source; adding each data processing rule in the rule configuration management page and saving it to the rule configuration table; the data processing rules include protocol parsing rules, SQL processing rules, data filtering rules, dimension association rules, primary key deduplication rules, indicator aggregation rules, and data structure rules; creating and editing Flink's DAG real-time processing canvas based on DAG interface operations and saving it to the canvas Table; In the DAG real-time processing canvas, dynamically select data processing plug-ins based on the DAG drag-and-drop function. Data processing plug-ins include Flink input plug-ins, Flink transformation plug-ins, and Flink output plug-ins. In the DAG real-time processing canvas, use the Flink real-time data processing model to divide the entire real-time data processing process into modules. The modules include the Flink input table processing module, the Flink transformation table processing module, the Flink output table processing module, and the Flink data storage processing module. In the DAG real-time processing canvas, select the Flink input plug-in and configure the input table. Select the Flink transformation plug-in and configure the transformation rules. , select the Flink output plug-in and configure the output table and storage table; in the real-time data processing process, register the data table in the module as a temporary table object based on the Flink internal temporary table registration model; in the real-time data processing process, use the Flink real-time data stream conversion model to obtain the real-time data stream from the temporary table objects registered by each module plug-in; in the real-time data processing process, use the data table configuration modeling model to perform data modeling on the data table configuration in each module plug-in and define the modeling features; in the Flink input table processing module, set the Flink input temporary table construction process and register the input table based on the temporary table registration model; in Fli In the nk conversion table processing module, the Flink conversion temporary table construction process is set up and the conversion table is registered as a temporary table based on the temporary table registration model. In the Flink output table processing module, the Flink output temporary table construction process is set up and the output table is registered as a temporary table based on the temporary table registration model. In the Flink data storage processing module, the Flink storage temporary table construction process is set up and the storage table is registered as a temporary table based on the temporary table registration model. In the Flink data storage processing module, the Flink real-time data synchronization model is used to execute the real-time data synchronization process of the external storage medium, finally completing the entire real-time data processing process.

[0178] It can be understood that in addition to the memory and processor mentioned above, the above-mentioned computer device also includes other software and hardware components not listed in this specification. The specific components can be determined according to the model of the specific computer device in different application scenarios. This specification will not list them one by one in detail.

[0179] In one embodiment, when the processor executes the computer program, it can also implement the steps or sub-steps added in each embodiment of the above-mentioned multi-source heterogeneous real-time data processing method.

[0180] In one embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the following processing steps are implemented: adding each real-time data source, configuring data source connection information, and completing a connectivity test in the data source management page of the heterogeneous data source; adding each data processing rule in the rule configuration management page and saving it to the rule configuration table; the data processing rules include protocol parsing rules, SQL processing rules, data filtering rules, dimension association rules, primary key deduplication rules, indicator aggregation rules, and data structure rules; creating and editing Flink's DAG real-time processing canvas based on DAG interface operations and saving it to the canvas table; in D In the AG real-time processing canvas, you can dynamically select data processing plug-ins based on the DAG drag-and-drop function. Data processing plug-ins include Flink input plug-ins, Flink transformation plug-ins, and Flink output plug-ins. In the DAG real-time processing canvas, the Flink real-time data processing model is used to divide the entire real-time data processing process into modules. The modules include Flink input table processing module, Flink transformation table processing module, Flink output table processing module, and Flink data storage processing module. In the DAG real-time processing canvas, select the Flink input plug-in and configure the input table, select the Flink transformation plug-in and configure the transformation rules, and select Flink output plug-in and configure output table and storage table; in the real-time data processing process, the data table in the module is registered as a temporary table object based on Flink's internal temporary table registration model; in the real-time data processing process, the Flink real-time data stream conversion model is used to obtain real-time data streams from the temporary table objects registered by each module plug-in; in the real-time data processing process, the data table configuration modeling model is used to perform data modeling on the data table configuration in each module plug-in and define modeling features; in the Flink input table processing module, the Flink input temporary table construction process is set and the input table is registered as a temporary table based on the temporary table registration model; in Flink In the conversion table processing module, the Flink conversion temporary table construction process is set up and the conversion table is registered as a temporary table based on the temporary table registration model; in the Flink output table processing module, the Flink output temporary table construction process is set up and the output table is registered as a temporary table based on the temporary table registration model; in the Flink data storage processing module, the Flink storage temporary table construction process is set up and the storage table is registered as a temporary table based on the temporary table registration model; in the Flink data storage processing module, the Flink real-time data synchronization model is used to execute the real-time data synchronization process of the external storage medium, finally completing the entire real-time data processing process.

[0181] In one embodiment, when the computer program is executed by a processor, it can also implement the steps or sub-steps added in each embodiment of the above-mentioned multi-source heterogeneous real-time data processing method.

[0182] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus dynamic random access memory (Rambus DRAM, referred to as RDRAM) and interface dynamic random access memory (DRDRAM).

[0183] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0184] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A multi-source heterogeneous real-time data processing method, characterized in that: Including steps: In the data source management page of the heterogeneous data source, add each real-time data source, configure the data source connection information, and complete the connectivity test; Add data processing rules in the rule configuration management page and save them to the rule configuration table; the data processing rules include protocol parsing rules, SQL processing rules, data filtering rules, dimension association rules, primary key deduplication rules, indicator aggregation rules and data structure rules; Create and edit a Flink DAG real-time processing canvas based on the DAG interface and save it to the canvas table; dynamically select data processing plug-ins in the DAG real-time processing canvas by dragging and dropping the DAG. The data processing plug-ins include Flink input plug-ins, Flink conversion plug-ins, and Flink output plug-ins. In the DAG real-time processing canvas, the Flink real-time data processing model is used to divide the entire real-time data processing process into modules. The modules include the Flink input table processing module, the Flink conversion table processing module, the Flink output table processing module, and the Flink data storage processing module. In the DAG real-time processing canvas, select the Flink input plugin and configure the input table, select the Flink transformation plugin and configure the transformation rules, select the Flink output plugin and configure the output table and storage table. During the real-time data processing, the data table in the module is registered as a temporary table object based on the Flink internal temporary table registration model; In the real-time data processing process, the Flink real-time data stream conversion model is used to obtain the real-time data stream from the temporary table objects registered by each module plug-in; In the real-time data processing process, a data table configuration modeling model is used to perform data modeling on the data table configuration in each module plug-in and define modeling features; In the Flink input table processing module, set up the Flink input temporary table construction process and register the input table based on the temporary table registration model; In the Flink conversion table processing module, set up the Flink conversion temporary table construction process and register the conversion table based on the temporary table registration model; In the Flink output table processing module, set up the Flink output temporary table construction process and register the output table based on the temporary table registration model; In the Flink data storage processing module, set up the Flink storage temporary table construction process and register the storage table based on the temporary table registration model; In the Flink data storage and processing module, the Flink real-time data synchronization model is used to execute the real-time data synchronization process of the external storage medium, and finally complete the entire real-time data processing process.

2. The multi-source heterogeneous real-time data processing method according to claim 1, characterized in that: The temporary table registration process includes the following steps: Get data table configuration; Parsing registration features according to the data table configuration; If the current data table is not a storage table, the Flink API internal temporary table registration model is used to register the temporary table and return the registration result. Perform registration result verification; Returns the registered entities for the specified table.

3. The multi-source heterogeneous real-time data processing method according to claim 2, characterized in that: The temporary table registration process further includes the following steps: If the current data table is a storage table, the FlinkSQL internal temporary table registration model is used to register the temporary table and return the registration result; Perform registration result verification; Returns the registered entities for the specified storage table.

4. The multi-source heterogeneous real-time data processing method according to any one of claims 1 to 3, characterized in that: The real-time data synchronization process includes: Get output plugin configuration; Parsing configuration features according to the output plugin configuration; Execute the output table configuration and materialization process, call the synchronization template and obtain the data synchronization statement, and execute the storage table configuration and materialization process; Synchronize the real-time output stream from the output temporary table entity to the storage temporary table entity; Export the real-time stream from the temporary table entity to the target table in the external storage medium based on the Flink multi-source connector.

5. A multi-source heterogeneous real-time data processing system, characterized in that: include: The source configuration module is used to add real-time data sources, configure data source connection information, and complete connectivity testing in the data source management page of heterogeneous data sources; The rule configuration module is used to add data processing rules in the rule configuration management page and save them to the rule configuration table; the data processing rules include protocol parsing rules, SQL processing rules, data filtering rules, dimension association rules, primary key deduplication rules, indicator aggregation rules and data structure rules; The canvas plug-in module is used to create and edit Flink's DAG real-time processing canvas based on the DAG interface and save it to the canvas table; Dynamically select data processing plug-ins based on DAG dragging and dropping in the DAG real-time processing canvas, wherein the data processing plug-ins include Flink input plug-ins, Flink conversion plug-ins, and Flink output plug-ins; A process partitioning module is used to divide the entire real-time data processing process into modules in the DAG real-time processing canvas using the Flink real-time data processing model. The modules include a Flink input table processing module, a Flink conversion table processing module, a Flink output table processing module, and a Flink data storage processing module. A data table configuration module is used to select the Flink input plug-in and configure the input table, select the Flink transformation plug-in and configure the transformation rules, select the Flink output plug-in and configure the output table and storage table in the DAG real-time processing canvas; A temporary table registration module is used to register the data table in the module as a temporary table object based on Flink's internal temporary table registration model during the real-time data processing process; A real-time streaming module is used to obtain real-time data streams from temporary table objects registered by each module plug-in using the Flink real-time data stream conversion model during the real-time data processing process; A configuration modeling module is used to perform data modeling on the data table configuration in each module plug-in and define modeling features using a data table configuration modeling model during the real-time data processing process; The input construction module is used to set up the Flink input temporary table construction process in the Flink input table processing module and register the input table based on the temporary table registration model; The conversion construction module is used to set up the Flink conversion temporary table construction process in the Flink conversion table processing module and register the conversion table based on the temporary table registration model; The output conversion module is used to set up the Flink output temporary table construction process in the Flink output table processing module and register the output table based on the temporary table registration model; The storage construction module is used to set up the Flink storage temporary table construction process in the Flink data storage processing module and register the storage table based on the temporary table registration model; The data synchronization module is used in the Flink data storage and processing module to perform real-time data synchronization of external storage media using the Flink real-time data synchronization model, ultimately completing the entire real-time data processing process.

6. The multi-source heterogeneous real-time data processing system according to claim 5, characterized in that: The temporary table registration process includes the following steps: Get data table configuration; Parsing registration features according to the data table configuration; If the current data table is not a storage table, the Flink API internal temporary table registration model is used to register the temporary table and return the registration result. Perform registration result verification; Returns the registered entities for the specified table.

7. The multi-source heterogeneous real-time data processing system according to claim 6, characterized in that: The temporary table registration process further includes the following steps: If the current data table is a storage table, the FlinkSQL internal temporary table registration model is used to register the temporary table and return the registration result; Perform registration result verification; Returns the registered entities for the specified storage table.

8. The multi-source heterogeneous real-time data processing system according to any one of claims 5 to 7, characterized in that: The real-time data synchronization process includes: Get output plugin configuration; Parsing configuration features according to the output plugin configuration; Execute the output table configuration and materialization process, call the synchronization template and obtain the data synchronization statement, and execute the storage table configuration and materialization process; Synchronize the real-time output stream from the output temporary table entity to the storage temporary table entity; Export the real-time stream from the temporary table entity to the target table in the external storage medium based on the Flink multi-source connector.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the multi-source heterogeneous real-time data processing method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-source heterogeneous real-time data processing method according to any one of claims 1 to 4 are implemented.

Images