Data processing method and system based on flink unified stream and batch processing, and device and medium

Abstract

A data processing method and system based on Flink unified stream and batch processing, and a device and a medium, which relate to the field of big data processing. The method comprises: establishing an SQL statement construction rule on the basis of data elements specified in an SQL file standard and a logic relationship between the data elements, so as to perform SQL statement construction on a service to be performed, and obtain a query statement file, wherein the data elements comprise a source table SQL, a destination table SQL and a data conversion SQL; submitting configuration parameters of the SQLs and the query statement file to a Flink cluster, so as to obtain corresponding data processing task information; and the Flink cluster receiving a target data source table on the basis of the data processing task information, processing the data source table on the basis of a predetermined processing mechanism, and then writing corresponding data into a data destination table. The present invention is applicable to different data processing tasks without the data read and write modules in the present invention being rewritten, and it can be ensured that data is executed at least once during a running process of a task.

Description

Data processing method, system, device and medium based on Flink stream-batch integration Technical Field

[0001] The present invention relates to the field of big data processing, and more specifically, to a data processing method, system, device, and medium based on Flink stream-batch integration. Background Art

[0002] Currently, existing data processing tasks are implemented by writing corresponding logic for individual businesses. Technical issues

[0003] Even with the same data processing logic, if the data input and output sources differ from previous tasks, the data reading and writing modules must be rewritten. Furthermore, during the task execution, abnormal situations such as network fluctuations may occur, causing the entire task to fail. Technical Solutions

[0004] The purpose of the present invention is to provide a data processing method, system, device and medium based on Flink's stream-batch integration, which can be applied to different data processing tasks without rewriting the data reading and writing modules, and can ensure that the data is executed at least once during the running of the task.

[0005] The present invention is achieved in that:

[0006] First, the data processing method based on Flink stream-batch integration provided by this application includes the following steps:

[0007] Based on the data elements specified in the SQL file standard and the logical relationships between the above data elements, SQL statement construction rules are established, and the above SQL statement construction rules are used to construct SQL statements for the intended business to obtain a query statement file; the above data elements include source table SQL, destination table SQL and data conversion SQL; the SQL configuration parameters and the above query statement file are submitted to the Flink cluster to obtain corresponding data processing task information; the above Flink cluster receives the target data source table based on the above data processing task information, processes the above data source table based on a predetermined processing mechanism, and writes the corresponding data into the data destination table.

[0008] Furthermore, based on the aforementioned solution, the data structure and connection parameter format of the above-mentioned source table SQL and the above-mentioned destination table SQL are: create table (data format) with (connection parameters); the above-mentioned data conversion SQL carries a UDF function and specifies the mapping relationship between fields. The calling format of the above-mentioned UDF function is: insert into select * from table.

[0009] Furthermore, based on the above solution, the content of the above UDF function is the code implementation of the data conversion logic in the above predetermined processing mechanism, so as to inherit the corresponding interface and complete the corresponding data conversion logic.

[0010] Furthermore, based on the above-mentioned solution, the above-mentioned SQL configuration parameters and the above-mentioned query statement file are submitted to the Flink cluster to obtain corresponding data processing task information, including: converting the source table SQL into the data receiving operator of the data processing task, converting the destination table SQL into the data output operator of the data processing task, and converting the data conversion SQL into the data conversion operator of the data processing task, and based on the above-mentioned data receiving operator, the above-mentioned data output operator, the above-mentioned data conversion operator and the SQL configuration parameters, the corresponding data processing task information is obtained.

[0011] Furthermore, based on the above solution, the Flink cluster receives the target data source table based on the data processing task information, processes the data source table based on a predetermined processing mechanism, and writes the corresponding data into the data destination table, including:

[0012] The above-mentioned data receiving operator consumes or batch reads the original data from the target data source table in real time according to the configuration parameters of the above-mentioned SQL, and forwards the above-mentioned original data to the above-mentioned data conversion operator; the above-mentioned data conversion operator parses the above-mentioned original data into the output standard data format according to the predetermined analysis rules, obtains standard output data, and forwards it to the above-mentioned data output operator; the above-mentioned data output operator writes the above-mentioned standard output data into the data destination table according to the configuration parameters of the above-mentioned SQL.

[0013] Furthermore, based on the aforementioned solution, the data processing tasks corresponding to the above data processing task information include tasks for processing real-time data writing into the data lake and offline tasks for data migration.

[0014] Furthermore, based on the above solution, it also includes displaying the process and / or result of writing the corresponding data into the data destination table in real time in the user interface.

[0015] Secondly, the present application provides a data processing system based on Flink stream-batch integration, which includes:

[0016] The query statement module is configured to establish SQL statement construction rules based on the data elements specified in the SQL file standard and the logical relationships between the above data elements, and use the above SQL statement construction rules to construct SQL statements for the pre-performed business to obtain a query statement file; the above data elements include source table SQL, destination table SQL and data conversion SQL; the task determination module is configured to submit the SQL configuration parameters and the above query statement file to the Flink cluster to obtain the corresponding data processing task information; the data processing module is configured for the Flink cluster to receive the target data source table based on the above data processing task information, and after processing the above data source table based on a predetermined processing mechanism, write the corresponding data into the data destination table.

[0017] In a third aspect, the present application provides an electronic device comprising at least one processor, at least one memory and a data bus; wherein: the processor and the memory communicate with each other through the data bus; the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute a method as described in any one of the first aspects above.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first aspects above. Beneficial effects

[0019] Compared with the prior art, the present invention has at least the following advantages or beneficial effects:

[0020] The present invention proposes a data processing method based on Flink stream-batch integration, which can use pre-built SQL statement construction rules to construct SQL statements for pre-performed business, wherein the SQL statement construction rules are established based on the three data elements of source table SQL, destination table SQL and data conversion SQL themselves and the logical relationship between them. The constructed SQL statement is then submitted to the Flink cluster, and the Flink cluster will generate corresponding data processing tasks based on the received SQL statement. Then, the data in the data source table can be received according to the data processing task, and after corresponding processing, the processed data will be written into the data destination table. It does not need to rewrite the data reading and writing modules therein, and can be applied to different data processing tasks, and can ensure that the data will be executed at least once during the operation of the task. That is, it can quickly build data processing tasks on demand, and at the same time, it can also ensure the running status of the task. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0022] FIG1 is a flow chart of an embodiment of a data processing method based on Flink stream-batch integration according to the present invention;

[0023] FIG2 is a schematic diagram of an embodiment of a data processing method based on Flink stream-batch integration according to the present invention;

[0024] FIG3 is a specific flow chart of the steps of receiving a target data source table based on the data processing task information, processing the data source table based on a predetermined processing mechanism, and then writing the corresponding data into a data destination table according to an embodiment of the present invention;

[0025] FIG4 is a structural block diagram of an embodiment of a data processing system based on Flink stream-batch integration according to the present invention;

[0026] FIG5 is a structural block diagram of an electronic device provided by an embodiment of the present invention.

[0027] Icons: 1. Query statement module; 2. Task determination module; 3. Data processing module; 4. Processor; 5. Memory; 6. Data bus. Modes for Carrying Out the Invention

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0029] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features thereof may be combined with each other. Example 1

[0030] Currently, data processing tasks are implemented by writing logic specifically for individual businesses, making it difficult to reuse. Furthermore, during the execution of a task, an exception may occur, causing the entire task to fail.

[0031] To address the above issues, the embodiments of the present application provide a data processing method, system, device, and medium based on Flink's integrated stream and batch processing. The specific process of executing data processing tasks is defined through SQL, including the source, destination, and conversion logic of the data. The data conversion logic can be reused in multiple different tasks, making task development more efficient. At the same time, the tasks use the Flink framework, and the created tasks have a failure retry mechanism and a task checkpoint mechanism, which can ensure that the data is executed at least once during the task operation.

[0032] Referring to Figure 1, the data processing method based on Flink's stream-batch integration includes the following steps:

[0033] Step S101: Establish SQL statement construction rules based on the data elements specified in the SQL file standard and the logical relationship between the above data elements, and use the above SQL statement construction rules to construct SQL statements for the pre-performed business to obtain a query statement file; the above data elements include source table SQL, destination table SQL and data conversion SQL.

[0034] In the above steps, the SQL file is the query statement file. By predefining the types of data elements involved in the query statement file and their mutual associations and logical relationships, standard SQL statement construction rules can be obtained. Among them, the source table SQL can be used to obtain the specified data table, and the data conversion SQL can convert the data in the specified data table obtained through the source table SQL according to the predetermined conversion rules. Then, the converted data can be written into the corresponding target table through the target table SQL. In this process, the data conversion SQL can be reused in different tasks, thereby improving the efficiency of SQL statement construction, that is, improving the efficiency of query statement file acquisition. In addition, for data reading (based on source table SQL) and data writing (based on target table SQL), it is only necessary to construct the source or purpose of the data, and the construction task volume is small and the construction efficiency is high.

[0035] The "stream and batch" part of stream-batch integration refers to stream processing and batch processing, two different data processing methods, rather than a classification of data types. Specifically, data can be divided into historical data and real-time data based on the time of generation, and into flow data and sliced ​​data based on the level of data detail. Data processing methods can be divided into stream processing and batch processing based on window size, and into real-time processing and offline processing based on processing latency.

[0036] For example, it can be constructed by using the existing source table SQL and target table SQL construction model, or it can be constructed manually.

[0037] In order to ensure that the source table SQL can be used to receive data from the data source table and the target table SQL can be used to write data to the target table, the source table SQL and the target table SQL need to construct the data format of the corresponding table and the connection parameters used to connect to the corresponding table. In some embodiments of the present invention, the data structure and connection parameter format of the source table SQL and the target table SQL are: create table (data format) with (connection parameters), wherein the data format represents the format of the data in the data source table (or the corresponding table corresponding to the target table), and the connection parameters represent the connection parameters used to connect to the data source table (or the corresponding table corresponding to the target table); the data conversion SQL carries a UDF function and specifies the mapping relationship between fields. The calling format of the UDF function is: insert into select * from table. In some embodiments of the present invention, the content of the UDF function is the code implementation of the data conversion logic in the predetermined processing mechanism, so as to inherit the corresponding interface and complete the corresponding data conversion logic.

[0038] A UDF (User Defined Function) is a user-defined function defined, implemented, and applied in data processing using the UDF framework. It's primarily used to address the limitations of custom data processing in classic SQL distributed queries, as well as complex logic that's difficult to implement in SQL queries. UDFs can efficiently implement complex logic, making data processing easier and simpler.

[0039] In other words, you can use UDF functions to pre-build complex logical data conversion methods, which can then be called in the data conversion SQL. By specifying the mapping relationships between fields, you can effectively map the fields involved in the conversion process.

[0040] Step S102: Submit the SQL configuration parameters and the query statement file to the Flink cluster to obtain the corresponding data processing task information.

[0041] In the above steps, you can package the SQL configuration parameters and the query statement file into a JAR file and store it in Flink's lib directory as a dependency for the task. When submitted to the Flink cluster for execution, the source table SQL will be converted into a data receiving operator for the data processing task, the destination table SQL will be converted into a data output operator for the data processing task, and the data conversion SQL will be converted into a data conversion operator for the data processing task. Subsequently, based on the data receiving operator, data output operator, data conversion operator, and SQL configuration parameters, the corresponding data processing task information can be obtained.

[0042] The lib directory contains shared libraries required by programs on the root file system. These files contain code that can be shared by many programs, avoiding the need for each program to contain a copy of the same subroutine. This helps to reduce the size of the executable file and save space.

[0043] Step S103: The Flink cluster receives the target data source table based on the data processing task information, processes the data source table based on a predetermined processing mechanism, and writes the corresponding data into the data destination table.

[0044] As shown in FIG3 , in some embodiments of the present invention, the Flink cluster receives the target data source table based on the data processing task information, processes the data source table based on a predetermined processing mechanism, and writes the corresponding data into the data destination table, including:

[0045] The above-mentioned data receiving operator consumes or batch reads the original data from the target data source table in real time according to the configuration parameters of the above-mentioned SQL, and forwards the above-mentioned original data to the above-mentioned data conversion operator; the above-mentioned data conversion operator parses the above-mentioned original data into the output standard data format according to the predetermined analysis rules, obtains standard output data, and forwards it to the above-mentioned data output operator; the above-mentioned data output operator writes the above-mentioned standard output data into the data destination table according to the configuration parameters of the above-mentioned SQL.

[0046] As shown in Figure 2, when the present invention is implemented, it is possible to use pre-built SQL statement construction rules to construct SQL statements for the pre-performed business, wherein the SQL statement construction rules are established based on the three data elements of the source table SQL, the destination table SQL and the data conversion SQL themselves and the logical relationship between them. The constructed SQL statement is then submitted to the Flink cluster, and the Flink cluster will generate the corresponding data processing task based on the received SQL statement. Then, the data in the data source table can be received according to the data processing task, and after corresponding processing, the processed data will be written into the data destination table. In this way, data processing tasks can be quickly constructed on demand, and the running status of the task can also be guaranteed. In addition, the data conversion SQL can be applied to different tasks and can be directly reused without rewriting, which can improve the code development efficiency of its data processing tasks.

[0047] In some embodiments of the present invention, the data processing tasks corresponding to the above-mentioned data processing task information include tasks for processing real-time data written into the data lake and offline tasks for data migration.

[0048] For example, in some implementations of the present invention, creating a task for processing real-time data written into a data lake includes the following steps:

[0049] Write the source table SQL, select Kafka as the data type, and configure Kafka connection parameters, such as the server address and topic name.

[0050] Write a custom UDF parsing function, inherit TableFunction, rewrite the eval method to write the data conversion logic. The function input parameters are the parsing expression and original data, and the function outputs data in a standard format.

[0051] Write the SQL for the destination table, selecting HDFS as the data type and configuring the corresponding connection parameters, such as the HDFS address and table type. Write the SQL for the conversion operator to specify the mapping between fields, call the UDF, and write to the destination table. Submit the SQL task through Flink's sql-client.

[0052] In some implementations of the present invention, creating an offline data migration task includes the following steps:

[0053] Write the source table SQL, select MySQL as the data type, and configure the MySQL connection address, such as the account number, password, and other parameters;

[0054] Write a custom UDF parsing function, inherit ScalarFunction and rewrite the eval method to write the data conversion logic function. The input parameters are the parsing expression and the original data, and the function outputs data in a standard format.

[0055] Write the SQL for the target table, select hudi as the data type, and configure the corresponding connection parameters, such as the HDFS address and table type. Submit the SQL task using Flink's sql-client.

[0056] For ease of understanding, the terms involved in the above implementation are briefly described as follows:

[0057] eval is a built-in JavaScript function that evaluates a string expression. For example, eval(2+3) returns 5.

[0058] HDFS definition: HDFS (Hadoop Distributed File System) is a file system used to store files and locate files through a directory tree. Secondly, it is distributed and is implemented by many servers working together to achieve its functions. Servers in the cluster have their own roles.

[0059] ScalarFunction, a custom scalar function in FlinkSQL, is similar to Map and operates in a one-to-one manner, inputting one row of data and outputting one row of data.

[0060] In some embodiments of the present invention, the process and / or result of writing the corresponding data into the data destination table is also displayed in real time in the user interface. This allows users to clearly and timely understand whether the corresponding data has been successfully written into the destination table, and by displaying the writing process, users can monitor the writing process in real time, as well as trace abnormal situations. And at the beginning: SQL statement construction rules are established based on the data elements specified in the SQL file standard and the logical relationship between the data elements, and the SQL statement construction rules are used to construct SQL statements for the pre-performed business to obtain a query statement file. Therefore, for errors that occur during the writing process, the query statement file can be used to construct the corresponding interpretation language (related language information for explaining the error situation) and interpretation nodes (data nodes related to the error situation). Example 2

[0061] Referring to FIG4 , an embodiment of the present application provides a data processing system based on Flink stream-batch integration, which includes:

[0062] The query statement module 1 is configured to establish SQL statement construction rules based on the data elements specified in the SQL file standard and the logical relationship between the above data elements, and use the above SQL statement construction rules to construct SQL statements for the pre-performed business to obtain a query statement file; the above data elements include source table SQL, destination table SQL and data conversion SQL; the task determination module 2 is configured to submit the SQL configuration parameters and the above query statement file to the Flink cluster to obtain the corresponding data processing task information; the data processing module 3 is configured for the above Flink cluster to receive the target data source table based on the above data processing task information, and after processing the above data source table based on a predetermined processing mechanism, write the corresponding data into the data destination table.

[0063] Optionally, the data structure and connection parameter format of the above-mentioned source table SQL and the above-mentioned destination table SQL are: create table (data format) with (connection parameters); the above-mentioned data conversion SQL carries a UDF function and specifies the mapping relationship between fields. The calling format of the above-mentioned UDF function is: insert in to select * from table.

[0064] Optionally, the content of the UDF function is a code implementation of the data conversion logic in the predetermined processing mechanism, so as to inherit the corresponding interface and complete the corresponding data conversion logic.

[0065] Optionally, the above-mentioned task determination module 2 is specifically used to: convert the source table SQL into the data receiving operator of the data processing task, convert the destination table SQL into the data output operator of the data processing task, convert the data conversion SQL into the data conversion operator of the data processing task, and obtain the corresponding data processing task information based on the configuration parameters of the above-mentioned data receiving operator, the above-mentioned data output operator, the above-mentioned data conversion operator and SQL.

[0066] Optionally, the task data processing module 3 is specifically used for: the data receiving operator consumes or batch reads the original data from the target data source table in real time according to the configuration parameters of the SQL, and forwards the original data to the data conversion operator; the data conversion operator parses the original data into an output standard data format according to predetermined analysis rules, obtains standard output data, and forwards it to the data output operator; the data output operator writes the standard output data into the data destination table according to the configuration parameters of the SQL.

[0067] Optionally, the data processing tasks corresponding to the above data processing task information include tasks for processing real-time data and writing it into the data lake, and offline tasks for data migration.

[0068] Optionally, the above system further includes:

[0069] The process and result display module is configured to display the process and / or result of writing the corresponding data into the data destination table in real time in the user interface.

[0070] For the specific implementation process of the above system, please refer to the data processing method based on Flink stream-batch integration provided in Example 1, which will not be repeated here. Example 3

[0071] Referring to Figure 5 , an embodiment of the present application provides an electronic device comprising at least one processor 4 , at least one memory 5 , and a data bus 6 ; wherein: the processor 4 and the memory 5 communicate with each other via the data bus 6 ; the memory 5 stores program instructions executable by the processor 4 , and the processor 4 calls the program instructions to execute a data processing method based on Flink stream-batch integration. For example, the following is implemented:

[0072] Based on the data elements specified in the SQL file standard and the logical relationships between the above data elements, SQL statement construction rules are established, and the above SQL statement construction rules are used to construct SQL statements for the intended business to obtain a query statement file; the above data elements include source table SQL, destination table SQL and data conversion SQL; the SQL configuration parameters and the above query statement file are submitted to the Flink cluster to obtain corresponding data processing task information; the above Flink cluster receives the target data source table based on the above data processing task information, processes the above data source table based on a predetermined processing mechanism, and writes the corresponding data into the data destination table.

[0073] Among them, the memory 5 can be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0074] The processor 4 can be an integrated circuit chip with signal processing capabilities. The processor 4 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0075] It is understood that the structure shown in Figure 5 is merely illustrative, and the electronic device may include more or fewer components than shown in Figure 5, or have a configuration different from that shown in Figure 5. Each component shown in Figure 5 may be implemented using hardware, software, or a combination thereof. Example 4

[0076] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor 4, implements a data processing method based on Flink stream-batch integration. For example, it implements:

[0077] Based on the data elements specified in the SQL file standard and the logical relationships between the above data elements, SQL statement construction rules are established, and the above SQL statement construction rules are used to construct SQL statements for the intended business to obtain a query statement file; the above data elements include source table SQL, destination table SQL and data conversion SQL; the SQL configuration parameters and the above query statement file are submitted to the Flink cluster to obtain corresponding data processing task information; the above Flink cluster receives the target data source table based on the above data processing task information, processes the above data source table based on a predetermined processing mechanism, and writes the corresponding data into the data destination table.

[0078] If the above functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0079] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above and that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the present application is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A data processing method based on Flink stream-batch integration, characterized in that: The following steps are involved: Establishing SQL statement construction rules based on data elements specified in the SQL file standard and the logical relationship between the data elements, and using the SQL statement construction rules to construct SQL statements for the business to be performed, to obtain a query statement file; the data elements include source table SQL, destination table SQL and data conversion SQL; Submit the SQL configuration parameters and the query statement file to the Flink cluster to obtain the corresponding data processing task information; The Flink cluster receives the target data source table based on the data processing task information, processes the data source table based on a predetermined processing mechanism, and writes the corresponding data into the data destination table.

2. The data processing method based on Flink stream-batch integration as claimed in claim 1, characterized in that: The data structure and connection parameter format of the source table SQL and the target table SQL are: create table (data format) with (connection parameters); The data conversion SQL carries a UDF function and specifies the mapping relationship between fields. The calling format of the UDF function is: insert into select*from table.

3. The data processing method based on Flink stream-batch integration as claimed in claim 2, characterized in that: The content of the UDF function is the code implementation of the data conversion logic in the predetermined processing mechanism, so as to inherit the corresponding interface and complete the corresponding data conversion logic.

4. The data processing method based on Flink stream-batch integration as claimed in claim 1, characterized in that: Submitting the SQL configuration parameters and the query statement file to the Flink cluster to obtain the corresponding data processing task information includes: The source table SQL is converted into a data receiving operator of a data processing task, the destination table SQL is converted into a data output operator of a data processing task, the data conversion SQL is converted into a data conversion operator of a data processing task, and based on the configuration parameters of the data receiving operator, the data output operator, the data conversion operator and the SQL, the corresponding data processing task information is obtained.

5. The data processing method based on Flink stream-batch integration as claimed in claim 4 is characterized in that: The Flink cluster receives the target data source table based on the data processing task information, processes the data source table based on a predetermined processing mechanism, and writes the corresponding data into the data destination table, including: The data receiving operator consumes raw data in real time or in batches from the target data source table according to the configuration parameters of the SQL, and forwards the raw data to the data conversion operator; The data conversion operator parses the original data into a standard output data format according to a predetermined analysis rule, obtains standard output data, and forwards it to the data output operator; The data output operator writes the standard output data into the data destination table according to the configuration parameters of the SQL.

6. The data processing method based on Flink stream-batch integration as claimed in claim 1, characterized in that: The data processing tasks corresponding to the data processing task information include tasks for processing real-time data written into the data lake and offline tasks for data migration.

7. The data processing method based on Flink stream-batch integration as claimed in claim 1, characterized in that: The method also includes displaying in real time in a user interface the process and / or result of writing the corresponding data into the data destination table.

8. A data processing system based on Flink stream-batch integration, characterized in that: include: A query statement module is configured to establish SQL statement construction rules based on data elements specified in the SQL file standard and the logical relationship between the data elements, and use the SQL statement construction rules to construct SQL statements for the pre-performed business to obtain a query statement file; the data elements include source table SQL, target table SQL and data conversion SQL; The task determination module is configured to submit the SQL configuration parameters and the query statement file to the Flink cluster to obtain the corresponding data processing task information; The data processing module is configured so that the Flink cluster receives the target data source table based on the data processing task information, processes the data source table based on a predetermined processing mechanism, and then writes the corresponding data into the data destination table.

9. An electronic device, characterized in that: It includes at least one processor, at least one memory and a data bus; wherein: the processor and the memory communicate with each other through the data bus; the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images