Online data processing method and system, electronic equipment and storage medium

Abstract

The invention relates to an online data processing method based on Flink. The processing method comprises the following steps: selecting icons of a data source, a data processing target and a data processing method in a visual page for connection; generating Flink SQL codes based on preset codes corresponding to different icons; submitting the Flink SQL code to an Flink cluster, and analyzing the Flink SQL code through the Flink cluster to obtain a data source table, a data target table and a corresponding data processing method; and based on the Flink cluster, processing the to-be-processed data of each data source in the data source table through the data target table and the data processing method to obtain a data processing result. According to the scheme, an efficient and flexible online data processing mode is provided, graphical configuration is provided for flow control, the development efficiency and the data processing efficiency are improved, and high throughput rate and low delay can be achieved only through few configurations.

Description

Technical Field

[0001] The present disclosure belongs to the field of data processing technology, and in particular, relates to an online data processing method, system, electronic device, and storage medium based on Flink. Background Art

[0002] In the era of big data, data processing and analysis have become particularly important. The traditional data processing method is batch processing, that is, storing data on disk first and then performing batch calculations. However, with the continuous increase in data volume and the growing demand for data processing, this processing method can no longer meet real-time requirements.

[0003] As data processing becomes more and more complex, technical personnel need to write and debug a large amount of code. In addition, the workload of writing code for data processing methods suitable for different scenarios corresponding to different data sources is very large and takes up a lot of resources. Summary of the invention

[0004] To solve the above problems, the present disclosure provides an online data processing method, system, electronic device and storage medium based on Flink. This solution provides an efficient and flexible online data processing method, provides graphical configuration for process control, improves development efficiency and data processing efficiency, and can achieve high throughput and low latency with very little configuration.

[0005] In order to solve the above technical problems, the first aspect of the present invention proposes an online data processing method based on Flink, the method comprising:

[0006] Select the icon of the data source, the icon of the data processing target, and the icon of the data processing method in the visualization page to connect;

[0007] Generate Flink SQL code based on the preset codes corresponding to different icons through the icons of data sources, data processing targets, data processing methods, and connection relationships;

[0008] Submit the Flink SQL code to a Flink cluster, and parse the Flink SQL code through the Flink cluster to obtain a data source table, a data target table, and a corresponding data processing method;

[0009] Based on the Flink cluster, the data to be processed of each data source in the data source table is processed by the data target table and the data processing method to obtain a data processing result.

[0010] According to a preferred embodiment of the present invention, the step of selecting an icon of a data source, an icon of a data processing target, and an icon of a data processing method in a visualization page for connection; and generating a Flink SQL code through the icon of the data source, the icon of the data processing target, the icon of the data processing method, and the connection relationship based on preset codes corresponding to different icons, includes:

[0011] In the visualization page, according to the drag-and-drop connection instructions for the icon of the data source, the icon of the data processing target, and the icon of the data processing method, the icons are connected to generate a data flow connection diagram;

[0012] Configuring parameters for the icon of the data source, the icon of the data processing target and the icon of the data processing method respectively, and constructing a data processing flow chart;

[0013] Generate Flink SQL code according to the code corresponding to each icon in the data processing flowchart.

[0014] According to a preferred embodiment of the present invention, the processing method further comprises:

[0015] Determine each SQL task corresponding to the Flink SQL code;

[0016] Add parallelism, checkpoints, and backup points to the Flink SQL code corresponding to each of the SQL tasks;

[0017] The processing of the to-be-processed data of each data source in the data source table based on the Flink cluster by using the data target table and the data processing method includes:

[0018] When the Flink cluster executes the SQL task and reaches the backup point, the data in the data source table that has been processed is stored;

[0019] When the Flink cluster executes the SQL task and reaches the checkpoint, it is determined whether the Flink cluster fails. When the Flink cluster fails, data recovery is performed using the processed data stored at the backup point with the closest time interval to the checkpoint.

[0020] According to a preferred embodiment of the present invention, based on the Flink cluster, the data to be processed of each data source in the data source table is processed by the data target table and the data processing method to obtain a data processing result, including:

[0021] Traverse and register each of the acquired data processing methods into the running environment of the Flink cluster;

[0022] Traversing each of the data sources in the data source table, and registering the data to be processed in the data source into the operating environment;

[0023] In the operating environment, the data to be processed in the data source is processed by the data processing method corresponding to the data target table to obtain a data processing result.

[0024] According to a preferred embodiment of the present invention, registering the data to be processed in the data source into the operating environment includes:

[0025] Converting the data type of the to-be-processed data in the data source into a data type recognizable by the operating environment;

[0026] The data to be processed after the data type conversion is registered in the operating environment.

[0027] According to a preferred embodiment of the present invention, the processing method further comprises:

[0028] Acquire the processing status and performance index of the data to be processed;

[0029] Obtaining the execution path and variable status of the code for processing the data to be processed;

[0030] The processing status, performance indicators, execution path and variable status are visualized.

[0031] According to a preferred implementation manner of the present invention, before processing the data to be processed of each data source in the data source table, the processing method further includes:

[0032] According to the events and event times corresponding to the data to be processed in each of the data sources, an association operation is performed on the data to be processed.

[0033] In order to solve the above technical problems, the second aspect of the present invention proposes an online data processing system based on Flink, and the processing system includes:

[0034] A visualization module, used to select an icon of a data source, an icon of a data processing target, and an icon of a data processing method in a visualization page for connection;

[0035] The code generation module is used to generate Flink SQL code based on the preset codes corresponding to different icons, using the icons of data sources, data processing targets, data processing methods, and connection relationships;

[0036] A code parsing module, used to submit the Flink SQL code to a Flink cluster, and parse the Flink SQL code through the Flink cluster to obtain a data source table, a data target table, and a corresponding data processing method;

[0037] The data processing module is used to process the to-be-processed data of each data source in the data source table based on the Flink cluster through the data target table and the data processing method to obtain a data processing result.

[0038] In order to solve the above technical problem, the third aspect of the present invention provides an electronic device, comprising:

[0039] Processor; and

[0040] A memory storing computer executable instructions, wherein when the computer executable instructions are executed, the processor executes the method described in any one of the above embodiments.

[0041] In order to solve the above technical problems, the fourth aspect of the present invention proposes a computer storage medium, wherein the computer storage medium stores one or more programs, and when the one or more programs are executed by a processor, the method described in any one of the above embodiments is implemented.

[0042] Compared with the prior art, the present disclosure has the following advantages: the present disclosure provides selectable data sources, data processing targets, and data processing method icons through a visualization page for graphical configuration to achieve process control, generates Flink SQL code based on the constructed graphical process, parses the Flink SQL code through the Flink cluster to generate a data source table, a data target table, and a corresponding data processing method, and processes the data to be processed of each data source in the data source table through the data target table and the data processing method to generate the final data processing result. In practical applications, it can provide an efficient and flexible online data processing method based on Flink SQL, provide graphical configuration for process control, reduce the complexity of developers, improve development efficiency and data processing efficiency, and only require very little configuration to achieve high throughput and low latency, which has broad application prospects.

[0043] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by implementing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A schematic diagram of a flow chart of an online data processing method based on Flink according to an embodiment of the present disclosure is shown;

[0046] Figure 2 A schematic diagram of a method flow for processing data to be processed according to an embodiment of the present disclosure is shown;

[0047] Figure 3 A second flow chart of an online data processing method based on Flink according to an embodiment of the present disclosure is shown;

[0048] Figure 4 A schematic diagram of a checking method flow of a data processing method according to an embodiment of the present disclosure is shown;

[0049] Figure 5 The third flowchart of an online data processing method based on Flink according to an embodiment of the present disclosure is shown;

[0050] Figure 6 A block diagram of an online data processing system based on Flink according to an embodiment of the present disclosure is shown;

[0051] Figure 7 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0053] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0054] See also Figure 1 , Figure 1 This is a flow chart of an online data processing method based on Flink provided by the present invention, as shown in FIG. Figure 1 As shown, the method includes:

[0055] S11. Select the icon of the data source, the icon of the data processing target, and the icon of the data processing method in the visualization page for connection.

[0056] In this embodiment, the visualization page can be the main page of the developed application, or it can be a web page, or other display page. The web page can be convenient for users to access remotely and can reduce the occupation of local resources. After the visualization page is updated on the server side, users can also use the latest version of the service in real time.

[0057] In this solution, data source connection: users can connect to various data sources such as databases, CSV files, APIs, etc. through a graphical interface. After connecting, users can extract data from these data sources.

[0058] Data transformation: Users can define data transformation processes in a graphical interface, such as filtering, sorting, merging, grouping, etc. These operations are usually completed by dragging and dropping elements or configuring simple options.

[0059] Data processing: Users can apply various data processing operations, such as data cleaning, filling missing values, data format conversion, etc. These operations can be completed by simply dragging and dropping.

[0060] Data analysis: Graphical interfaces usually also provide various data analysis tools, such as chart generation, statistical calculations, cluster analysis, etc. Users can generate visual charts and reports through simple operations.

[0061] Workflow design: The graphical drag-and-drop environment allows users to create complex data processing workflows. Users can connect multiple steps to form a complete data processing flow.

[0062] In this embodiment, the user can drag icons of multiple data sources and configure address parameters or identifiers for the icons of the data sources respectively to correspond the icons to different data sources.

[0063] In this embodiment, the icon of the data processing method can be an icon corresponding to the above-mentioned operations such as data cleaning, filling missing values, data format conversion, etc., or it can be an icon corresponding to data analysis such as chart generation, statistical calculation, cluster analysis, etc. Of course, it can also be an icon corresponding to operations such as filtering, sorting, merging, and grouping.

[0064] In this embodiment, the icon of the data processing target is to match the data processing method with the data in the data source, so that when the code is subsequently built, the data and the data processing method are bound to generate data processing results. This method allows users to extract, convert and load data from the data source through a visual interface and drag-and-drop operations, and perform data cleaning, conversion, processing and analysis without having to have a deep understanding of programming languages ​​or complex data processing tools.

[0065] S12. Based on the preset codes corresponding to different icons, generate Flink SQL code through the icon of the data source, the icon of the data processing target, the icon of the data processing method, and the connection relationship.

[0066] In this embodiment, this solution generates Flink SQL by dragging and dropping on the page, aiming to allow users to easily define and manage data processing logic without writing complex code. This method is particularly suitable for users who are not familiar with programming but need to process data, such as business analysts, data scientists, and domain experts.

[0067] In this embodiment, drag-and-drop flink development is a method to complete data processing and analysis tasks without writing a lot of code. This method allows users to extract, transform and load data from data sources through a visual interface and drag-and-drop operations, and perform data cleaning, conversion, processing and analysis without having to have a deep understanding of programming languages ​​or complex data processing tools.

[0068] Automation and Scheduling: Some tools also support automation and scheduling features, allowing users to run data processing tasks regularly to ensure that data is continuously updated and processed.

[0069] No programming experience required: One of the main advantages of this approach is that no in-depth programming experience is required. Users can complete tasks simply by operating through an intuitive interface without having to write complex code.

[0070] Visual feedback: Graphical interfaces usually provide real-time visual feedback to help users understand data processing steps and results.

[0071] This graphical drag-and-drop data development method is suitable for users at all levels, from business analysts to data engineers. It can speed up data processing and analysis, lower the threshold for learning and implementation, and provide an intuitive way to explore and understand data. Different tools and platforms may differ in implementation and functionality, and users can choose the right tool according to their needs.

[0072] In this embodiment, code templates can be constructed as preset codes according to different icons. For example, the data source is generally a database or a data list. It only needs to construct the code to obtain data from the database or the data list as the preset code, which is equivalent to generating different SDK packages for different functions, and then defining the API interface requirements of each SDK package. It only needs to pass the data that meets the API interface requirements into the SDK package to complete the data processing and return the data processing results. SDK (Software Development Kit), that is, software development kit, is a set of development tools provided by software developers, usually including the code library, integrated development environment (IDE), compiler, debugger, documentation, etc. required for development. Application Programming Interface (Application Programming Interface, referred to as: API), is a set of pre-defined functions, the purpose of which is to provide applications and developers with the ability to access a set of routines based on a certain software or hardware without accessing the source code or understanding the details of the internal working mechanism.

[0073] In this embodiment, the connection relationship between different icons is to connect the output of one operation to the input of another operation, thereby constructing a complete data processing flow.

[0074] S13. Submit the Flink SQL code to the Flink cluster, and parse the Flink SQL code through the Flink cluster to obtain the data source table, data target table, and corresponding data processing method.

[0075] In this embodiment, the Flink cluster uses apache calcite to parse SQL statements, parses SQL statements through SQL Parser, determines the type of SQL statements based on regular expressions, parses SQL statements into AST syntax trees SqlNode, and verifies whether SQL statements comply with the specifications in combination with metadata; and obtains a set of UDF functions, data source tables, and data target tables by further parsing the parsed syntax tree SqlNode.

[0076] In this embodiment, Apache Calcite is a dynamic data management framework that provides typical database management functions such as SQL parsing, SQL verification, SQL query optimization, SQL generation, and data connection query.

[0077] In this embodiment, SQL Parser, that is, SQL parser, is a tool or component that can convert SQL statements into an executable plan or some intermediate representation (such as an abstract syntax tree AST).

[0078] In this embodiment, parsing the SQL statement into a SqlNode of an abstract syntax tree (AST) is a key step in the SQL parsing process. SqlNode is a node in an abstract syntax tree (AST) used to represent a parsed SQL statement in a database management system (DBMS).

[0079] In this embodiment, UDF (User-Defined Function) is a function that allows users to create custom logic in a database management system (DBMS). Through UDF, users can expand the functions of the database and implement operations that cannot be completed by some built-in functions of the database. UDF is usually used in scenarios such as data processing, data conversion, data cleaning, and business logic implementation. The UDF function is the code function corresponding to the data processing method in this solution.

[0080] S14. Based on the Flink cluster, the data to be processed of each data source in the data source table is processed through the data target table and the data processing method to obtain the data processing result.

[0081] In this embodiment, the data stream is processed in real time by the Flink cluster, which can ensure the consistency and reliability of the data. It can process the data stream generated in real time and perform batch analysis on static data sets.

[0082] In this embodiment, this solution can also support event time processing, can process out-of-order events in the data stream, and perform window calculations based on event time. It supports operations such as connection and union between stream data and batch data. Specifically, the data to be processed can be associated according to the events and event times corresponding to the data to be processed in each data source. After the data to be processed is associated, in the subsequent data processing or data analysis process, the unprocessed data to be processed can be further analyzed in combination with the processed data to be processed, so that the data between different data sources can be effectively associated and analyzed.

[0083] In this embodiment, if Figure 2 As shown, the process of processing the data to be processed includes:

[0084] S21. Traverse each acquired data processing method and register it in the running environment of the Flink cluster.

[0085] In this embodiment, specifically, the obtained udf function set is traversed, and the class object is dynamically loaded and registered to the flink table environment through the class.forName method.

[0086] In this example, Class.forName(String className) is a static method that is used to dynamically load a class and return the Class object of the class. This method is usually used to load classes at runtime, not at compile time. This is part of the Reflection API, which allows programs to inspect and manipulate classes, interfaces, fields, and methods at runtime.

[0087] In this embodiment, Flink's Table Environment is the core concept of Table API and SQL program, which provides a context environment for creating, executing, and managing tables and SQL statements.

[0088] In this embodiment, in addition, Scalar Functions can also be used to perform data cleaning, data verification, data deduplication, data normalization, data comparison and data identification on the fields. Scalar Functions play an important role in database management systems. They are used to process and transform data.

[0089] S22. Traverse each data source in the data source table, and register the data to be processed in the data source into the operating environment.

[0090] In this embodiment, specifically, the obtained data source table set is traversed, and then converted into a type recognizable by flink Table according to the parsed field field and type type, and then registered with flink TableEnvironment. Flink Table is an API provided by Apache Flink for processing structured data, which allows users to query, process and analyze data streams and static data sets in the form of tables. Specifically, the data type of the data to be processed in the data source is converted into a data type recognizable by the operating environment; the data to be processed after the data type conversion is registered with the operating environment.

[0091] S23. In the operating environment, the data to be processed in the data source is processed by the data processing method corresponding to the data target table to obtain a data processing result.

[0092] In this embodiment, the FlinkTable result set is obtained by executing the data processing method through the flink Table Environment.

[0093] In this embodiment, the Flink Table result set can also be converted into a FlinkDataStream through the flink Table Environment. Flink DataStream is the core abstraction for processing streaming data in Apache Flink. DataStream: It is the basic abstraction of streaming data in Flink, representing an unbounded data stream, which can be real-time data or event-driven data. DataStream can be a finite stream (bounded) or an infinite stream (unbounded), but the API used to process them is the same.

[0094] The present disclosure provides selectable data sources, data processing targets, and data processing method icons through a visualization page for graphical configuration to achieve process control, generates Flink SQL code based on the constructed graphical process, parses the Flink SQL code through the Flink cluster to generate a data source table, a data target table, and a corresponding data processing method, and processes the to-be-processed data of each data source in the data source table through the data target table and the data processing method to generate the final data processing result. In practical applications, it can provide an efficient and flexible online data development method based on Flink SQL, provide a drag-and-drop graphical configuration for process control, reduce the complexity of developers, improve development efficiency, and only require very little configuration to achieve high throughput and low latency, which has a wide range of application prospects.

[0095] See also Figure 3 , Figure 3 The second is a flow chart of an online data processing method based on Flink provided by the present invention. Figure 3 As shown, S11 and S12 specifically include the following steps:

[0096] S31. In the visualization page, according to the drag-and-drop connection instructions for the icon of the data source, the icon of the data processing target and the icon of the data processing method, the icons are connected to generate a data flow connection diagram.

[0097] In this embodiment, connecting to a data source: a user can select data from various data sources (such as databases, files, streaming data, etc.) through a drag-and-drop operation and connect it to the workspace.

[0098] Select an operation: After the data source is connected, users can select the required operation from the predefined operation list, such as filtering, aggregation, and joining.

[0099] Drag-and-drop operations: Users can build data processing logic by dragging and dropping operations into the data flow. For example, you can drag and drop a "filter" operation to filter out data rows with specific conditions.

[0100] Data flow connection: Users can connect the output of one operation to the input of another operation through drag-and-drop operations to build a complete data processing flow.

[0101] S32, respectively configuring parameters for the icon of the data source, the icon of the data processing target, and the icon of the data processing method, and constructing a data processing flow chart.

[0102] In this embodiment, configuration parameters: For each operation, the user can configure the required parameters and conditions through the GUI interface. This may include filter conditions, aggregation functions, connection keys, etc.

[0103] S33. Generate Flink SQL code according to the code corresponding to each icon in the data processing flow chart.

[0104] In this embodiment, Flink SQL is generated: After completing the connection and configuration of the operation, the system can automatically generate corresponding Flink SQL query statements, which will implement the user-defined data processing logic.

[0105] Debugging and optimization: Users can debug and optimize the generated Flink SQL queries to ensure that the data processing logic is correct and has good performance.

[0106] In this example, dragging and dropping to generate Flink SQL can speed up the development process of data processing tasks and reduce coding work and error rates. It provides a friendly interface for users who want to focus on business logic rather than programming details, allowing them to take advantage of Flink's powerful data processing capabilities without having to deeply understand the underlying programming languages ​​and frameworks. This approach also helps to reduce the learning curve of data processing, allowing more people to participate in complex data processing tasks.

[0107] In this embodiment, after generating the Flink SQL code, you can first debug the Flink SQL code, and then use it directly after debugging. Specifically, the steps can be divided into:

[0108] Check the configuration of your data flow: Make sure that the parameters, conditions, and connections for each step are set correctly.

[0109] Run single-step debugging: Some tools allow you to run your data flow in single-step mode, stepping through the output of each step.

[0110] View intermediate results: During the data flow, you can view the intermediate results to ensure that the data is processed correctly at each step.

[0111] Use logging and error handling: If your data processing tool supports logging and error handling features, make sure to enable them. This way, you can check the logs for errors, warnings, or unusual situations.

[0112] View a preview of your data: At the appropriate steps in the data processing tools, you can view a preview of your data to ensure that the processing and transformation of your data is as expected.

[0113] Run the complete flow: When you think your data processing flow is debugged, you can run the complete data flow to process the entire dataset.

[0114] Review output and results: Once a data flow has finished running, you can review the generated output data, reports, or visualizations to ensure that they are as expected.

[0115] Feedback and optimization: If you find problems or need further optimization, you can return to the graphical interface, adjust the data processing flow, and then rerun and debug.

[0116] Save and Schedule: If you are satisfied with your data processing flow, you can save the configuration and set up a schedule to automatically run the flow on a regular basis in the future.

[0117] In summary, online debugging of data development enables you to build, debug, and optimize data processing flows through a visual interface and interactive operations without having to have a deep understanding of programming. The interface and steps of each tool may be different, but they generally follow the general steps above.

[0118] like Figure 4 As shown, in this embodiment, this solution also provides a method for checking a data processing method, including:

[0119] S41. Determine each SQL task corresponding to the Flink SQL code.

[0120] In this embodiment, each SQL task in the Flink SQL code can be determined through the data target table.

[0121] S42. Add parallelism, checkpoints, and backup points to the Flink SQL code corresponding to each SQL task.

[0122] In this embodiment, the parallelism, checkpoints and backup points of the SQL task are defined in the written SQL statement; further, the code for checking the execution status of the SQL task, executing statistics and canceling the SQL task can be added. Specifically, for example, a code function can be added at the corresponding position, and different functions can be implemented in the code function, so that the corresponding code can be executed when the code reaches the corresponding point when it is running.

[0123] S43. When the Flink cluster reaches the backup point after executing the SQL task, the data in the data source table that has been processed is stored.

[0124] S44. When the Flink cluster reaches a checkpoint while executing the SQL task, determine whether the Flink cluster fails. When the Flink cluster fails, restore the data using the processed data stored in the backup point closest to the checkpoint time interval.

[0125] In this embodiment, the data that has been processed and stored in any backup point selected from a plurality of backup points within a preset time interval with the checkpoint may also be restored.

[0126] In this embodiment, this solution ensures that the streaming data processing application can be restored from the previously saved state when a failure occurs by setting backup points and checkpoints to ensure the consistency and fault tolerance of data processing. During the execution process, its state can be saved to persistent storage regularly so that it can be restored when a failure occurs. It ensures the data consistency and reliability of the application when a failure occurs. By regularly saving state snapshots, the Flink cluster can effectively handle the failure of the task manager, the failure of the coordinator, and other unexpected situations, thereby ensuring that the application can run continuously and stably.

[0127] See also Figure 5 , Figure 5 The third is a flow chart of an online data processing method based on Flink provided by the present invention. Figure 5 As shown, the data processing method also includes the following steps:

[0128] S51. Obtain the processing status and performance index of the data to be processed.

[0129] In this embodiment, online debugging and monitoring: The present invention provides an online debugging and monitoring interface, and developers can view the processing status, performance indicators and other information of the data flow in real time, so as to discover and solve problems in time.

[0130] Online debugging and monitoring refers to a series of methods and tools for real-time tracking, analysis and debugging of real-time systems or applications while they are running. These methods and tools can help developers identify problems, monitor performance, locate errors and take timely measures to ensure the stability, reliability and efficiency of the system.

[0131] S52: Obtain the execution path and variable status of the code for processing the data to be processed.

[0132] In this embodiment, online debugging is a method of detecting and fixing problems while the application is running. It involves using various tools and techniques to track code execution, find errors and make immediate repairs.

[0133] Real-time logging: Insert appropriate log statements in your application so that you can monitor the code execution path and variable status in real time. This helps identify errors and abnormal conditions in the code.

[0134] Remote debugging: Using remote debugging tools, developers can connect to a running application to inspect and modify code in real time. This is particularly useful when dealing with hard-to-reproduce bugs.

[0135] Conditional breakpoints: Set conditional breakpoints in your code to pause code execution when a certain condition is met so you can inspect the state of your code.

[0136] S53, visually displaying the processing status, performance indicators, execution path and variable status.

[0137] In this embodiment, real-time visualization: Using visualization tools, developers can view the data flow, interaction, and execution of the application in real time, which helps to understand the behavior of the code at runtime.

[0138] Automatic optimization: This solution also implements the automatic optimization function. Based on the developer's query and data flow design, the system can automatically infer the best execution plan and resource allocation to improve data processing performance. Automatic inference of the best execution plan and resource allocation can optimize data flow and resource allocation through the data processing efficiency corresponding to the different data processing methods of each terminal in the Flink cluster.

[0139] See also Figure 6 , Figure 6 This is a block diagram of an online data processing system based on Flink provided by the present invention, such as Figure 6 As shown, the processing system includes: a visualization module 11, a code generation module 12, a code parsing module 13 and a data processing module 14.

[0140] In this embodiment, the visualization module 11 is used to select the icon of the data source, the icon of the data processing target and the icon of the data processing method in the visualization page for connection.

[0141] In this embodiment, the code generation module 12 is used to generate Flink SQL code based on preset codes corresponding to different icons through icons of data sources, icons of data processing targets, icons of data processing methods, and connection relationships.

[0142] In this embodiment, the code parsing module 13 is used to submit the Flink SQL code to the Flink cluster, and parse the Flink SQL code through the Flink cluster to obtain a data source table, a data target table, and a corresponding data processing method.

[0143] In this embodiment, the data processing module 14 is used to process the to-be-processed data of each data source in the data source table through the data target table and the data processing method based on the Flink cluster to obtain the data processing result.

[0144] In this embodiment, the visualization module 11 is specifically used to connect the icons of the data source, the data processing target and the data processing method in the visualization page according to the drag-and-drop connection instructions to generate a data flow connection diagram; configure parameters for the icon of the data source, the icon of the data processing target and the icon of the data processing method respectively to construct a data processing flow chart.

[0145] In this embodiment, the code generation module 12 is specifically used to generate Flink SQL code according to the code corresponding to each icon in the data processing flow chart.

[0146] In this embodiment, the code generation module 12 is further used to determine each SQL task corresponding to the Flink SQL code; and add parallelism, checkpoints, and backup points to the Flink SQL code corresponding to each SQL task.

[0147] In this embodiment, the data processing module 14 is specifically used to store the data in the data source table that has been processed when the Flink cluster executes the SQL task and reaches the backup point; when the Flink cluster executes the SQL task and reaches the checkpoint, determine whether the Flink cluster fails, and when the Flink cluster fails, restore the data through the backup point with the closest time interval to the checkpoint.

[0148] In this embodiment, the data processing module 14 is specifically used to traverse and register the acquired data processing methods into the running environment of the Flink cluster; traverse each data source in the data source table, and register the data to be processed in the data source into the running environment; in the running environment, process the data to be processed in the data source by the data processing method corresponding to the data target table to obtain the data processing result.

[0149] In this embodiment, the data processing module 14 is specifically used to convert the data type of the to-be-processed data in the data source into a data type recognizable by the operating environment; and register the to-be-processed data after the data type conversion into the operating environment.

[0150] In this embodiment, the processing system also includes: an online monitoring module for obtaining the processing status and performance indicators of the data to be processed; obtaining the execution path and variable status of the code for processing the data to be processed; and visually displaying the processing status, performance indicators, execution path and variable status.

[0151] In this embodiment, the data processing module 14 is further used to perform association operations on the data to be processed according to the events and event times corresponding to the data to be processed in each data source.

[0152] like Figure 7 As shown, an embodiment of the present invention provides an electronic device, including a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140;

[0153] Memory 1130, used for storing computer programs;

[0154] The processor 1110 is used to implement any of the above processing methods when executing the program stored in the memory 1130.

[0155] In the electronic device provided by the embodiment of the present invention, the processor 1110 selects the icon of the data source, the icon of the data processing target and the icon of the data processing method in the visualization page for connection by executing the program stored in the memory 1130; based on the preset codes corresponding to different icons, generates Flink SQL code through the icon of the data source, the icon of the data processing target, the icon of the data processing method and the connection relationship; submits the Flink SQL code to the Flink cluster, and parses the Flink SQL code through the Flink cluster to obtain the data source table, the data target table and the corresponding data processing method; based on the Flink cluster, processes the data to be processed of each data source in the data source table through the data target table and the data processing method to obtain the data processing result.

[0156] The communication bus 1140 mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0157] The communication interface 1120 is used for communication between the above electronic device and other devices.

[0158] The memory 1130 may include a random access memory 1130 (RAM), or may include a non-volatile memory 1130 (non-volatile memory), such as at least one disk memory 1130. Optionally, the memory 1130 may also be at least one storage device located away from the processor 1110.

[0159] The above-mentioned processor 1110 can be a general-purpose processor 1110, including a central processing unit 1110 (CPU), a network processor 1110 (NP), etc.; it can also be a digital signal processor 1110 (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0160] An embodiment of the present invention provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors 1110 to implement the processing method of any of the above embodiments.

[0161] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidStateDisk (SSD)), etc.

[0162] Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A Flink-based online data processing method, characterized in that: The processing method comprises: Select the icon of the data source, the icon of the data processing target, and the icon of the data processing method in the visualization page to connect; Generate Flink SQL code based on the preset codes corresponding to different icons through the icons of data sources, data processing targets, data processing methods, and connection relationships; Submit the Flink SQL code to a Flink cluster, and parse the Flink SQL code through the Flink cluster to obtain a data source table, a data target table, and a corresponding data processing method; Based on the Flink cluster, the data to be processed of each data source in the data source table is processed by the data target table and the data processing method to obtain a data processing result.

2. The processing method according to claim 1, characterized in that: The step of selecting an icon of a data source, an icon of a data processing target, and an icon of a data processing method in a visualization page for connection; and generating FlinkSQL code through the icon of the data source, the icon of the data processing target, the icon of the data processing method, and the connection relationship based on preset codes corresponding to different icons, includes: In the visualization page, according to the drag-and-drop connection instructions for the icon of the data source, the icon of the data processing target, and the icon of the data processing method, the icons are connected to generate a data flow connection diagram; Configuring parameters for the icon of the data source, the icon of the data processing target and the icon of the data processing method respectively, and constructing a data processing flow chart; Generate Flink SQL code according to the code corresponding to each icon in the data processing flowchart.

3. The processing method according to claim 2, characterized in that: The processing method also includes: Determine each SQL task corresponding to the Flink SQL code; Add parallelism, checkpoints, and backup points to the Flink SQL code corresponding to each of the SQL tasks; The processing of the to-be-processed data of each data source in the data source table based on the Flink cluster by using the data target table and the data processing method includes: When the Flink cluster executes the SQL task and reaches the backup point, the data in the data source table that has been processed is stored; When the Flink cluster executes the SQL task and reaches the checkpoint, it is determined whether the Flink cluster fails. When the Flink cluster fails, data recovery is performed using the processed data stored at the backup point with the closest time interval to the checkpoint.

4. The processing method according to claim 1, characterized in that: Based on the Flink cluster, the data to be processed of each data source in the data source table is processed by the data target table and the data processing method to obtain a data processing result, including: Traverse and register each of the acquired data processing methods into the running environment of the Flink cluster; Traversing each of the data sources in the data source table, and registering the data to be processed in the data source into the operating environment; In the operating environment, the data to be processed in the data source is processed by the data processing method corresponding to the data target table to obtain a data processing result.

5. The processing method according to claim 4, characterized in that: Registering the data to be processed in the data source into the operating environment includes: Converting the data type of the to-be-processed data in the data source into a data type recognizable by the operating environment; The data to be processed after the data type conversion is registered in the operating environment.

6. The processing method according to claim 1, characterized in that: The processing method also includes: Acquire the processing status and performance index of the data to be processed; Obtaining the execution path and variable status of the code for processing the data to be processed; The processing status, performance indicators, execution path and variable status are visualized.

7. The processing method according to any one of claims 1 to 6, characterized in that: Before processing the to-be-processed data of each data source in the data source table, the processing method further includes: According to the events and event times corresponding to the data to be processed in each of the data sources, an association operation is performed on the data to be processed.

8. An online data processing system based on Flink, characterized in that: The processing system comprises: A visualization module, used to select an icon of a data source, an icon of a data processing target, and an icon of a data processing method in a visualization page for connection; The code generation module is used to generate Flink SQL code based on the preset codes corresponding to different icons, using the icons of data sources, data processing targets, data processing methods, and connection relationships; A code parsing module, used to submit the Flink SQL code to a Flink cluster, and parse the Flink SQL code through the Flink cluster to obtain a data source table, a data target table, and a corresponding data processing method; The data processing module is used to process the to-be-processed data of each data source in the data source table based on the Flink cluster through the data target table and the data processing method to obtain a data processing result.

9. An electronic device, characterized in that: include: processor; as well as A memory storing computer executable instructions which, when executed, cause the processor to perform the method according to any one of claims 1-7.

10. A computer storage medium, characterized in that: in, The computer storage medium stores one or more programs, and when the one or more programs are executed by a processor, the method of any one of claims 1 to 7 is implemented.

Images