Data processing method, device, electronic device and storage medium based on Flink platform
By presetting multi-layer data layers and plug-in mechanisms on the Flink platform, the problem of insufficient scalability of the Flink platform is solved, multi-level flexible configuration and customization of data processing tasks is realized, and data processing capabilities and compatibility are improved.
Patent Information
- Application Number
- CN202210198267.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-01
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-03-01
AI Technical Summary
The existing Flink big data processing engine only retains a single expansion mode, cannot fully utilize the advantages of the Flink platform, and lacks multi-level data processing capabilities.
By presetting multi-layer data layers on the Flink platform, including interfaces of different data types, calling corresponding data processing operators, generating data processing tasks, and using plug-in mechanisms to achieve data type compatibility between different data layers, supporting multiple Customization of hierarchical configuration and data processing tasks.
It realizes the multi-level flexible configuration of the Flink platform and the easy construction of data processing tasks, improves data processing capabilities, meets multiple data processing needs, and solves compatibility problems between different data layers through data conversion plug-ins.
Smart Images

Figure CN114564480B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of big data technology, and specifically, to a data processing method, device, electronic device, and storage medium based on a Flink platform. Background Art
[0002] The Flink big data platform can use data processing operators for data processing. However, the current big data processing engines only retain a single expansion mode after packaging Flink, which cannot fully utilize all the advantages of Flink. Summary of the invention
[0003] The purpose of the embodiments of the present application is to provide a data processing method, device, electronic device and storage medium based on a Flink platform, which can be expanded at different levels, develop operators at different levels, and improve the data processing capabilities of the Flink platform.
[0004] In a first aspect, an embodiment of the present application provides a data processing method based on a Flink platform, including:
[0005] Preset a multi-layer data layer according to the data structure of the Flink platform, wherein the multi-layer data layer includes interfaces based on different data types;
[0006] Calling corresponding data processing operators through interfaces of different data types of the multiple data layers;
[0007] Generate a data processing task according to the data processing operator;
[0008] The data processing task is sent to the Flink platform for calculation.
[0009] In the above implementation process, the data structure of the Flink platform is analyzed, and a multi-layer data layer is set according to the data structure of the Flink platform. The multi-layer data layer includes data interfaces based on different data types. The data processing operators implement the interfaces of different data layers, making the data operators plug-in-based. Based on the above implementation, multi-level flexible configuration can be achieved based on Flink, and data processing tasks can be easily constructed.
[0010] Furthermore, the multi-layer data layer includes:
[0011] Original data layer, TableAPI layer, FlinkSQL layer, and dataset layer;
[0012] Wherein, the data type of the interface of the original data layer is a basic data structure;
[0013] The data type of the interface of the TableAPI layer is a data structure based on a two-dimensional table;
[0014] The data type of the interface of the FlinkSQL layer is a data structure based on a database table, and the FlinkSQL layer is configured with an execution function based on querying structured query language variables;
[0015] The data type of the interface of the dataset layer is based on the data structure of the Flink dataset.
[0016] In the above implementation process, the data type of the interface of the original data layer is a basic data structure. Based on the basic data structure, the data processing operator can process data of multiple basic data types. The data type of the interface of the TableAPI layer is a data structure based on a two-dimensional table. The data processing operator that implements the interface of the TableAPI layer can process data based on a two-dimensional table data structure. The FlinkSQL layer is configured with an execution function based on querying structured query language variables. Users can directly configure SQL statements so that the data operator that implements the FlinkSQL layer has structured query capabilities. The data type of the interface of the data set layer is a data structure interface based on the Flink data set. The data processing operator that implements the interface can implement an API based on the data set. Based on the above implementation, by using the data operators corresponding to different data layers, various data processing requirements can be met.
[0017] Furthermore, the step of generating a data processing task according to the data processing operator includes:
[0018] Determine a connection order between the called data processing operators, and generate a data processing task with the data processing operators as nodes according to the connection order, so that the Flink platform performs operations according to the connection order of the data processing operators in the data processing task.
[0019] In the above implementation process, data processing tasks are generated according to the connection order, and the Flink platform performs operations according to the connection order of the data processing operators in the data processing tasks. Based on the above implementation method, data processing tasks can be customized.
[0020] Further, after the step of generating data processing tasks with the data processing operators as nodes according to the connection sequence, the method further includes:
[0021] A first plug-in calling module is added between the data processing operator corresponding to the original data layer and the data processing operator corresponding to the TableAPI layer according to the connection direction of the data processing operator corresponding to the original data layer and the data processing operator corresponding to the TableAPI layer, so as to call the data conversion plug-in of the Flink platform through the first plug-in calling module to convert the output data of the data processing operator corresponding to the original data layer or the output data of the data processing operator corresponding to the TableAPI layer; or
[0022] According to the connection direction of the data processing operator corresponding to the original data layer and the data processing operator corresponding to the FlinkSQL layer, a second plug-in calling module is added between the data processing operator corresponding to the original data layer and the data processing operator corresponding to the FlinkSQL layer, so as to call the data conversion plug-in of the Flink platform through the second plug-in calling module to convert the output data of the data processing operator corresponding to the original data layer or the output data of the data processing operator corresponding to the FlinkSQL layer.
[0023] In the above implementation process, due to the differences in data structures between the original data layer and the TableAPI layer and FlinkSQL layer, a calling module needs to be added between the original data layer and the TableAPI layer and FlinkSQL layer. The calling module calls the data conversion plug-in of the Flink platform to achieve data type compatibility between different data layers.
[0024] Furthermore, before the step of generating a data processing task according to the data processing operator, the method further includes: receiving configuration parameters, and configuring the data processing operator according to the configuration parameters.
[0025] In the above implementation process, by receiving configuration parameters, the data processing operator can be configured to improve the data processing capability of the data processing operator.
[0026] Furthermore, before the step of generating a data processing task according to the data processing operator, the step further includes:
[0027] The data operator corresponding to the original data layer is packaged using a wrapper so that the Flink platform can call the packaged data processing operator, wherein the packaged data processing operator has at least one function of synchronization, asynchrony and aggregation.
[0028] In the above implementation process, since the data processing operator corresponding to the original data layer is based on the basic data structure, in order to enable the Flink platform data layer to call the data processing operator, it is necessary to make the data processing operator corresponding to the original data layer meet the interface calling requirements of the Flink platform. The data processing operator can be packaged based on the wrapper, so that the data processing operator corresponding to the packaged original data layer meets the interface calling requirements of the Flink platform.
[0029] Furthermore, the interface of the original data layer implements the consumer interface in the message queue interface group.
[0030] In the above implementation process, since the Flink platform has its own collection and call function, in order to avoid the Flink platform calling the data processing operator corresponding to the original data layer, the interface of the original data layer implements the consumer interface in the message queue interface group, so that the data processing operator corresponding to the original data layer and the Flink platform are decoupled.
[0031] In a second aspect, an embodiment of the present application provides a data processing device based on a Flink platform, including:
[0032] A data layer setting module is used to preset a multi-layer data layer according to the data structure of the Flink platform, wherein the multi-layer data layer includes interfaces based on different data types;
[0033] A data processing operator acquisition module, used to call corresponding data processing operators through interfaces of different data types of the multiple data layers;
[0034] A data task generation module, used for generating a data processing task from the data processing operator;
[0035] The data task sending module is used to send the data processing task to the Flink platform for calculation.
[0036] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of the first aspects when executing the computer program.
[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer executes the method as described in any one of the first aspects.
[0038] Other features and advantages disclosed in the present application will be described in the following description, or some features and advantages can be inferred or determined without doubt from the description, or can be learned by implementing the above-mentioned technology disclosed in the present application.
[0039] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0041] Figure 1 A flowchart of a data processing method based on the Flink platform provided in an embodiment of the present application;
[0042] Figure 2 A schematic diagram of the structure of a data processing device based on the Flink platform provided in an embodiment of the present application;
[0043] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0044] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0045] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0046] Example 1
[0047] See also Figure 1 , the embodiment of the present application provides a data processing method based on the Flink platform, including:
[0048] S1: Preset multiple data layers according to the data structure of the Flink platform. The multiple data layers include interfaces based on different data types.
[0049] S2: Call corresponding data processing operators through interfaces of different data types in multiple data layers;
[0050] In object-oriented languages, data processing operators inherit interfaces of different data types according to the standards corresponding to the interfaces of different data types in multiple data layers, thereby calling corresponding data processing operators through the interfaces of different data types in multiple data layers.
[0051] In a possible implementation, a plug-in framework is pre-generated, and the interface standard of the plug-in framework is the same as the standard corresponding to the interface of different data types of the multi-layer data layer. Through the driving framework, multiple data processing operators can be called simultaneously to improve data processing capabilities.
[0052] S3: Generate data processing tasks according to data processing operators;
[0053] S4: Send the data processing task to the Flink platform for calculation.
[0054] In the above embodiment, each data layer may include interfaces of multiple data types, and the division of the multiple data layers is based on the processing time and speed of the Flink platform for data of different data structures and the data characteristics of common big data.
[0055] Exemplarily, in an object-oriented language, an interface class is a type of interface, based on which objects that perform different functions can be defined and distinguished. The data processing operator in the above implementation is an object corresponding to the class that implements the interface of the corresponding data layer.
[0056] In the above implementation process, the data structure of the Flink platform is analyzed, and a multi-layer data layer is set according to the data structure of the Flink platform. The multi-layer data layer includes data interfaces based on different data types, and the data processing operators implement the interfaces of different data layers, making the data operators plug-in-based. Based on the above implementation, multi-level flexible configuration can be achieved based on the Flink platform, and data processing tasks can be easily constructed.
[0057] In a possible implementation, the multiple data layers further include:
[0058] Original data layer, TableAPI layer, FlinkSQL layer, and dataset layer;
[0059] Among them, the data type of the interface of the original data layer is the basic data structure;
[0060] The data type of the interface of the TableAPI layer is a data structure based on a two-dimensional table;
[0061] The data type of the interface of the FlinkSQL layer is based on the data structure of the database table, and the FlinkSQL layer is configured with execution functions based on querying structured query language variables;
[0062] The data type of the dataset layer interface is based on the Flink dataset data structure.
[0063] For example, the data types of the raw data layer interface include standard data structures such as map, filter, flatmap, aggregate, and join. The TableAPI layer is similar to the FlinkSQL layer, and can operate data in the form of a two-dimensional table. The data type of the data set layer interface is a data structure based on the Flink platform data set, which is a representation of the data flow on the Flink platform.
[0064] It should be noted that the data structure of the dataset layer and the data structures of other data layers can be converted into each other.
[0065] In the above implementation process, the data type of the interface of the original data layer is a basic data structure. Based on the basic data structure, the data processing operator can process data of multiple basic data types. The data type of the interface of the TableAPI layer is a data structure based on a two-dimensional table. The data processing operator that implements the interface of the TableAPI layer can process data based on a two-dimensional table data structure. The FlinkSQL layer is configured with an execution function based on querying structured query language variables. Users can directly configure SQL statements so that the data operator that implements the FlinkSQL layer has structured query capabilities. The data type of the interface of the data set layer is a data structure interface based on the Flink data set. The data processing operator that implements the interface can implement an API based on the data set. Based on the above implementation, by using the data operators corresponding to different data layers, various data processing requirements can be met.
[0066] In a possible implementation, the step of generating a data processing task according to a data processing operator includes:
[0067] Determine the connection order between the called data processing operators, and generate data processing tasks with the data processing operators as nodes according to the connection order, so that the Flink platform performs operations according to the connection order of the data processing operators in the data processing tasks.
[0068] In the above implementation process, data processing tasks are generated according to the connection order, and the Flink platform performs operations according to the connection order of the data processing operators in the data processing tasks. Based on the above implementation method, data processing tasks can be customized.
[0069] In a possible implementation manner, after the step of generating data processing tasks with data processing operators as nodes according to the connection order, the method further includes:
[0070] Add a first plug-in calling module between the data processing operator corresponding to the original data layer and the data processing operator corresponding to the TableAPI layer according to the connection direction of the data processing operator corresponding to the original data layer and the data processing operator corresponding to the TableAPI layer, so as to call the data conversion plug-in of the Flink platform through the first plug-in calling module to convert the output data of the data processing operator corresponding to the original data layer or the output data of the data processing operator corresponding to the TableAPI layer; or
[0071] According to the connection direction of the data processing operator corresponding to the original data layer and the data processing operator corresponding to the FlinkSQL layer, a second plug-in calling module is added between the data processing operator corresponding to the original data layer and the data processing operator corresponding to the FlinkSQL layer, so as to call the data conversion plug-in of the Flink platform through the second plug-in calling module to convert the output data of the data processing operator corresponding to the original data layer or the output data of the data processing operator corresponding to the FlinkSQL layer.
[0072] In the above embodiment, different modules are set to call different data conversion plug-ins of the Flink platform through the execution order between different nodes in the data processing task to convert the format of the output data stream of the node.
[0073] Exemplarily, the data types of the interfaces in the TableAPI layer and the FlinkSQL layer are both based on the data structure of the two-dimensional table. The computing platform can obtain the table column names through the data structure of the two-dimensional table. If the subsequent data processing operator connected to the data processing operator corresponding to the TableAPI layer / the data processing operator corresponding to the FlinkSQL layer is a data processing operator based on the original data layer, since the data type of the interface based on the original data layer is a format without Schema (such as json), it is necessary to set a first calling module between the two data processing operators. The module calls the conversion plug-in of the Flink platform to convert the output data of the data processing operator corresponding to the TableAPI layer and the output data of the data operator corresponding to the FlinkSQL layer into key-value pairs of column name + value.
[0074] The data structure in the data processing operator corresponding to the original data layer may not have a defined schema. In theory, the computing platform can obtain the table column names from the processed data through the data set layer and the data processing operator corresponding to the original data layer, but this is too inefficient and cannot identify fields with null values. Therefore, when the data processing operator corresponding to the original data layer is subsequently connected to the data processing operator corresponding to the TableAPI layer / the data processing operator corresponding to the FlinkSQL layer, a second call module is added after the original data layer. This module calls the data conversion plug-in, which can convert key-value pair data into a two-dimensional table with a schema. After the conversion is implemented, operators at different levels can be combined and used to achieve the effect of flexibly configuring the data processing process.
[0075] In the above embodiment, Schema is a data structure that defines the field names and types contained in the data.
[0076] In the above implementation process, due to the differences in data structures between the original data layer and the TableAPI layer and FlinkSQL layer, a calling module needs to be added between the original data layer and the TableAPI layer and FlinkSQL layer. The calling module calls the data conversion plug-in of the Flink platform to achieve data type compatibility between different data layers.
[0077] In a possible implementation, before the step of generating a data processing task according to the data processing operator, the step further includes: receiving configuration parameters, and configuring the data processing operator according to the configuration parameters.
[0078] Exemplarily, the corresponding configuration parameters of the FlinkSQL layer include SQL statements, field mappings, time fields, etc.
[0079] It is understandable that the data processing operator is provided with a function related to the configuration parameter. In the above implementation process, by receiving the configuration parameter, the data processing operator can be flexibly configured.
[0080] In a possible implementation manner, before the step of generating a data processing task according to the data processing operator, the method further includes:
[0081] The data operator corresponding to the original data layer is packaged using a wrapper so that the Flink platform can call the packaged data processing operator, wherein the packaged data processing operator has at least one of the functions of synchronization, asynchrony and aggregation.
[0082] In the above implementation process, since the data processing operator corresponding to the original data layer is based on the basic data structure, in order to enable the Flink data layer to call the data processing operator, the data processing operator corresponding to the original data layer needs to meet the interface call requirements of the Flink platform. The data processing operator can be packaged based on the wrapper so that the data processing operator corresponding to the original data layer after packaging meets the interface call requirements of Flink. After being packaged, the data processing operator has at least one of the functions of synchronization, asynchrony and aggregation, which can further meet the call request of Flink.
[0083] Furthermore, the interface of the original data layer implements the consumer interface in the message queue interface group.
[0084] In the above implementation process, since the Flink platform has its own collection and call function, in order to avoid the Flink platform calling the data processing operator corresponding to the original data layer, the interface of the original data layer implements the consumer interface in the message queue interface group, so that the data processing operator corresponding to the original data layer and the Flink platform are decoupled.
[0085] For example, if the data operator is written in Java language, the Java Consumer interface is implemented.
[0086] Example 2
[0087] See also Figure 2 , the embodiment of the present application provides a data processing device based on the Flink platform, including:
[0088] Data layer setting module 1, used to preset a multi-layer data layer according to the data structure of the Flink platform, and the multi-layer data layer includes interfaces based on different data types;
[0089] The data processing operator acquisition module 2 is used to call the corresponding data processing operator through the interfaces of different data types of multiple data layers;
[0090] A data task generation module 3 is used to generate data processing tasks from data processing operators;
[0091] The data task sending module 4 is used to send the data processing task to the Flink platform for calculation.
[0092] In the above implementation process, the data structure of the Flink platform is analyzed, and a multi-layer data layer is set according to the data structure of the Flink platform. The multi-layer data layer includes data interfaces based on different data types, and the data processing operators implement the data processing operators of different data layers, making the data operators plug-in-based. Based on the above implementation, multi-level, flexible and easy-to-build data processing tasks can be implemented based on the Flink platform.
[0093] In a possible implementation, the multi-layer data layer includes:
[0094] Original data layer, TableAPI layer, FlinkSQL layer, and dataset layer;
[0095] Among them, the data type of the interface of the original data layer is the basic data structure;
[0096] The data type of the interface of the TableAPI layer is a data structure based on a two-dimensional table;
[0097] The data type of the interface of the FlinkSQL layer is based on the data structure of the database table, and the FlinkSQL layer is configured with execution functions based on querying structured query language variables;
[0098] The data type of the dataset layer interface is based on the Flink dataset data structure.
[0099] In a possible implementation, the data task generation module 3 is also used to determine the connection order between the called data processing operators, where the connection order includes the connection order and connection direction of the data operators; and generate data processing tasks with the data processing operators as nodes according to the connection order, so that the Flink platform performs operations according to the connection direction of the data processing operators in the data processing tasks.
[0100] In a possible implementation, the device also includes: a setting module, which is used to add a first plug-in calling module between the data processing operator corresponding to the original data layer and the data processing operator corresponding to the TableAPI layer according to the connection direction of the data processing operator corresponding to the original data layer and the data processing operator corresponding to the TableAPI layer, so as to call the data conversion plug-in of the Flink platform through the first plug-in calling module to convert the output data of the data processing operator corresponding to the original data layer or the output data of the data processing operator corresponding to the TableAPI layer; or, according to the connection direction of the data processing operator corresponding to the original data layer and the data processing operator corresponding to the FlinkSQL layer, add a second plug-in calling module between the data processing operator corresponding to the original data layer and the data processing operator corresponding to the FlinkSQL layer, so as to call the data conversion plug-in of the Flink platform through the second plug-in calling module to convert the output data of the data processing operator corresponding to the original data layer or the output data of the data processing operator corresponding to the FlinkSQL layer.
[0101] In a possible implementation, the device further includes a configuration module, which is used to receive configuration parameters, receive the configuration parameters, and configure the data processing operator according to the configuration parameters.
[0102] In a possible implementation, the device further includes an adaptation module for packaging the data processing operator corresponding to the original data layer using a wrapper so that the Flink platform can call the packaged data processing operator, wherein the packaged data processing operator has at least one of the functions of synchronization, asynchrony and aggregation.
[0103] In a possible implementation, the interface of the original data layer implements the consumer interface in the message queue interface group.
[0104] Example 3
[0105] This application also provides an electronic device, see Figure 3 , Figure 3 A block diagram of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 31, a communication interface 32, a memory 33, and at least one communication bus 34. The communication bus 34 is used to realize direct connection and communication between these components. The communication interface 32 of the electronic device in the embodiment of the present application is used to communicate signaling or data with other node devices. The processor 31 may be an integrated circuit chip with signal processing capabilities.
[0106] The processor 31 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor 31 can also be any conventional processor, etc.
[0107] The memory 33 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 33 stores computer-readable instructions, and when the computer-readable instructions are executed by the processor 31, the electronic device may execute the various steps involved in the above method embodiment.
[0108] Optionally, the electronic device may further include a storage controller and an input / output unit.
[0109] The memory 33, storage controller, processor 31, peripheral interface, input and output unit components are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses 34. The processor 31 is used to execute executable modules stored in the memory 33, such as software function modules or computer programs included in the electronic device.
[0110] The input and output unit is used to provide users with the task creation and to create a start optional time period or preset execution time for the task to realize the interaction between the user and the server. The input and output unit can be, but is not limited to, a mouse and a keyboard.
[0111] Understandably, Figure 3 The structure shown is for illustration only, and the electronic device may also include Figure 3 More or fewer components as shown, or with Figure 3 Different configurations are shown. Figure 3 Each component shown in the figure can be implemented by hardware, software or a combination thereof.
[0112] An embodiment of the present application further provides a computer-readable storage medium having instructions stored thereon. When the instructions are run on a computer, the computer program is executed by a processor to implement the method described in the method embodiment. To avoid repetition, it will not be described again here.
[0113] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0114] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0115] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0116] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.
[0117] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0118] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
Claims
1. A data processing method based on the Flink platform, It is characterized in that include: Preset a multi-layer data layer according to the data structure of the Flink platform, wherein the multi-layer data layer includes interfaces based on different data types; wherein the multi-layer data layer includes a raw data layer, a TableAPI layer, and a FlinkSQL layer; Calling corresponding data processing operators through interfaces of different data types of the multiple data layers; Determine a connection order between called data processing operators, and generate a data processing task with the data processing operators as nodes according to the connection order, so that the Flink platform performs calculations according to the connection order of the data processing operators in the data processing task; According to the connection direction of the data processing operator corresponding to the original data layer and the data processing operator corresponding to the TableAPI layer, a first plug-in calling module is added between the data processing operator corresponding to the original data layer and the data processing operator corresponding to the TableAPI layer, so as to call the data conversion plug-in of the Flink platform through the first plug-in calling module to convert the output data of the data processing operator corresponding to the original data layer or the output data of the data processing operator corresponding to the TableAPI layer; or According to the connection direction of the data processing operator corresponding to the original data layer and the data processing operator corresponding to the FlinkSQL layer, a second plug-in calling module is added between the data processing operator corresponding to the original data layer and the data processing operator corresponding to the FlinkSQL layer, so as to call the data conversion plug-in of the Flink platform through the second plug-in calling module to convert the output data of the data processing operator corresponding to the original data layer or the output data of the data processing operator corresponding to the FlinkSQL layer; The data processing task is sent to the Flink platform for calculation.
2. According to the data processing method based on the Flink platform according to claim 1, It is characterized in that The multi-layer data layer also includes a data set layer; The data type of the interface of the original data layer is a basic data structure; The data type of the interface of the TableAPI layer is a data structure based on a two-dimensional table; The data type of the interface of the FlinkSQL layer is a data structure based on a database table, and the FlinkSQL layer is configured with an execution function based on querying structured query language variables; The data type of the interface of the dataset layer is based on the data structure of the Flink dataset.
3. According to the data processing method based on the Flink platform of claim 1, It is characterized in that Before the step of determining the connection order between the called data processing operators and generating data processing tasks with the data processing operators as nodes according to the connection order, the method further includes: receiving configuration parameters and configuring the data processing operators according to the configuration parameters.
4. According to the data processing method based on the Flink platform according to claim 2, It is characterized in that Before the step of determining the connection sequence between the called data processing operators and generating data processing tasks with the data processing operators as nodes according to the connection sequence, the step further includes: The data processing operator corresponding to the original data layer is packaged using a wrapper so that the Flink platform can call the packaged data processing operator, wherein the packaged data processing operator has at least one function of synchronization, asynchrony and aggregation.
5. According to the data processing method based on the Flink platform according to claim 2, It is characterized in that The interface of the original data layer implements the consumer interface in the message queue interface group.
6. A data processing device based on the Flink platform, It is characterized in that include: A data layer setting module is used to preset a multi-layer data layer according to the data structure of the Flink platform, wherein the multi-layer data layer includes interfaces based on different data types; wherein the multi-layer data layer includes an original data layer, a TableAPI layer, and a FlinkSQL layer; A data processing operator acquisition module, used to call corresponding data processing operators through interfaces of different data types of the multiple data layers; A data task generation module, used for generating a data processing task from the data processing operator; A data task sending module, used to send the data processing task to the Flink platform for calculation; The data task generation module is further used to determine the connection order between the called data processing operators, the connection order including the connection order and connection direction of the data operators; generate data processing tasks with the data processing operators as nodes according to the connection order, so that the Flink platform performs operations according to the connection direction of the data processing operators in the data processing tasks; Among them, the data processing device also includes: a setting module, which is used to add a first plug-in calling module between the data processing operator corresponding to the original data layer and the data processing operator corresponding to the TableAPI layer according to the connection direction of the data processing operator corresponding to the original data layer and the data processing operator corresponding to the TableAPI layer, so as to call the data conversion plug-in of the Flink platform through the first plug-in calling module to convert the output data of the data processing operator corresponding to the original data layer or the output data of the data processing operator corresponding to the TableAPI layer; or, according to the connection direction of the data processing operator corresponding to the original data layer and the data processing operator corresponding to the FlinkSQL layer, add a second plug-in calling module between the data processing operator corresponding to the original data layer and the data processing operator corresponding to the FlinkSQL layer, so as to call the data conversion plug-in of the Flink platform through the second plug-in calling module to convert the output data of the data processing operator corresponding to the original data layer or the output data of the data processing operator corresponding to the FlinkSQL layer.
7. An electronic device, It is characterized in that include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the data processing method based on the Flink platform as described in any one of claims 1 to 5 are implemented.
8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the data processing method based on the Flink platform according to any one of claims 1 to 5.
