External data query method, device and computer equipment based on Flink

Abstract

The present invention provides a method, device and computer equipment for querying external data based on Flink, and the method includes: expanding the Java interface of the Flink system and adding a parallelism setting option to obtain a first interface and a second interface with the parallelism setting option; modifying the SQL compiler built into the Flink system; modifying the code for connecting the Flink system to an external database; writing a Flink SQL statement for querying the external database, and judging whether the user has set the parallelism of the LookupTable through the first interface or the second interface; if the user has set the parallelism of the LookupTable, the Flink SQL statement is run using the parallelism of the LookupTable. Thus, the user can set the parallelism of the LookupTable according to the actual situation, and then control the number of threads of the LookupTable, and avoid the phenomenon that the external data source cannot bear due to too many threads, and the phenomenon that too few threads reduce the real-time query performance, thereby improving the usability of the system.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a Flink-based external data query method, a Flink-based external data query device and a computer device. Background Art

[0002] With the development of digital transformation of enterprises, enterprises have higher and higher requirements for real-time data processing, especially real-time processing of massive data. Among the current open source code projects for real-time processing of massive data, Flink has the best functions and performance. After massive data enters the Flink system, the results are calculated in milliseconds. This system can be used in various fields such as real-time recommendation, real-time monitoring, and real-time intelligence analysis. During the execution of real-time calculations, it is necessary to query data tables of external data sources, such as MySQL, PostgreSQL, etc. These tables of external data sources are abstracted as Lookup Tables in the Flink system. Flink provides a LookupableTableSource, which can be used to implement dimension tables.

[0003] In related technologies, you can only specify the overall parallelism of FlinkSQL, but you cannot customize the parallelism of this LookupTable. Generally speaking, one parallelism corresponds to one thread. This leads to resource or performance bottlenecks during the query process, either too many threads make the external data source unbearable, or too few threads reduce real-time computing performance. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems in the above-mentioned technology to a certain extent. To this end, the first purpose of the present invention is to propose an external data query method based on Flink, which allows users to set the parallelism of LookupTable according to actual conditions, and then control the number of threads of LookupTable, which can avoid the phenomenon that the external data source cannot bear due to too many threads, and the phenomenon that too few threads reduce the real-time query performance, thereby improving the usability of the system.

[0005] The second objective of the present invention is to provide an external data query device based on Flink.

[0006] A third object of the present invention is to provide a computer device.

[0007] To achieve the above object, a first embodiment of the present invention proposes a Flink-based external data query method, including:

[0008] The Java interface of the Flink system is extended and a parallelism setting option is added to obtain a first interface and a second interface having a parallelism setting option.

[0009] Modify a SQL compiler built into the Flink system so that the SQL compiler recognizes the parallelism setting option;

[0010] Modify the code for connecting the Flink system to the external database so that the Flink system uses the first interface and the second interface to connect to the external database;

[0011] Write a Flink SQL statement to query the external database, and determine whether the user has set the parallelism of the LookupTable through the first interface or the second interface;

[0012] If the user sets the parallelism of the LookupTable, the Flink SQL statement is run using the parallelism of the LookupTable;

[0013] If the user does not set the parallelism of the LookupTable, the global parallelism is obtained and the SQL statement is run using the global parallelism.

[0014] In addition, the Flink-based external data query method proposed in the above embodiment of the present invention may also have the following additional technical features:

[0015] According to one embodiment of the present invention, the Java interface of the Flink system is extended and a parallelism setting option is added to obtain a first interface and a second interface having the parallelism setting option, including:

[0016] Call the TableFunctionProvider interface, AsyncTableFunctionProvider interface, and ParallelismProvider interface of the Flink system;

[0017] Inherit the TableFunctionProvider interface and the ParallelismProvider interface using Java standard syntax to obtain the first interface, and add a parallelism setting option to the first interface;

[0018] Inherit the AsyncTableFunctionProvider interface and the ParallelismProvider interface using Java standard syntax to obtain the second interface, and add a parallelism setting option to the second interface;

[0019] Put the first interface and the second interface in the org.apache.flink.table.connector.source package of the flink-table-common module of the Flink system;

[0020] Compile the flink-table-common module and use the compiled package to replace the program Jar package starting with flink-table-common in the Flink lib directory.

[0021] According to one embodiment of the present invention, the first interface is named

[0022] ParallelTableFunctionProvider, the second interface is named

[0023] ParallelAsyncTableFunctionProvider.

[0024] According to an embodiment of the present invention, modifying a SQL compiler built into the Flink system so that the SQL compiler recognizes the parallelism setting option includes:

[0025] Modify the source code class CommonLookupJoin of the SQL compiler built into the Flink system so that the SQL compiler recognizes the parallelism setting option;

[0026] Get the modified class and put it in the org.apache.flink.table.planner.plan.nodes.common package of the flink-table-planner-blink module;

[0027] Compile the flink-table-planner-blink module and use the compiled package to replace the program Jar package starting with flink-table-planner-blink in the Flink lib directory.

[0028] According to an embodiment of the present invention, the external data query method based on Flink further includes:

[0029] Add judgment logic to the source code class CommonLookupJoin;

[0030] Based on the judgment logic, it is determined whether the ParallelismProvider interface is implemented, and if the ParallelismProvider interface is implemented, the parallelism of the LookupTable is read and used;

[0031] If the ParallelismProvider interface is not implemented, the original code is executed.

[0032] According to an embodiment of the present invention, modifying the code for connecting the Flink system to the external database so that the Flink system uses the first interface and the second interface to connect to the external database includes:

[0033] Modify the source code of any flink-connectors so that the source code of the flink-connectors uses the first interface and the second interface;

[0034] Add a parallelism preset option in the flink-connectors source code, and set the parallelism preset option to the parallelism setting option of the first interface or the second interface.

[0035] According to one embodiment of the present invention, the parallelism of the LookupTable is set by the user by setting the value of the parallelism setting option.

[0036] To achieve the above object, a second embodiment of the present invention proposes an external data query device based on Flink, including:

[0037] An extension module is used to extend the Java interface of the Flink system and add a parallelism setting option to obtain a first interface and a second interface with the parallelism setting option;

[0038] A first modification module is used to modify a SQL compiler built into the Flink system so that the SQL compiler recognizes the parallelism setting option;

[0039] A second modification module is used to modify the code of the Flink system connecting to the external database, so that the Flink system uses the first interface and the second interface to connect to the external database;

[0040] A writing module, used to write a Flink SQL statement for querying the external database, and determine whether the user has set the parallelism of the LookupTable through the first interface or the second interface;

[0041] A first running module, configured to run the Flink SQL statement using the parallelism of the LookupTable if the user sets the parallelism of the LookupTable;

[0042] The second running module is used to obtain the global parallelism if the user does not set the parallelism of the LookupTable, and use the global parallelism to run the SQL statement.

[0043] To achieve the above objectives, an embodiment of the third aspect of the present invention proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the Flink-based external data query method proposed in the embodiment of the first aspect of the present invention is implemented.

[0044] The technical solution of the present invention extends the Java interface of the Flink system, adds a parallelism setting option, modifies the SQL compiler built into the Flink system and the code for connecting the Flink system to an external database, so that the user can customize the parallelism of the LookupTable through the extended new interface, and can adjust the parallelism of the LookupTable according to specific needs, and then when querying the external database, the query statement can be run through the parallelism of the LookupTable set by the user to implement the query of the external database. Thus, the user can set the parallelism of the LookupTable according to the actual situation, and then control the number of threads of the LookupTable, which can avoid the phenomenon that the external data source cannot bear due to too many threads, and the phenomenon that too few threads reduce the real-time query performance, thereby improving the usability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The figure is a flowchart of an external data query method based on Flink according to an embodiment of the present invention.

[0046] Figure 2 This is a logical diagram for querying external data according to an embodiment of the present invention.

[0047] Figure 3 The figure is a schematic diagram of an extended interface structure according to an embodiment of the present invention.

[0048] Figure 4 4 is a block diagram of an external data query device based on Flink according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Figure 1 The figure is a flowchart of an external data query method based on Flink according to an embodiment of the present invention.

[0051] like Figure 1 As shown, the method comprises the following steps:

[0052] S1, expand the Java interface of the Flink system and add a parallelism setting option to obtain a first interface and a second interface having the parallelism setting option.

[0053] The parallelism setting option is used to customize the parallelism of the Lookup Table. Parallelism is a concept defined in the Flink system. The parallelism setting option essentially corresponds to threads in the computer operating system. If the parallelism is set to 5, five operating system threads will be started to run the LookupTable code.

[0054] Specifically, the original Java interface of the Flink system, such as the TableFunctionProvider interface and the AsyncTableFunctionProvider interface, can be extended. The original interface does not have a parallelism setting option. After the extension, the parallelism setting option is added to obtain a first interface with a parallelism setting option and a second interface with a parallelism setting option, that is, the interface has a parallelism setting function.

[0055] S2: Modify the SQL compiler built into the Flink system so that the SQL compiler recognizes the parallelism setting option.

[0056] Specifically, the SQL compiler built into the Flink system may be modified according to the Blink Planner compiler standard of Flink. In this standard, there is a translateToPlanInternal method of the CommonLookupJoin class. This method itself does not recognize the parallelism option. In step S2, the code of this method is expanded so that it can recognize the parallelism setting options of the first interface and the second interface.

[0057] S3: Modify the code for connecting the Flink system to the external database so that the Flink system uses the first interface and the second interface to connect to the external database.

[0058] Specifically, the code for Flink to connect to the external database can be modified according to the Flink Table API Connector development standard so that it uses the first interface and the second interface to connect to the external database.

[0059] S4: Write a Flink SQL statement to query the external database, and determine whether the user has set the parallelism of the LookupTable through the first interface or the second interface.

[0060] S5: If the user sets the parallelism of the LookupTable, the Flink SQL statement is run using the parallelism of the LookupTable.

[0061] S6: If the user does not set the parallelism of the LookupTable, the global parallelism is obtained and the SQL statement is run using the global parallelism.

[0062] The global parallelism refers to the overall parallelism of Flink SQL, which is a user-defined setting.

[0063] Specifically, the user can customize the parallelism of the LookupTable through the first interface or the second interface. After writing the SQL statement to query the external database, such as Figure 2 As shown, it is determined whether the user has set the parallelism of the LookupTable. If so, the set value, that is, the parallelism of the LookupTable, is used to run the SQL statement; otherwise, the user-defined global parallelism (global value) is used to run the SQL statement. After the running is completed, the data of the external database can be queried.

[0064] Compared with the method in the related art that can only use global parallelism to query external databases, the embodiment of the present invention expands the Java interface of the Flink system on the basis of retaining the global parallelism method, and adds parallelism setting options, and modifies the SQL compiler built into the Flink system and the code for the Flink system to connect to the external database, so that users can customize the parallelism of the LookupTable through the extended new interface, and can adjust the parallelism of the LookupTable according to specific needs. When querying the external database, the query statement can be run through the parallelism of the LookupTable set by the user to realize the query of the external database. Therefore, compared with the prior art, the embodiment of the present invention can prevent the occurrence of resource or performance bottlenecks during the query process. Users can set the parallelism of the LookupTable according to specific circumstances to avoid the phenomenon that the external data source cannot bear due to too many threads, and the phenomenon that too few threads reduce the real-time query performance and real-time computing performance.

[0065] Therefore, the Flink-based external data query method of the embodiment of the present invention allows the user to set the parallelism of the LookupTable according to actual conditions, thereby controlling the number of threads of the LookupTable, avoiding the phenomenon that the external data source cannot bear due to too many threads and the phenomenon that too few threads reduce the real-time query performance, thereby improving the usability of the system.

[0066] In one embodiment of the present invention, the above step S1 may include the following steps S11 to S15.

[0067] S11, call the TableFunctionProvider interface, AsyncTableFunctionProvider interface, and ParallelismProvider interface of the Flink system.

[0068] Among them, TableFunctionProvider interface, AsyncTableFunctionProvider interface and ParallelismProvider interface are Java Interfaces in Flink source code. TableFunctionProvider interface and AsyncTableFunctionProvider interface are used to create tables in Flink, and ParallelismProvider interface is used to set parallelism.

[0069] S12, using Java standard syntax to inherit the TableFunctionProvider interface and the ParallelismProvider interface to obtain a first interface, and adding a parallelism setting option to the first interface.

[0070] Among them, the first interface is named ParallelTableFunctionProvider.

[0071] Specifically, the relevant Flink technology only provides two interfaces: TableFunctionProvider interface and AsyncTableFunctionProvider interface. These two interfaces cannot set the degree of parallelism. Figure 3 As shown, in the embodiment of the present invention, the original TableFunctionProvider interface and ParallelismProvider interface of Flink are inherited using Java standard syntax, and the parallelism function is added. The inherited sub-interface (ie, the first interface) can be named ParallelTableFunctionProvider.

[0072] The ParallelTableFunctionProvider interface is a fusion of the TableFunctionProvider interface and the ParallelismProvider interface, which can be used to create a table and set the degree of parallelism.

[0073] S13, using Java standard syntax to inherit the AsyncTableFunctionProvider interface and the ParallelismProvider interface to obtain a second interface, and adding a parallelism setting option to the second interface.

[0074] Among them, the second interface is named ParallelAsyncTableFunctionProvider.

[0075] Specifically, refer to Figure 3 , you can use Java standard syntax to inherit Flink's original AsyncTableFunctionProvider interface and ParallelismProvider interface, add parallelism function, and name the inherited sub-interface (i.e. the second interface)

[0076] ParallelTableFunctionProvider.

[0077] The ParallelTableFunctionProvider interface is a fusion of the TableFunctionProvider interface and the ParallelismProvider interface, which can be used to create a table and set the degree of parallelism.

[0078] It should be noted that the naming of the first interface and the second interface is not limited to those proposed in the embodiment of the present invention, and can also be defined as other names that do not affect the operation of the system.

[0079] S14, place the first interface and the second interface in the org.apache.flink.table.connector.source package of the flink-table-common module of Flink.

[0080] Among them, the flink-table-common module of Flink is the Maven module of the Flink source code, and all the common codes related to the FlinkTable module are in this module. The org.apache.flink.table.connector.source package is the Flink source code package, and the contents inside are all the official standard interfaces for the Flink system to query external system data.

[0081] S15, compile the flink-table-common module and use the compiled package to replace the program Jar package starting with flink-table-common in the Flink lib directory.

[0082] Specifically, the new first interface ParallelTableFunctionProvider and the second interface ParallelAsyncTableFunctionProvider are placed in the org.apache.flink.table.connector.source package of the flink-table-common module of Flink. After that, compile the flink-table-common module to obtain the compiled package. Only the compiled files placed in the Flink lib directory can be loaded by the system, so replace the program Jar package starting with flink-table-common in the Flink lib directory with the compiled package. After the Java program is compiled, it is a Jar package. After the code is changed, the original Jar package in the Flink lib directory must be replaced to ensure that the modified code in the embodiment of the present invention takes effect.

[0083] Through the above steps S11 to S15, the settings of the ParallelTableFunctionProvider interface and the ParallelAsyncTableFunctionProvider interface are implemented. The set interface structure can be seen in Figure 3 shown.

[0084] In one embodiment of the present invention, the above step S2 may include the following steps S21 to S23.

[0085] S21, modify the source code class CommonLookupJoin of the SQL compiler built into the Flink system so that the SQL compiler recognizes the parallelism setting option.

[0086] The source code class CommonLookupJoin is part of the Flink Blink planner standard compiler. The code in this class is executed whenever LookupTable is used.

[0087] S22, obtain the modified class and put the modified class in the org.apache.flink.table.planner.plan.nodes.common package of the flink-table-planner-blink module.

[0088] The flink-table-planner-blink module is the Maven module of the Flink source code. All FlinkBlink planner standard compiler codes are in this module. The org.apache.flink.table.planner.plan.nodes.common package of the flink-table-planner-blink module contains some Flink Blink planner standard compiler codes.

[0089] S23, compile the flink-table-planner-blink module, and use the compiled package to replace the program Jar package starting with flink-table-planner-blink in the Flink lib directory.

[0090] Furthermore, the external data query method based on Flink may also include: adding judgment logic to the source code class CommonLookupJoin; judging whether the ParallelismProvider interface is implemented based on the judgment logic, and if the ParallelismProvider interface is implemented, reading and using the parallelism of the LookupTable; if the ParallelismProvider interface is not implemented, executing the original code. After the Java program is compiled, it is a Jar package. After the code is changed, the original Jar package in the Flink lib directory must be replaced to ensure that the changed code takes effect.

[0091] Specifically, modify the Flink SQL compiler source code so that it can recognize the new ParallelTableFunctionProvider interface and ParallelAsyncTableFunctionProvider interface. The specific source code class is CommonLookupJoin. Add judgment logic to the source code class CommonLookupJoin to determine if the ParallelismProvider interface is implemented and read and use its parallelism. If not, execute the original code. After that, put the modified class in the org.apache.flink.table.planner.plan.nodes.common package of the flink-table-planner-blink module. After that, compile the flink-table-planner-blink module to get the compiled package. Only the compiled file placed in the Flink lib directory can be loaded by the system. Replace the program Jar package starting with flink-table-planner-blink in the Flink lib directory with the compiled package.

[0092] In one embodiment of the present invention, the above step S3 may include the following steps S31 and S32.

[0093] S31, modify the source code of any flink-connectors so that the source code of the flink-connectors uses the first interface and the second interface.

[0094] The modules that connect Flink to external data systems (query or write) are called "flink-connectors". For example, if a file system is connected, it is called "flink-connector-filesystem".

[0095] S32, add a parallelism preset option in any flink-connectors source code, and set the parallelism preset option to the parallelism setting option of the first interface or the second interface.

[0096] The parallelism preset option refers to an option added to the flink-connectors source code that can set the parallelism. In the embodiment of the present invention, it can be named, for example, the parallelism preset option can be named lookup-parallelism.

[0097] Furthermore, the parallelism of the LookupTable in the above steps S4 to S6 is set by the user by setting the value of the parallelism setting option.

[0098] Specifically, you can obtain any flink-connectors source code and modify the flink-connectors source code to use the new ParallelTableFunctionProvider interface or

[0099] ParallelAsyncTableFunctionProvider interface. The previous steps S1 and S2 have completed the modification of the Flink kernel. The purpose of this step S3 is to enable the flink connector plug-in to use the new Flink kernel code.

[0100] You can add the lookup-parallelism option in any flink-connectors source code, and set the option value of this parallelism option to the parallelism setting option of the ParallelTableFunctionProvider interface or the ParallelAsyncTableFunctionProvider interface. This step allows users to use all the codes modified by the embodiments of the present invention by configuring the lookup-parallelism option.

[0101] After executing steps S1 to S3 above, execute steps S4 to S6, that is, write a FlinkSQL statement to query external data. At this time, you can freely control the parallelism of the LookupTable by setting the lookup-parallelism option in the DDL.

[0102] It should be noted that the above steps S11 to S15, steps S21 to S23, and steps S31 and S32 can be implemented based on the Flink 1.12 version source code.

[0103] Taking the transmission of a large number of users into a real-time computing engine and querying user identity information as an example, the present invention is implemented through the following steps:

[0104] Step 1: Write three SQL statements: reading the message queue, checking the database, and writing the database logic.

[0105] Step 2: Set the global parallelism to 5.

[0106] Step 3: Checking the external database In this part, since the Flink source code is modified according to the embodiment of the present invention, the parallelism can be changed to 3 to reduce the pressure on accessing the database.

[0107] Step 4: Submit the computing task to the Flink computing cluster to execute the computing logic.

[0108] In summary, the Flink-based external data query method in an embodiment of the present invention extends and modifies the Apache Flink source code, so that the Lookup Table module can customize the parallelism and improve the usability of the entire system.

[0109] Figure 4 4 is a block diagram of an external data query device based on Flink according to an embodiment of the present invention.

[0110] like Figure 4 As shown, the Flink-based external data query device 100 includes: an expansion module 10, a first modification module 20, a second modification module 30, a writing module 40, a first operation module 50 and a second operation module 60.

[0111] Among them, the extension module 10 is used to expand the Java interface of the Flink system and add a parallelism setting option to obtain a first interface and a second interface with a parallelism setting option; the first modification module 20 is used to modify the SQL compiler built into the Flink system so that the SQL compiler recognizes the parallelism setting option; the second modification module 30 is used to modify the code for the Flink system to connect to the external database so that the Flink system uses the first interface and the second interface to connect to the external database; the writing module 40 is used to write Flink SQL statements for querying the external database and determine whether the user has set the parallelism of the LookupTable through the first interface or the second interface; the first running module 50 is used to run the Flink SQL statement using the parallelism of the LookupTable if the user has set the parallelism of the LookupTable; the second running module 60 is used to obtain the global parallelism if the user has not set the parallelism of the LookupTable and run the SQL statement using the global parallelism.

[0112] In one embodiment, the extension module 10 can be specifically used to: call the TableFunctionProvider interface, AsyncTableFunctionProvider interface and ParallelismProvider interface of the Flink system; use Java standard syntax to inherit the

[0113] TableFunctionProvider interface and the ParallelismProvider interface to obtain the first interface, and add a parallelism setting option to the first interface; use Java standard syntax to inherit the AsyncTableFunctionProvider interface and the ParallelismProvider interface to obtain the second interface, and add a parallelism setting option to the second interface; put the first interface and the second interface in the org.apache.flink.table.connector.source package of the flink-table-common module of the Flink system; compile the flink-table-common module, and use the compiled package to replace the program Jar package starting with flink-table-common in the Flink lib directory.

[0114] In one embodiment, the first modification module 20 can be specifically used to: modify the source code class CommonLookupJoin of the SQL compiler built into the Flink system so that the SQL compiler recognizes the parallelism setting option; obtain the modified class, and put the modified class in the org.apache.flink.table.planner.plan.nodes.common package of the flink-table-planner-blink module; compile the flink-table-planner-blink module, and use the compiled package to replace the program Jar package starting with flink-table-planner-blink in the Flink lib directory.

[0115] Furthermore, the external data query device 100 based on Flink may also include: a joining module, used to add judgment logic to the source code class CommonLookupJoin; a judgment module, used to judge whether the ParallelismProvider interface is implemented based on the judgment logic, and if the ParallelismProvider interface is implemented, read and use its parallelism; and an execution module, used to execute the original code if the ParallelismProvider interface is not implemented.

[0116] In one embodiment, the second modification module 30 can be specifically used to: modify the source code of any flink-connectors to use the first interface or the second interface; add a parallelism preset option in the source code of any flink-connectors, and set the parallelism preset option to the parallelism setting option of the first interface or the second interface.

[0117] It should be noted that, for other specific implementations of the external data query device based on Flink in the embodiment of the present invention, reference can be made to the specific implementations of the external data query method based on Flink mentioned above, which will not be described again here to avoid redundancy.

[0118] The device allows the user to set the parallelism of the LookupTable according to actual conditions, and then control the number of threads of the LookupTable, which can avoid the phenomenon that the external data source cannot bear due to too many threads and the phenomenon that too few threads reduce the real-time query performance, thereby improving the usability of the system.

[0119] Corresponding to the above embodiment, the present invention also provides a computer device.

[0120] The computer device of an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the Flink-based external data query method according to the above embodiment of the present invention can be implemented.

[0121] According to the computer device of the embodiment of the present invention, when the processor executes the computer program, the Java interface of the Flink system is extended, and a parallelism setting option is added, and the SQL compiler built into the Flink system and the code for connecting the Flink system to the external database are modified, so that the user can customize the parallelism of the LookupTable through the extended new interface. Thus, the user can set the parallelism of the LookupTable according to the actual situation, and then control the number of threads of the LookupTable, which can avoid the phenomenon that the external data source cannot bear due to too many threads, and the phenomenon that too few threads reduce the real-time query performance, thereby improving the usability of the system.

[0122] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. "Multiple" means two or more, unless otherwise clearly and specifically defined.

[0123] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0124] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0125] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0126] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0127] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0128] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0129] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for querying external data based on Flink. It is characterized in that include: The Java interface of the Flink system is extended and a parallelism setting option is added to obtain a first interface and a second interface having the parallelism setting option. Modify a SQL compiler built into the Flink system so that the SQL compiler recognizes the parallelism setting option; Modify the code for connecting the Flink system to the external database so that the Flink system uses the first interface and the second interface to connect to the external database; Write a Flink SQL statement to query the external database, and determine whether the user has set the parallelism of the LookupTable through the first interface or the second interface; If the user sets the parallelism of the LookupTable, the Flink SQL statement is run using the parallelism of the LookupTable; If the user does not set the parallelism of the LookupTable, the global parallelism is obtained and the SQL statement is run using the global parallelism.

2. According to the Flink-based external data query method of claim 1, It is characterized in that The Java interface of the Flink system is extended and a parallelism setting option is added to obtain a first interface and a second interface with a parallelism setting option, including: Call the TableFunctionProvider interface of the Flink system. AsyncTableFunctionProvider interface and ParallelismProvider interface; Inherit the TableFunctionProvider interface and the ParallelismProvider interface using Java standard syntax to obtain the first interface, and add a parallelism setting option to the first interface; Inherit the AsyncTableFunctionProvider interface and the ParallelismProvider interface using Java standard syntax to obtain the second interface, and add a parallelism setting option to the second interface; Put the first interface and the second interface in the org.apache.flink.table.connector.source package of the flink-table-common module of the Flink system; Compile the flink-table-common module and use the compiled package to replace the program Jar package starting with flink-table-common in the Flink lib directory.

3. According to the Flink-based external data query method of claim 2, It is characterized in that The first interface is named ParallelTableFunctionProvider, and the second interface is named ParallelAsyncTableFunctionProvider.

4. According to the Flink-based external data query method of claim 1, It is characterized in that Modifying a SQL compiler built into the Flink system so that the SQL compiler recognizes the parallelism setting option includes: Modify the source code class CommonLookupJoin of the SQL compiler built into the Flink system so that the SQL compiler recognizes the parallelism setting option; Get the modified class and put it in the org.apache.flink.table.planner.plan.nodes.common package of the flink-table-planner-blink module; Compile the flink-table-planner-blink module and use the compiled package to replace the program Jar package starting with flink-table-planner-blink in the Flink lib directory.

5. According to the Flink-based external data query method of claim 4, It is characterized in that Also includes: Add judgment logic to the source code class CommonLookupJoin; Based on the judgment logic, it is determined whether the ParallelismProvider interface is implemented, and if the ParallelismProvider interface is implemented, the parallelism of the LookupTable is read and used; If the ParallelismProvider interface is not implemented, the original code is executed.

6. The external data query method based on Flink according to claim 1, It is characterized in that Modifying the code for connecting the Flink system to the external database so that the Flink system uses the first interface and the second interface to connect to the external database includes: Modify the source code of any flink-connectors so that the source code of the flink-connectors uses the first interface and the second interface; Add a parallelism preset option in the flink-connectors source code, and set the parallelism preset option to the parallelism setting option of the first interface or the second interface.

7. The external data query method based on Flink according to claim 6, It is characterized in that The parallelism of the LookupTable is set by the user by setting the value of the parallelism setting option.

8. An external data query device based on Flink, It is characterized in that include: An extension module is used to extend the Java interface of the Flink system and add a parallelism setting option to obtain a first interface and a second interface with the parallelism setting option; A first modification module is used to modify a SQL compiler built into the Flink system so that the SQL compiler recognizes the parallelism setting option; A second modification module is used to modify the code of the Flink system connecting to the external database, so that the Flink system uses the first interface and the second interface to connect to the external database; A writing module, used to write a Flink SQL statement for querying the external database, and determine whether the user has set the parallelism of the LookupTable through the first interface or the second interface; A first running module, configured to run the Flink SQL statement using the parallelism of the LookupTable if the user sets the parallelism of the LookupTable; The second running module is used to obtain the global parallelism if the user does not set the parallelism of the LookupTable, and use the global parallelism to run the SQL statement.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the Flink-based external data query method according to any one of claims 1 to 7 is implemented.

Images