Flink-oriented dimension table association method, device and storage medium

Abstract

This application provides a method, device, and storage medium for dimension table joins in Flink, relating to the field of big data technology. The method identifies a target subtask among multiple subtasks of a unified dimension table join operator in the current task manager. After controlling the target subtask to send a query request to an external database and obtain the dimension table data sent by the external database, the method controls the target subtask to load the dimension table data into a static dimension table object set in the metadata space of the task manager. This avoids putting significant operational pressure on the external database in a short period. The method controls the target subtask and multiple other subtasks to perform dimension table join operations based on the obtained raw data and the dimension table data in the static dimension table object, allowing all subtasks in the same task manager to share the same dimension table data, thereby saving physical memory resources.

Description

Technical Field

[0001] This application relates to the field of big data technology, specifically to a method, device, and storage medium for dimension table association in Flink. Background Technology

[0002] In streaming computing scenarios using the Flink framework, dimension table joins are a common operation in real-time stream processing. Dimension table joins enable business objectives such as field population, data filtering, and quality control. A dimension table, sometimes called a lookup table, is a table that corresponds to a fact table. It stores the attribute values ​​of dimensions and can be joined with the fact table. Essentially, it extracts and standardizes frequently occurring attributes from the fact table and manages them in a single table. Common dimension tables include date tables and location tables.

[0003] Existing dimension table join methods include "real-time query database join", "preloaded dimension table join", and "broadcast real-time update dimension table join". Among them, "preloaded dimension table join" is the most common implementation. However, when the dimension table is loaded for the first time after the Flink job starts, all subtasks of the dimension table join operator need to connect to the external database that stores the dimension table data. Furthermore, when the dimension table is updated periodically, all subtasks of the dimension table join operator also need to connect to the external database. Therefore, this puts a lot of pressure on the external database in a short period of time. At the same time, since all subtasks of the dimension table join operator need to keep a copy of the dimension table object in memory to store the dimension table data, additional memory is required.

[0004] Existing technologies, when performing data dimension table joins with the Flink framework, still fall short in terms of putting significant operational pressure on external databases in a short period of time and consuming considerable memory resources. Summary of the Invention

[0005] This application provides a method, device, and storage medium for dimension table association in Flink, which addresses the problems of existing technologies causing significant operational pressure on external databases and insufficient memory resources when performing dimension table association in the Flink framework.

[0006] Firstly, this application provides a method for dimension table joins in Flink, including:

[0007] Confirm the target subtask among the multiple subtasks in the current Task Manager;

[0008] After controlling the target subtask to send a query request to an external database and obtain the dimension table data sent by the external database, the target subtask loads the dimension table data into a static dimension table object, wherein the static dimension table object is set in the metadata space of the task manager.

[0009] The target subtask and the other multiple subtasks are controlled to perform dimension table join operations based on the acquired raw data and the dimension table data in the static dimension table object.

[0010] In one possible design, identifying the target subtask among multiple subtasks in the current task manager includes obtaining the working status of each subtask in the current task manager, wherein the multiple subtasks are subtasks of the same dimension table association operator in the current task manager.

[0011] If there is only one subtask in the working state that is idle, then the currently idle subtask is identified as the target subtask. If there are multiple subtasks in the working state that are idle, then the target subtask is identified among the multiple idle subtasks.

[0012] In one possible design, identifying the target subtask among multiple idle subtasks includes obtaining execution information of each idle subtask, wherein the execution information is used to indicate the execution speed of the subtask.

[0013] The subtask with the fastest execution speed is identified as the target subtask.

[0014] In one possible design, after the target subtask loads the dimension table data into the static dimension table object, the method further includes updating the static dimension table object based on the static identifier of the static dimension table object, wherein the static identifier is used to indicate whether the target subtask has completed the task of loading the dimension table data into the static dimension table object.

[0015] If the static identifier indicates that the target subtask has completed the work of loading dimension table data of the static dimension table object, then after a first preset time, a new query request is sent to the external database, the query result sent by the external database is obtained, and the static dimension table object is updated according to the query result, and so on.

[0016] In one possible design, the static dimension table object is updated based on the query result, the query result including new dimension table data and update information, including if the update information indicates that the data in the external database has not changed, then the target subtask maintains the loading of the dimension table data onto the static dimension table object;

[0017] If the update information indicates that the data in the external database has changed, the target subtask will load the new dimension table data into the static dimension table object to complete the update of the static dimension table object.

[0018] In one possible design, after the static identifier indicates that the target subtask has completed the dimension table data loading work of the static dimension table object, a new static identifier is obtained after a second preset time. If the new static identifier continues to indicate that the target subtask has completed the dimension table data loading work of the static dimension table object, a new static identifier is obtained again after a second preset time. This process is repeated until a new static identifier indicates that the target subtask has not completed the dimension table data loading work of the static dimension table object. The second preset time is much shorter than the first preset time.

[0019] In one possible design, if a new static identifier indicates that the target subtask has not completed the work of loading dimension table data into the static dimension table object, it includes canceling the current target subtask and reconfirming a new target subtask among multiple subtasks other than the previous target subtask.

[0020] Secondly, this application provides a dimension table association apparatus for Flink, comprising:

[0021] The acquisition module is used to identify a target subtask among multiple subtasks in the current Task Manager;

[0022] The processing module is used to control the target subtask to send a query request to an external database, and after obtaining the dimension table data sent by the external database, the target subtask loads the dimension table data into a static dimension table object, wherein the static dimension table object is set in the metadata space of the task manager.

[0023] The execution module is used to control the target subtask and the other multiple subtasks to perform dimension table join operations based on the acquired raw data and the dimension table data in the static dimension table object.

[0024] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0025] The memory stores computer-executed instructions;

[0026] The processor executes computer-executed instructions stored in the memory to implement a Flink-oriented dimension table association method.

[0027] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement a Flink-oriented dimension table association method.

[0028] The dimension table join method, device, and storage medium provided in this application for Flink identify a target subtask among multiple subtasks pointing to a unified dimension table join operator in the current task manager. After controlling the target subtask to send a query request to an external database and obtain the dimension table data sent by the external database, the target subtask loads the dimension table data into a static dimension table object set in the metadata space of the task manager. This avoids putting a lot of running pressure on the external database in a short period of time. The target subtask and the other subtasks perform dimension table join operations based on the obtained raw data and the dimension table data in the static dimension table object, so that all subtasks in the same task manager share the same dimension table data, thereby saving physical memory resources. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a diagram illustrating the runtime structure of the Flink framework provided in the embodiments of this application.

[0031] Figure 2 This is a schematic diagram of the structure of the task manager provided in an embodiment of this application;

[0032] Figure 3 A flowchart illustrating the Flink-oriented dimension table join method provided in the embodiments of this application. Figure 1 ;

[0033] Figure 4 A flowchart illustrating the Flink-oriented dimension table join method provided in the embodiments of this application. Figure 2 ;

[0034] Figure 5 A schematic diagram of the structure of a Flink-oriented dimension table association device provided in an embodiment of this application;

[0035] Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0037] The Flink framework typically consists of two types of processes at runtime, such as Figure 1 As shown, a JobManager and one or more TaskManagers (i.e., task managers) are used. The JobManager is responsible for management and allocation functions in the distributed computing system, while the TaskManagers are responsible for execution functions. In other words, the JobManager is responsible for coordinating the distributed execution of the Flink framework. The TaskManager is responsible for executing tasks in the job stream and caching and exchanging data streams.

[0038] The smallest unit of resource scheduling in TaskManager is the task slot, such as... Figure 2 As shown, the number of task slots in a TaskManager represents the number of tasks processed concurrently. A task slot is provided as a physical resource to Flink operators to execute subtasks of those operators. Each TaskManager is a JVM process that can execute one or more subtasks, such as dimension table joins, in a separate thread. A TaskManager with multiple task slots means that multiple subtasks share the same JVM process. The JVM process is an operating system process that executes .class files compiled by javac. Subtasks within the same JVM process can share TCP connections and heartbeat information, as well as datasets and data structures.

[0039] Existing dimension table join methods include "real-time database query join," "preloaded dimension table join," and "broadcast real-time update dimension table join." "Real-time database query join" refers to directly accessing an external database within a user-defined operator and retrieving dimension attribute values ​​from the external database based on the join fields; processing each piece of data to be joined requires a request to the external database. "Preloaded dimension table join" involves loading the dimension table into the TaskManager's memory when the Flink job starts and retrieving dimension attribute values ​​based on the join fields by accessing the dimension table object in memory. "Broadcast real-time update dimension table join" involves outputting the dimension table data to a broadcast stream, then connecting the broadcast stream and the data stream, storing the dimension table data in the broadcast state of the Flink operator, and retrieving dimension attribute values ​​based on the join fields by accessing the broadcast state. These three dimension table join methods each have their own suitable practical business scenarios in terms of implementation complexity, throughput, real-time performance, resource consumption, scalability, and result determinism.

[0040] One common implementation is "preloading dimension table joins," which includes two parts: preloading dimension tables at startup and periodically updating dimension tables. Preloading dimension tables at startup means that during Flink job initialization, the raw dimension table data from the external database is read into memory in the `open` method of the dimension table join operator. The dimension table data in memory is not used as the Flink operator state storage. The `open` method is executed again every time the job restarts, thus loading the latest dimension table data. However, with the preloading dimension table join method, when the dimension table is loaded for the first time after the Flink job starts, all subtasks of the dimension table join operator need to connect to the external database that stores the dimension table data. Similarly, when the dimension table is updated periodically, all subtasks of the dimension table join operator also need to connect to the external database. Therefore, this puts a lot of pressure on the external database in a short period of time. At the same time, since all subtasks of the dimension table join operator need to keep a copy of the dimension table object in memory to store the dimension table data, additional memory is required. Furthermore, since dimension table changes are synchronized to the Flink job framework periodically rather than in real time, there may be problems with the data stream not being able to associate the newly added dimension table attribute values ​​before the periodic synchronization.

[0041] This application provides a dimension table join method for Flink. By identifying a target subtask within multiple subtasks pointing to a unified dimension table join operator in the current TaskManager, the target subtask sends a query request to an external database and retrieves the dimension table data from the external database. Then, the target subtask loads the dimension table data into a static dimension table object located in the TaskManager's metadata space. This avoids putting significant operational pressure on the external database in a short period. The target subtask and the other subtasks perform dimension table join operations based on the retrieved raw data and the dimension table data in the static dimension table object. This allows all subtasks within the same TaskManager to share the same dimension table data, thereby reducing the number of dimension table "copy" operations within the same TaskManager process and saving physical memory resources.

[0042] The technical solutions of this application and how they solve the aforementioned technical problems are described in detail below using specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0043] Example 1

[0044] Figure 3 A flowchart illustrating the Flink dimension table join method provided in the embodiments of this application. Figure 1 .like Figure 3 As shown, the method includes:

[0045] S301. Confirm the target subtask among multiple subtasks in the current Task Manager;

[0046] Specifically, in existing technologies, preloading dimension table association schemes generally initialize dimension tables through the open method of the dimension table association operator. The open method is executed during the initialization process of the dimension table association operator, so that each concurrent subtask will execute the open method of its respective dimension table association operator. Therefore, all subtasks under the same dimension table association operator in the same TaskManager will compete to acquire the lock when executing the open method. The subtask that acquires the lock is identified as the target subtask, that is, the target subtask is identified among multiple subtasks.

[0047] S302. After controlling the target subtask to send a query request to the external database and obtain the dimension table data sent by the external database, the target subtask loads the dimension table data into the static dimension table object, wherein the static dimension table object is set in the metadata space of the task manager.

[0048] Specifically, after confirming that the subtask that has acquired the lock, i.e. the target subtask, has acquired the lock, the subtask is controlled to load dimension table data into the static dimension table object. The dimension table, as a kind of business reference information, has the characteristic of being written once and read everywhere. By modifying the dimension table object to a static class member and storing it in the metadata space of the task manager at runtime, all subtasks in the same task manager can share the same dimension table data, thereby reducing the number of dimension table copies in the same task manager and saving physical memory resources.

[0049] S303. Control the target subtask and the remaining multiple subtasks to perform dimension table association operations based on the acquired raw data and the dimension table data in the static dimension table object.

[0050] Specifically, after the target subtask, which acquires the lock, loads the obtained dimension table data into the metadata space and performs timed updates to the dimension table data in the metadata space, the open method ends and begins to acquire the raw data from upstream. When the subtask in the dimension table association operator performs the dimension table association operation, it performs dimension table association based on the dimension table data in the metadata space and the acquired raw data.

[0051] The method provided in this embodiment identifies a target subtask among multiple subtasks pointing to a unified dimension table join operator in the current Task Manager (TaskManager). After controlling the target subtask to send a query request to an external database and obtain the dimension table data sent by the external database, the method controls the target subtask to load the dimension table data into a static dimension table object set in the metadata space of the Task Manager. This avoids putting a lot of running pressure on the external database in a short period of time. The method controls the target subtask and the other subtasks to perform dimension table join operations based on the obtained raw data and the dimension table data in the static dimension table object. This allows all subtasks in the same Task Manager to share the same dimension table data, thereby reducing the number of dimension table "copies" in the same Task Manager process and saving physical memory resources.

[0052] The following specific embodiment will be used to describe in detail the dimension table association method for Flink in this application.

[0053] Example 2

[0054] Figure 4 A flowchart illustrating the Flink dimension table join method provided in the embodiments of this application. Figure 2 .like Figure 4 As shown, the method includes:

[0055] S401. Obtain the working status of each subtask in the current Task Manager;

[0056] Specifically, it retrieves the working status (such as idle or running) of all subtasks under the same dimension table association operator in the same TaskManager. If all subtasks are currently in the running state, the subtask that finishes first and is in the idle state is identified as the target subtask.

[0057] S402. If there is only one subtask whose working state is idle, then the currently idle subtask is identified as the target subtask.

[0058] Specifically, after obtaining the working status of all subtasks under the same dimension table association operator, if it is detected that there is a subtask with an idle working status among the multiple subtasks under the current same dimension table association operator, or if the subtask that first changes from the executing state to the idle state within a certain period of time is identified as the target subtask.

[0059] S403. If there are multiple subtasks in the working state that are idle, then identify the target subtask among the multiple idle subtasks.

[0060] Specifically, if multiple idle subtasks are detected among the multiple subtasks under the same dimension table association operator, the execution information of each idle subtask is obtained. The execution information is used to indicate the execution speed of the subtask, and the subtask with the fastest execution speed is identified as the target subtask. That is, all subtasks under the same dimension table association operator in the current task manager compete to acquire the lock when executing the open method, and the subtask that acquires the lock is identified as the target subtask. Of course, it can also be initialized by combining other common thread lock mechanisms, as long as the target subtask can be acquired.

[0061] S404. After controlling the target subtask to send a query request to the external database and obtaining the dimension table data sent by the external database, the target subtask loads the dimension table data into the static dimension table object.

[0062] Specifically, after obtaining the target subtask, the target subtask connects to an external database, such as the Hive data warehouse, through the JDBC (Java Database Connectivity) interface to query dimension table data. That is, it sends a query request to the external database. The external database returns the corresponding dimension table data found according to the query request to the target subtask. The target subtask loads the obtained dimension table data into a static dimension table object set in the metadata space, such as the JVM MetaSpace, so that other subtasks can call the dimension table data in the static dimension table object when performing dimension table association operations.

[0063] S405. Update the static dimension table object according to the static identifier of the static dimension table object;

[0064] Specifically, the static flag of the static dimension table object is used to indicate whether the subtask that acquired the lock has completed the data loading work of the static dimension table object. The static flag itself is usually initialized to a boolean value of false to indicate that the target subtask has not completed the dimension table data loading work of the static dimension table object and needs to continue to switch the static flag to true. When the static flag switches to true, it indicates that the target subtask has completed the dimension table data loading work of the static dimension table object. The static dimension table object can be initialized to different data structures. Generally speaking, a Map data structure is usually used for field filling scenarios, and a Set data structure is usually used for data filtering and quality control scenarios.

[0065] Furthermore, the static identifier of a static dimension table object can also be used solely to indicate that the static dimension table object has completed its first data loading operation. When the static identifier switches from the initial boolean value false to true, the static dimension table object has completed its first data loading operation. If it is detected that the static identifier remains at the initial boolean value false, the static identifier is checked again after waiting for ten seconds. This process is repeated until the static identifier is detected to switch to true, at which point the open method ends, the subtask begins to execute the dimension table join operation, and multiple subtasks under the same dimension table join operator begin to execute the dimension table join operation.

[0066] S406. If the static identifier indicates that the target subtask has completed the work of loading dimension table data of the static dimension table object, then after a first preset time, a new query request is sent to the external database, and the query result sent by the external database is obtained.

[0067] Specifically, when the static flag is detected to be true, the target subtask is instructed to complete the loading of dimension table data for the static dimension table object. At this time, the target subtask creates a timed dimension table update thread and sets a loop in the timed dimension table update thread, so as to connect to the external database where the original dimension table is located to query the dimension table data every certain period, that is, after the first preset time, such as 24 hours, and obtain the query results sent by the external database.

[0068] S407. Update the static dimension table object according to the query result;

[0069] Specifically, after obtaining the query results, which include new dimension table data and update information, if the update information in the query results indicates that the data in the external database has not changed, the target subtask continues to load the dimension table data into the static dimension table object. If the update information indicates that the data in the external database has changed, the target subtask loads the new dimension table data into the static dimension table object to complete the update of the static dimension table object.

[0070] S408. After the static identifier indicates that the target subtask has completed the work of loading the dimension table data of the static dimension table object, a new static identifier is obtained after a second preset time.

[0071] Specifically, when the static flag is detected to be true, the target subtask is instructed to complete the loading of dimension table data for the static dimension table object and to periodically update the dimension table data in the metadata space. After a second preset time, such as ten seconds, the static flag is continuously detected to obtain a new static flag. If the new static flag continues to indicate that the target subtask has completed the loading of dimension table data for the static dimension table object, a new static flag is obtained again after the second preset time. This process is repeated until a new static flag indicates that the target subtask has not completed the loading of dimension table data for the static dimension table object. The second preset time is much shorter than the first preset time.

[0072] S409. If the new static identifier indicates that the target subtask has not completed the work of loading the dimension table data of the static dimension table object, cancel the current target subtask and reconfirm the new target subtask in multiple subtasks other than the previous target subtask.

[0073] Specifically, if the static flag is detected to be false, indicating that the target subtask has not completed the work of loading dimension table data into the static dimension table object, then a new target subtask needs to be reconfirmed based on the new static flag. That is, the new target subtask needs to be reconfirmed among the other subtasks besides the previous target subtask. If the static flag is still true, then the open method ends and the subtask begins to execute the dimension table join operation.

[0074] S410. Control the target subtask and the remaining multiple subtasks to perform dimension table association operations based on the acquired raw data and the dimension table data in the static dimension table object;

[0075] Specifically, when each subtask performs dimension table association operations, after receiving the original data, each subtask can search in the dimension table data in the metadata space for whether there is a dimension table data record with the same association field as the original data record. If it is detected, the subtask will match the association field of the dimension table data record with the original data record, thereby associating the dimension table data record with the original data record.

[0076] The method provided in this embodiment identifies a target subtask among multiple subtasks pointing to a unified dimension table join operator in the current Task Manager (TaskManager). After controlling the target subtask to send a query request to an external database and obtain the dimension table data sent by the external database, the method controls the target subtask to load the dimension table data into a static dimension table object set in the metadata space of the Task Manager. This avoids putting a lot of running pressure on the external database in a short period of time. The method controls the target subtask and the other subtasks to perform dimension table join operations based on the obtained raw data and the dimension table data in the static dimension table object. This allows all subtasks in the same Task Manager to share the same dimension table data, thereby reducing the number of dimension table "copies" in the same Task Manager process and saving physical memory resources.

[0077] In this embodiment of the invention, electronic devices or main control devices can be divided into functional modules according to the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional module. It should be noted that the module division in this embodiment of the invention is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0078] Figure 5 This is a schematic diagram of the structure of a Flink-oriented dimension table association device provided in an embodiment of this application. Figure 5 As shown, the device 50 includes:

[0079] The acquisition module 501 is used to identify the target subtask among multiple subtasks in the current Task Manager;

[0080] The processing module 502 is used to control the target subtask to send a query request to an external database and, after obtaining the dimension table data sent by the external database, the target subtask loads the dimension table data into a static dimension table object, wherein the static dimension table object is set in the metadata space of the task manager.

[0081] The execution module 503 is used to control the target subtask and the other multiple subtasks to perform dimension table association operations based on the acquired raw data and the dimension table data in the static dimension table object.

[0082] Furthermore, the acquisition module 501 is specifically used to acquire the working status of each subtask in the current task manager, wherein the multiple subtasks are subtasks of the same dimension table association operator in the current task manager. If there is only one subtask whose working status is idle, then the currently idle subtask is identified as the target subtask. If there are multiple subtasks whose working status is idle, then the target subtask is identified among the multiple idle subtasks.

[0083] Furthermore, the acquisition module 501 is specifically used to acquire the execution information of each idle subtask, wherein the execution information is used to indicate the execution speed of the subtask, and the subtask with the fastest execution speed is identified as the target subtask.

[0084] Furthermore, the processing module 502 is specifically used to update the static dimension table object according to the static identifier of the static dimension table object. The static identifier is used to indicate whether the target subtask has completed the dimension table data loading work of the static dimension table object. If the static identifier indicates that the target subtask has completed the dimension table data loading work of the static dimension table object, then after a first preset time, a new query request is sent to the external database, and the query result sent by the external database is obtained. The static dimension table object is then updated according to the query result, and so on.

[0085] Furthermore, in processing module 502, the query result includes new dimension table data and update information. Specifically, if the update information indicates that the data in the external database has not changed, the target subtask maintains the loading of the dimension table data into the static dimension table object; if the update information indicates that the data in the external database has changed, the target subtask loads the new dimension table data into the static dimension table object to complete the update of the static dimension table object.

[0086] Furthermore, the processing module 502 is also used to obtain a new static identifier after a second preset time. If the new static identifier continues to indicate that the target subtask has completed the work of loading the dimension table data of the static dimension table object, then a new static identifier is obtained again after a second preset time. This process is repeated until a new static identifier indicates that the target subtask has not completed the work of loading the dimension table data of the static dimension table object. The second preset time is much shorter than the first preset time.

[0087] Furthermore, the processing module 502 is specifically used to cancel the current target subtask and reconfirm a new target subtask among multiple subtasks other than the previous target subtask.

[0088] The dimension table association device for Flink provided in this embodiment can execute the dimension table association method for Flink in the above embodiment. Its implementation principle and technical effect are similar, and will not be described again in this embodiment.

[0089] In the specific implementation of the aforementioned Flink-oriented dimension table association device, each module can be implemented as a processor. The processor can execute computer execution instructions stored in memory, causing the processor to execute the aforementioned Flink-oriented dimension table association method.

[0090] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 60 includes at least one processor 601 and a memory 602. The electronic device 60 also includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus 604.

[0091] In the specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, causing at least one processor 601 to execute the Flink-oriented dimension table association method as executed on the electronic device side as described above.

[0092] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0093] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0094] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.

[0095] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0096] The above description of the functions implemented by electronic devices and main control devices has introduced the solutions provided by the embodiments of the present invention. It is understood that, in order to implement the above functions, the electronic device or main control device includes hardware structures and / or software modules corresponding to the execution of each function. By combining the units and algorithm steps of the various examples described in the embodiments of the present invention, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present invention.

[0097] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the dimension table association method for Flink as described above.

[0098] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0099] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in an electronic device or a host device.

[0100] This application also provides a computer program product, comprising: a computer program stored in a readable storage medium, wherein at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the scheme provided in any of the above embodiments.

[0101] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dimension table joins in Flink, characterized in that, The method includes: Confirm the target subtask among the multiple subtasks in the current Task Manager; After controlling the target subtask to send a query request to an external database and obtain the dimension table data sent by the external database, the target subtask loads the dimension table data into a static dimension table object, wherein the static dimension table object is set in the metadata space of the task manager. The static dimension table object is updated based on its static identifier, wherein the static identifier is used to indicate whether the target subtask has completed the dimension table data loading work of the static dimension table object; If the static identifier indicates that the target subtask has completed the work of loading dimension table data of the static dimension table object, then after a first preset time, a new query request is sent to the external database, the query result sent by the external database is obtained, and the static dimension table object is updated according to the query result, and so on. If the static identifier indicates that the target subtask has completed the dimension table data loading operation for the static dimension table object, the method further includes: After a second preset time, a new static identifier is obtained. If the new static identifier continues to indicate that the target subtask has completed the work of loading the dimension table data of the static dimension table object, then after another second preset time, a new static identifier is obtained. This process is repeated until a new static identifier indicates that the target subtask has not completed the work of loading the dimension table data of the static dimension table object. The second preset time is much shorter than the first preset time. The target subtask and the other multiple subtasks are controlled to perform dimension table join operations based on the acquired raw data and the dimension table data in the static dimension table object.

2. The dimension table association method for Flink according to claim 1, characterized in that, The method for identifying the target subtask among multiple subtasks in the current task manager includes: Get the working status of each subtask in the current task manager, wherein the multiple subtasks are subtasks of the same dimension table association operator in the current task manager; If there is only one subtask in the working state that is idle, then the currently idle subtask is identified as the target subtask. If there are multiple subtasks in the working state that are idle, then the target subtask is identified among the multiple idle subtasks.

3. The dimension table association method for Flink according to claim 2, characterized in that, The method for identifying the target subtask among multiple idle subtasks includes: Obtain execution information for each idle subtask, wherein the execution information is used to indicate the execution speed of the subtask; The subtask with the fastest execution speed is identified as the target subtask.

4. The dimension table association method for Flink according to claim 1, characterized in that, The method of updating the static dimension table object based on the query result, wherein the query result includes new dimension table data and update information, includes: If the update information indicates that the data in the external database has not changed, then the target subtask maintains the loading of the dimension table data into the static dimension table object; If the update information indicates that the data in the external database has changed, the target subtask will load the new dimension table data into the static dimension table object to complete the update of the static dimension table object.

5. The method for dimension table association in Flink according to claim 1, characterized in that, If the new static identifier indicates that the target subtask has not completed the dimension table data loading work for the static dimension table object, the method includes: Cancel the current target subtask and reconfirm the new target subtask among multiple subtasks other than the previously stated target subtask.

6. A dimension table association device for Flink, characterized in that, include: The acquisition module is used to identify a target subtask among multiple subtasks in the current Task Manager; The processing module is used to control the target subtask to send a query request to an external database, and after obtaining the dimension table data sent by the external database, the target subtask loads the dimension table data into a static dimension table object, wherein the static dimension table object is set in the metadata space of the task manager. The processing module is specifically used to update the static dimension table object according to the static identifier of the static dimension table object, wherein the static identifier is used to indicate whether the target subtask has completed the dimension table data loading work of the static dimension table object; if the static identifier indicates that the target subtask has completed the dimension table data loading work of the static dimension table object, then after a first preset time, a new query request is sent to the external database, the query result sent by the external database is obtained, and the static dimension table object is updated according to the query result, and so on. The processing module is further configured to obtain a new static identifier after a second preset time. If the new static identifier continues to indicate that the target subtask has completed the work of loading the dimension table data of the static dimension table object, then a new static identifier is obtained again after a second preset time. This process is repeated until a new static identifier indicates that the target subtask has not completed the work of loading the dimension table data of the static dimension table object. The second preset time is much shorter than the first preset time. The execution module is used to control the target subtask and the other multiple subtasks to perform dimension table join operations based on the acquired raw data and the dimension table data in the static dimension table object.

7. A dimension table association device for Flink, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 5.

Images