Table service processing method and system
By adopting a separate plug-in mode table service processing method in Hudi data lake, decoupling table service jobs and data writing jobs, optimizing resource configuration, solving the problems of low resource utilization and short-term impact, achieving high availability and flexibility data management.
Patent Information
- Application Number
- CN202210333994.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-30
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2042-03-30
AI Technical Summary
In Hudi data lake, the coupling between table service jobs and data write jobs leads to low resource utilization, short-term impact on resources, and difficult to control flexibly, affecting the resource management and data life cycle management of the data lake.
A table service processing method is proposed, which is used to preprocess and write batches to the file system during the Flink checkpoint cycle, and submits real-time data through streaming write coordinator in a separate plug-in mode, and generates and executes according to the table service scheduling strategy. Plan, execute table services separately, and support switching of embedded synchronization, embedded asynchronous and separate plug-in modes.
It realizes decoupling and resource isolation between table service jobs and data write jobs, optimizes resource configuration, improves the availability of table service plug-in jobs, ensures the guarantee of high SLA jobs, supports dynamic scheduling strategies and asynchronous scheduling, and simplifies operations. Maintenance cost.
Smart Images
Figure CN114692585B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data lake technology, and in particular to a table service processing method, system, electronic device, and computer-readable storage medium. Background Art
[0002] In recent years, with the rise of data lakes, technologies such as Hudi (a data lake solution that supports record-level transaction update / deletion operations) have been booming and widely used in big data platforms of various companies. When building a data lake with Hudi as the base, based on its supported MoR (Merge on Read, Hudi's MoR mode supports real-time data reading) tables and their table services (data orchestration services provided by Hudi in MoR mode, such as compact, cluster, clean, etc.), users can easily establish a data governance framework for the entire data lifecycle management.
[0003] In the table service solution currently provided by Hudi, the table service job is generally coupled with the data writing job for processing. However, in fact, the functional emphasis of the table service job and the data writing job is different. The former focuses on batch data reorganization and arrangement, while the latter focuses on continuous data ingestion. They are two different types of operations. In production, due to different task characteristics, the coupling of table service jobs and data writing jobs will bring many problems such as low resource utilization, large short-term resource impact, and inconvenience for flexible control. Summary of the invention
[0004] The main purpose of this application is to propose a table service processing method, system, electronic device and computer-readable storage medium, aiming to solve the problem of how to decouple table service operations from data writing operations in a data lake and isolate resources.
[0005] To achieve the above purpose, the present application provides a table service processing method, the method comprising:
[0006] In each Flink checkpoint cycle, data is received from the source and preprocessed, and then written to the file system in batches by the streaming write operator;
[0007] When the table service is in a separate plug-in mode, after each cycle ends, the written real-time data is submitted through the streaming write coordinator, and the table service is scheduled according to the table service scheduling strategy to generate a corresponding execution plan;
[0008] The table service is executed separately according to the preset execution strategy and the execution plan.
[0009] Optionally, before accessing data from the source end, the method further includes:
[0010] A configuration entry is provided for selecting and switching the operation mode of the table service, wherein the operation mode includes an embedded synchronous mode, an embedded asynchronous mode, and a separate plug-in mode.
[0011] Optionally, performing table service scheduling according to the table service scheduling strategy and generating a corresponding execution plan includes:
[0012] Dynamically monitor the scheduling strategies released by the external platform, and perform the table service scheduling and generate the execution plan according to the latest monitored strategies.
[0013] Optionally, the performing table service scheduling according to the table service scheduling strategy and generating a corresponding execution plan further includes:
[0014] The data compression service and the data clustering service are scheduled, and a corresponding execution plan is generated and saved in the file system, wherein operators related to the execution of the data compression service and the data clustering service are not connected during the scheduling.
[0015] Optionally, the performing table service scheduling according to the table service scheduling strategy and generating a corresponding execution plan further includes:
[0016] The data cleaning service is scheduled, a corresponding execution plan is generated, and the plan is saved in the file system, wherein during the scheduling, the data cleaning service operator is connected and the execution function is closed.
[0017] Optionally, executing the table service separately according to the preset execution strategy and the execution plan includes:
[0018] The execution strategy of the table service is configured according to the priority of the table service and the resource status of the server cluster.
[0019] Optionally, the execution strategy includes executing the table service in any single, single batch or streaming mode at any predetermined time.
[0020] In addition, to achieve the above-mentioned purpose, the embodiment of the present application also provides a table service processing system, the system comprising:
[0021] The write module is used to access data from the source and preprocess it in each Flink checkpoint cycle, and the streaming write operator writes it to the file system in batches;
[0022] A scheduling module, which is used to submit the written real-time data through the streaming write coordinator after each cycle when the table service is in a separate plug-in mode, and to perform table service scheduling according to the table service strategy and generate a corresponding execution plan;
[0023] The execution module is used to independently execute the table service according to a preset execution strategy and the execution plan.
[0024] To achieve the above objectives, an embodiment of the present application also provides an electronic device, which includes: a memory, a processor, and a table service processing program stored in the memory and executable on the processor, and when the table service processing program is executed by the processor, the table service processing method as described above is implemented.
[0025] To achieve the above objectives, an embodiment of the present application further provides a computer-readable storage medium, on which a table service processing program is stored. When the table service processing program is executed by a processor, the table service processing method as described above is implemented.
[0026] The table service processing method, system, electronic device and computer-readable storage medium proposed in the embodiments of the present application have specially constructed a table service separate plug-in operation mode for Hudi, which supports separate plug-in Flink jobs for table service processing without affecting the accuracy of write jobs and final data. The underlying table service scheduling mechanism of Hudi is optimized, the availability of table service plug-in jobs is improved, and high SLA jobs (such as data write jobs) are guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 An application environment architecture diagram for implementing various embodiments of the present application;
[0028] Figure 2 for Figure 1 Specific functional diagram of each device in the embodiment;
[0029] Figure 3 A flowchart of a table service processing method proposed in the first embodiment of the present application;
[0030] Figure 4 for Figure 3 Detailed flow chart of step S202;
[0031] Figure 5 for Figure 3 A flowchart of another form of the table service processing method described in;
[0032] Figure 6 A flowchart of a table service processing method proposed in the second embodiment of the present application;
[0033] Figure 7 A schematic diagram of the hardware architecture of an electronic device proposed in the third embodiment of the present application;
[0034] Figure 8A schematic diagram of a module of a table service processing system proposed in the fourth embodiment of the present application;
[0035] Fig. 9 This is a module diagram of a table service processing system proposed in the fifth embodiment of the present application. DETAILED DESCRIPTION
[0036] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0037] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0038] See also Figure 1 , Figure 1 An application environment architecture diagram for implementing various embodiments of the present application. The present application can be applied to application environments including, but not limited to, a data lake (Hudi) management platform 2, a resource scheduling system (YARN) 4, and a file system (HDFS) 6.
[0039] The data lake management platform 2 is used for job configuration, table service scheduling strategy configuration, task scheduling services, etc.
[0040] The resource scheduling system 4 is used to schedule server resources to perform data writing operations, table service operations, etc. based on Flink (a streaming computing engine that supports stateful computing) and Hudi. The server can be a computing device such as a rack server, a blade server, a tower server or a cabinet server. Generally, the server resource is a server cluster composed of multiple servers.
[0041] The file system 6 is mainly used to store data (including data written in streaming mode and data after table service execution, etc.), execution plans, etc.
[0042] See also Figure 2 As shown, Figure 1 The specific functional diagram of each device in the figure is as follows. It is worth noting that Figure 2 The diagram shown is a schematic diagram of the table service operation in a single hanging mode. Figure 2 The specific functions of each part will be described in detail in the following embodiments and will not be repeated here.
[0043] The existing Hudi technology generally couples the data writing job and the table service job together, and uses an embedded synchronous or embedded asynchronous mode to implement the table service solution. Among them, the embedded asynchronous mode uses a special Flink operator to asynchronously process the operations corresponding to the table service, so as to isolate the data writing job and the table service job to a certain extent. However, in production, due to different task characteristics, the following problems may occur:
[0044] (1) The SLA (Service Level Agreement) requirements for table service jobs are relatively lower. If they are coupled with data writing in one job, sufficient resources have to be reserved during job configuration, which will occupy too many online high-priority resources, resulting in low resource utilization and cannot be optimized through resource scheduling or dedicated queues.
[0045] (2) Table services are generally intermittent batch processing, which adds many impulse glitches to the resource consumption curve, causing short-term resource impacts, an unsmooth usage curve, and even the cause of memory overflow in many jobs, which is not conducive to upper-level resource coordination and expert system expansion and contraction diagnosis.
[0046] (3) The strategies involved in the generation and operation of table service related operations are relatively flexible. If they are too coupled with continuous data writing tasks, it will be difficult to control them flexibly.
[0047] To address the above issues, Hudi provides a dedicated Flink job that schedules table service jobs through the scheduling system, schedules and generates table service plans, and then executes corresponding operations to achieve the goal of decoupling from data writing jobs and resource isolation. However, this solution also has the following defects:
[0048] (1) The scheduling and choreography of table service jobs does not have the ability to receive external strategies and is not easy to integrate with the company's intelligent scheduling, expert diagnosis and other systems.
[0049] (2) In the context of Flink on Hudi's incomplete support for concurrency, directly using this solution to schedule and execute table services will cause timeline consistency issues, resulting in data loss.
[0050] (3) The table service job still relies on the fine-grained configuration of the Hudi table, which does not meet the requirements of the managed task.
[0051] (4) Insufficient functionality leads to additional operation and maintenance costs, including:
[0052] Compared with the embedded operation mode, this solution uses different operators, resulting in different job DAGs (directed acyclic graphs abstracted from operators). During the mode switching process of other solutions as backups, operation deviations are very likely to occur. Currently, it only supports single-time processing of a single execution plan and lacks a general configuration entry.
[0053] (5) Although the Hudi kernel already has a sound Clean (data cleaning) operation capability, the table service layer currently only exposes the inline scheduling-immediate execution method, which results in both data writing and table service jobs including Clean operations, which will be called and executed in parallel. This not only causes a confusing architecture, but is also likely to lead to uncontrollable results.
[0054] In general, the original solution provided by the above Hudi technology is currently more suitable for use as a tool supplement. Since the table service job lacks the ability to coordinate with the data writing job, it cannot be applied to production in a single operating mode.
[0055] In response to the above problems, this application proposes a new table service processing method, which can run the table service job in a separate plug-in mode (standalone) to better decouple the table service job from the data writing job.
[0056] Embodiment 1
[0057] like Figure 3 As shown, it is a flowchart of a table service processing method proposed in the first embodiment of the present application. It can be understood that the flowchart in the embodiment of the present method is not used to limit the order of executing steps. As needed, some steps in the flowchart can also be added or deleted. The method is explained below by taking the resource scheduling system 4 as the execution subject as an example.
[0058] The method comprises the following steps:
[0059] S200, in each Flink checkpoint cycle, accesses data from the source and preprocesses it, and writes it into the file system in batches by the streaming write operator.
[0060] In this embodiment, the data writing job is mainly deployed based on the Flink engine, and the writing job will stream data into Hudi. The process of the data writing job is basically consistent with the commonly used scheme of the existing Hudi technology. In each Flink checkpoint cycle, data flows in from the source end and undergoes preprocessing steps such as format conversion, indexing, and file bucket ID allocation (such as Figure 2 The data is then accumulated in batches and written into the underlying file system by the streaming write operator.
[0061] S202, when the table service is in a separate plug-in mode, after each cycle ends, the written real-time data is submitted through the streaming write coordinator, and the table service is scheduled according to the table service scheduling strategy to generate a corresponding execution plan.
[0062] If the table service job is in a separate plug-in operation mode, the Compact (data compression service, merging the basic file and the incremental log file to generate a new version of the column storage file to improve query performance) / Cluster (data clustering service, clustering the file according to certain columns to re-layout and optimize query performance) related operators will no longer be connected when constructing the job DAG. In addition, although the Clean (data cleaning service, used to clean expired versions of files) operator is connected, its inline scheduling mode is turned off, only asynchronous scheduling is turned on, and the execution function is turned off.
[0063] For more details, see Figure 4 , is a detailed flow chart of the above step S202. It can be understood that the flow chart is not used to limit the order of executing the steps. As needed, some steps in the flow chart can be added or deleted. In this embodiment, the step S202 specifically includes:
[0064] S2020, after each Flink checkpoint cycle ends, the streaming write coordinator submits the real-time data written by the streaming write operator.
[0065] Specifically, after each Flink checkpoint cycle ends, the streaming write coordinator (see Figure 2 The Stream Write Coordinator in step S200 submits the instant data written in step S200 (by submitting the instant data). Instant is the time abstraction of performing operations on the table. Hudi maintains a timeline containing all instant operations on the data set at different instant times, thereby providing an instant view of the table and effectively supporting data retrieval in the order of arrival.
[0066] S2022, the streaming write coordinator schedules the data compression service and the data clustering service, generates a corresponding execution plan, and saves it to the file system.
[0067] After submitting the instant data, the streaming write coordinator schedules the Compact / Cluster table service according to the table service scheduling strategy, generates the corresponding execution plan, and saves it to the underlying file system in the form of table service instant. In the scheduling, no operators related to the Compact / Cluster table service are connected, that is, the Compact / Cluster table service is not actually executed, and only the execution plan is generated after scheduling.
[0068] S2024, the streaming write coordinator initializes and allocates a new write instant.
[0069] To start a new data writing process, the streaming write coordinator also needs to initialize and allocate a new write instant for the streaming write operator to write the next batch of preprocessed data.
[0070] S2026, scheduling data cleaning service, generating corresponding execution plan, and saving it to the file system.
[0071] After each Flink checkpoint cycle ends, the Clean operator will also be triggered, and the clean table service will be scheduled according to the configured scheduling strategy, and the corresponding execution plan instant will be generated and saved to the underlying file system. It is worth noting that although the Clean operator is connected at this time, its inline scheduling mode is turned off, only asynchronous scheduling is turned on, and the execution function is turned off. In other words, the Clean table service is not actually executed, and the execution plan is only generated after scheduling.
[0072] When a new checkpoint cycle starts, the above step S200 will continue to be executed until all data is processed (written).
[0073] In addition, it is worth noting that the above steps S2022 and S2026 involve Figure 2 The table service scheduler (TableServiceScheduler) in the data writing operation is embedded in the data writing operation, which includes a plug-in dynamic configuration module that can monitor the scheduling strategy released by the external platform in real time. That is to say, the above step S202 also includes the following steps:
[0074] Dynamically monitor the scheduling strategies released by the external platform, and perform the table service scheduling and generate the execution plan according to the latest monitored strategies.
[0075] Specifically, the step includes:
[0076] (1) Before performing the scheduling, the dynamic configuration module starts remote monitoring of the latest configuration according to the loaded plug-in.
[0077] (2) If pulling the latest configuration fails, obtain the backup strategy based on the fault tolerance mechanism.
[0078] (3) According to the current scheduling table service type, the scheduling policy of the table service is updated in a targeted manner and flushed into the table service scheduler.
[0079] (4) The table service scheduler intervenes in the table service scheduling and generates the execution plan according to the latest strategy.
[0080] S204: Execute the table service separately according to the preset execution strategy and the execution plan.
[0081] In this embodiment, the table service can be executed at any scheduled time in any single (single instant), single batch or stream mode, just by passing in the path of the Hudi table. That is to say, this embodiment supports multiple execution strategies when executing the table service separately for the separate plug-in mode of the table service, such as executing the latest instant once at a scheduled time, or executing all instants once at a scheduled time, or executing them in parallel with the data writing job.
[0082] The execution strategy can be called up during off-peak hours according to the priority configured when the table service is submitted and the resource status of the server cluster that can be used to run the data writing service and the table service. This process is imperceptible to the user.
[0083] See also Figure 5 FIG. 1 is a flowchart of another form of the table service processing method. Figure 5 The table service processing method proposed in this embodiment can be combined with Figure 2-5 To understand. Figure 5 The specific functions and process steps of each operator are the same as above. Figure 3-4 The relevant instructions are similar and will not be repeated here.
[0084] Since the execution process of the table service is implemented through a separate plug-in mode, for high SLA data writing jobs ( Figure 5 The upper and middle parts) can be processed by a higher-performance service cluster (or processed by a high-priority task queue), while the table service jobs that are executed separately ( Figure 5 The middle and lower parts) can be processed by a service cluster with lower performance (or processed during idle time with lower load), thereby optimizing resource allocation and improving resource utilization.
[0085] The table service processing method proposed in this embodiment specifically builds a table service standalone plug-in operation mode for Hudi, supporting a separate plug-in Flink job for table service processing. Reconstruct the table service job so that the DAG of the plug-in table service operator and the write job part in the embedded mode are kept linear, without affecting the accuracy of the write job and the final data. Optimize the underlying table service scheduling mechanism of Hudi, and for the Clean operation, the external interface achieves the use paradigm of scheduling-execution separation, so that all table service operations eventually support the plug-in mode and asynchronous scheduling. For the table service scheduler, a plug-in dynamic configuration module is designed, which can monitor the external configuration platform in real time, refresh the Hudi table service scheduling strategy, support dynamic modification of the table service scheduling strategy, and facilitate service integration with the data lake platform. The operation mode of the extended table service execution can realize the flow running and single full batch running modes, as well as more general fine-grained configuration. Improve the availability of the table service plug-in job, achieve the guarantee of high SLA jobs (such as data writing jobs), and can be easily hosted on the platform.
[0086] Embodiment 2
[0087] like Figure 6 As shown, it is a flowchart of a table service processing method proposed in the second embodiment of the present application. In the second embodiment, the table service processing method further includes step S300 based on the above-mentioned first embodiment. It can be understood that the flowchart in the embodiment of this method is not used to limit the order of executing the steps. Some steps in the flowchart can also be added or deleted as needed.
[0088] The method comprises the following steps:
[0089] S300, providing a configuration entry for selecting and switching the operation mode of the table service.
[0090] The operation modes include embedded synchronization, embedded asynchronous and separate plug-in modes. In this embodiment, the Hudi data write operation generation process and its operators are reconstructed to support embedded synchronization, embedded asynchronous and newly added separate plug-in modes, as well as smooth switching between various modes. Through the provided configuration entry (configuration operation mode switch), users can configure properties and select the operation mode corresponding to the table service operation according to actual needs to better meet production requirements.
[0091] Moreover, the above three modes can back up and switch between each other.
[0092] S302: In each Flink checkpoint cycle, data is received from the source and preprocessed, and then written into the file system in batches by the streaming write operator.
[0093] In this embodiment, the data writing job is mainly deployed based on the Flink engine, and the writing job will stream data into Hudi. The process of the data writing job is basically consistent with the commonly used scheme of the existing Hudi technology. In each Flink checkpoint cycle, data flows in from the source, and is accumulated in batches after preprocessing such as format conversion, indexing, and file bucket ID allocation, and is written to the underlying file system batch by batch by the streaming writing operator.
[0094] S304, when the table service is in a separate plug-in mode, after each cycle ends, the written real-time data is submitted through the streaming write coordinator, and the table service is scheduled according to the table service scheduling strategy to generate a corresponding execution plan.
[0095] If the table service job is in a separate plug-in operation mode, the Compact / Cluster related operators will no longer be connected when constructing the job DAG. In addition, although the Clean operator is connected, its inline scheduling mode is turned off, only asynchronous scheduling is turned on, and the execution function is turned off.
[0096] Specifically, this step includes:
[0097] (1) After each Flink checkpoint cycle ends, the streaming write coordinator commits the real-time data written by the streaming write operator.
[0098] Specifically, after the cycle of each Flink checkpoint ends, the streaming write coordinator submits the instant data written in the previous step (submits the instant data by submitting instant).
[0099] (2) The streaming write coordinator schedules the data compression service and the data clustering service, generates a corresponding execution plan, and saves it in the file system.
[0100] After submitting the instant data, the streaming write coordinator schedules the Compact / Cluster table service according to the table service scheduling strategy, generates the corresponding execution plan, and saves it to the underlying file system in the form of table service instant. In the scheduling, no operators related to the Compact / Cluster table service are connected, that is, the Compact / Cluster table service is not actually executed, and only the execution plan is generated after scheduling.
[0101] (3) The streaming write coordinator initializes and allocates a new write instant.
[0102] To start a new data writing process, the streaming write coordinator also needs to initialize and allocate a new write instant for the streaming write operator to write the next batch of preprocessed data.
[0103] (4) Scheduling data cleaning services, generating corresponding execution plans, and saving them in the file system.
[0104] After each Flink checkpoint cycle ends, the Clean operator will also be triggered, and the clean table service will be scheduled according to the configured scheduling strategy, and the corresponding execution plan instant will be generated and saved to the underlying file system. It is worth noting that although the Clean operator is connected at this time, its inline scheduling mode is turned off, only asynchronous scheduling is turned on, and the execution function is turned off. In other words, the Clean table service is not actually executed, and the execution plan is only generated after scheduling.
[0105] When a new checkpoint cycle starts, the above step S302 will continue to be executed until all data is processed (written).
[0106] In addition, the above step S304 also includes: dynamically monitoring the scheduling strategy released by the external platform, and performing the table service scheduling and generating the execution plan according to the latest monitored strategy.
[0107] (1) Before performing the scheduling, the dynamic configuration module starts remote monitoring of the latest configuration according to the loaded plug-in.
[0108] (2) If pulling the latest configuration fails, obtain the backup strategy based on the fault tolerance mechanism.
[0109] (3) According to the current scheduling table service type, the scheduling policy of the table service is updated in a targeted manner and flushed into the table service scheduler.
[0110] (4) The table service scheduler intervenes in the table service scheduling and generates the execution plan according to the latest strategy.
[0111] S306: Execute the table service separately according to the preset execution strategy and the execution plan.
[0112] In this embodiment, the table service can be executed at any scheduled time in any single (single instant), single batch or stream mode, just by passing in the path of the Hudi table. That is to say, this embodiment supports multiple execution strategies when executing the table service separately for the separate plug-in mode of the table service, such as executing the latest instant once at a scheduled time, or executing all instants once at a scheduled time, or executing them in parallel with the data writing job.
[0113] The execution strategy can be called up during off-peak hours according to the priority configured when the table service is submitted and the resource status of the server cluster that can be used to run the data writing service and the table service. This process is imperceptible to the user.
[0114] The table service processing method proposed in this embodiment can reconstruct the Hudi data writing job generation process and its operators to support three modes: embedded synchronous, embedded asynchronous, and newly added separate plug-in, and support smooth switching between multiple table service modes including separate modes, while maintaining consistency in the operator structure.
[0115] Embodiment 3
[0116] like Figure 7 FIG. 2 is a schematic diagram of the hardware architecture of an electronic device 20 according to the third embodiment of the present application. In this embodiment, the electronic device 20 may include, but is not limited to, a memory 21, a processor 22, and a network interface 23 that can be interconnected via a system bus. It should be noted that Figure 7 Only the electronic device 20 is shown with components 21 - 23 , but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.
[0117] The memory 21 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 21 can be an internal storage unit of the electronic device 20, such as a hard disk or memory of the electronic device 20. In other embodiments, the memory 21 can also be an external storage device of the electronic device 20, such as a plug-in hard disk equipped on the electronic device 20, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 21 can also include both the internal storage unit of the electronic device 20 and its external storage device. In this embodiment, the memory 21 is generally used to store the operating system and various application software installed in the electronic device 20, such as the program code of the table service processing system 60, etc. In addition, the memory 21 can also be used to temporarily store various types of data that have been output or are to be output.
[0118] The processor 22 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 22 is generally used to control the overall operation of the electronic device 20. In this embodiment, the processor 22 is used to run the program code stored in the memory 21 or process data, such as running the table service processing system 60.
[0119] The network interface 23 may include a wireless network interface or a wired network interface. The network interface 23 is generally used to establish a communication connection between the electronic device 20 and other electronic devices.
[0120] Embodiment 4
[0121] like Figure 8 As shown, a module diagram of a table service processing system 60 is proposed in the fourth embodiment of the present application. The table service processing system 60 can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment.
[0122] In this embodiment, the table service processing system 60 includes:
[0123] The writing module 600 is used to access data from the source end and pre-process it in each Flink checkpoint cycle, and write it into the file system in batches by the streaming writing operator.
[0124] In this embodiment, the data writing job is mainly deployed based on the Flink engine, and the writing job will stream data into Hudi. The process of the data writing job is basically consistent with the commonly used scheme of the existing Hudi technology. In each Flink checkpoint cycle, data flows in from the source, and is accumulated in batches after preprocessing such as format conversion, indexing, and file bucket ID allocation, and is written to the underlying file system batch by batch by the streaming writing operator.
[0125] The scheduling module 602 is used to submit the written real-time data through the streaming write coordinator after each cycle when the table service is in a separate plug-in mode, and to perform table service scheduling according to the table service strategy to generate a corresponding execution plan.
[0126] If the table service job is in a separate plug-in operation mode, the Compact / Cluster related operators will no longer be connected when constructing the job DAG. In addition, although the Clean operator is connected, its inline scheduling mode is turned off, only asynchronous scheduling is turned on, and the execution function is turned off.
[0127] In addition, the scheduling strategies released by the external platform can also be monitored dynamically, and the table service scheduling and the execution plan can be generated according to the latest strategies monitored.
[0128] The execution module 604 is used to execute the table service separately according to the preset execution strategy and the execution plan.
[0129] In this embodiment, the table service can be executed at any scheduled time in any single (single instant), single batch or stream mode, just by passing in the path of the Hudi table. That is to say, this embodiment supports multiple execution strategies when executing the table service separately for the separate plug-in mode of the table service, such as executing the latest instant once at a scheduled time, or executing all instants once at a scheduled time, or executing them in parallel with the data writing job.
[0130] The execution strategy can be called up during off-peak hours according to the priority configured when the table service is submitted and the resource status of the server cluster that can be used to run the data writing service and the table service. This process is imperceptible to the user.
[0131] Embodiment 5
[0132] like Fig. 9 FIG. 6 is a block diagram of a table service processing system 60 according to the fifth embodiment of the present application. In this embodiment, the table service processing system 60 includes a selection module 606 in addition to the writing module 600 , the scheduling module 602 , and the execution module 604 in the fourth embodiment.
[0133] The selection module 606 is used to provide a configuration entry for selecting and switching the operation mode of the table service.
[0134] The operation modes include embedded synchronization, embedded asynchronous and separate plug-in modes. In this embodiment, the Hudi data write operation generation process and its operators are reconstructed to support embedded synchronization, embedded asynchronous and newly added separate plug-in modes, as well as smooth switching between various modes. Through the provided configuration entry (configuration operation mode switch), users can configure properties and select the operation mode corresponding to the table service operation according to actual needs to better meet production requirements.
[0135] Moreover, the above three modes can back up and switch between each other.
[0136] Embodiment 6
[0137] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores a table service processing program, and the table service processing program can be executed by at least one processor to enable the at least one processor to perform the steps of the table service processing method as described above.
[0138] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0139] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0140] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.
[0141] The above are only preferred embodiments of the embodiments of the present application, and are not intended to limit the patent scope of the embodiments of the present application. Any equivalent structure or equivalent process transformation made using the description and drawings of the embodiments of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the embodiments of the present application.
Claims
1. A table service processing method, characterized in that: The method comprises: In each Flink checkpoint cycle, data is received from the source and preprocessed, and then written to the file system in batches by the streaming write operator; When the table service is in a separate plug-in mode, after each cycle ends, the written real-time data is submitted through the streaming write coordinator, and the table service is scheduled according to the table service scheduling strategy to generate a corresponding execution plan, including: scheduling data compression service and data clustering service, generating a corresponding execution plan, and saving it to the file system, wherein, during the scheduling, operators related to the execution of the data compression service and the data clustering service are not connected; The table service is executed separately according to the preset execution strategy and the execution plan.
2. The table service processing method according to claim 1, characterized in that: The method further comprises, before accessing data from the source end: A configuration entry is provided for selecting and switching the operation mode of the table service, wherein the operation mode includes an embedded synchronous mode, an embedded asynchronous mode, and a separate plug-in mode.
3. The table service processing method according to claim 1 or 2, characterized in that: The performing table service scheduling according to the table service scheduling strategy and generating a corresponding execution plan further includes: Dynamically monitor the scheduling strategies released by the external platform, and perform the table service scheduling and generate the execution plan according to the latest monitored strategies.
4. The table service processing method according to claim 1, characterized in that: The performing table service scheduling according to the table service scheduling strategy and generating a corresponding execution plan further includes: The data cleaning service is scheduled, a corresponding execution plan is generated, and the plan is saved in the file system, wherein during the scheduling, the data cleaning service operator is connected and the execution function is closed.
5. The table service processing method according to claim 1, characterized in that: The separately executing the table service according to the preset execution strategy and the execution plan includes: The execution strategy of the table service is configured according to the priority of the table service and the resource status of the server cluster.
6. The table service processing method according to claim 5, characterized in that: The execution strategy includes executing the table service in any single, batch or streaming mode at any predetermined time.
7. A table service processing system, characterized in that: The system comprises: The write module is used to access data from the source and preprocess it in each Flink checkpoint cycle, and the streaming write operator writes it to the file system in batches; A scheduling module, which is used to submit the written real-time data through the streaming write coordinator after each cycle when the table service is in a separate plug-in mode, and to schedule the table service according to the table service strategy, and generate a corresponding execution plan, including: scheduling data compression service and data clustering service, generating a corresponding execution plan, and saving it to the file system, wherein operators related to the execution of the data compression service and the data clustering service are not connected during the scheduling; The execution module is used to independently execute the table service according to a preset execution strategy and the execution plan.
8. An electronic device, characterized in that: The electronic device comprises: a memory, a processor, and a table service processing program stored in the memory and executable on the processor, wherein the table service processing program implements the table service processing method according to any one of claims 1 to 6 when executed by the processor.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a table service processing program, and when the table service processing program is executed by a processor, the table service processing method according to any one of claims 1 to 6 is implemented.