Data compression management method, system, device and storage medium

By automatically managing the compression tasks of Hudi data tables, the problems of manual maintenance and resource waste are solved, efficient and reliable data compression management is achieved, and costs and difficulties are reduced.

CN118820194BActive Publication Date: 2024-11-29KINCHENG BANK OF TIANJIN CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411298073.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-18
Publication Date
2024-11-29
Estimated Expiration
2044-09-18

AI Technical Summary

Technical Problem

The compression task of Hudi data files requires manual maintenance and cannot automatically perceive new table scenarios, resulting in too large data files and slow query. The scheduling method is a full-scale strategy, resulting in waste of resources, and the inability to process the task status in a refined manner, which increases maintenance costs and difficulty. .

Method used

Provides a data compression management method, which generates compression plans by scanning Hudi data tables, detects and filters effective plans, builds concurrent compression tasks, monitors operating status, and realizes automated management and resource optimization.

Benefits of technology

It realizes the automated management of offline compression tasks of Hudi data tables, reduces the need for manual intervention, improves management efficiency and reliability, supports resource utilization maximization and task status visualization, and reduces maintenance costs and resource waste.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118820194B_ABST
    Figure CN118820194B_ABST
Patent Text Reader

Abstract

The present application relates to the technical field of data management, and provides a data compression management method, system, device and storage medium, the data compression management method comprising: scanning the initial Hudi data table and generating a compression plan; detecting the candidate Hudi data table with the compression plan, and filtering to obtain the target Hudi data table with the effective compression plan; constructing a compression task based on the target Hudi data table, detecting whether all target Hudi data tables have the same name, and retaining a target Hudi data table with the same name; submitting the compression task to the task compression container through the task submission controller to run the compression task corresponding to the filtered target Hudi data table. The data compression management method improves the level of automated management of Hudi data table compression tasks, and makes the management of Hudi offline compression tasks more efficient and reliable.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a data compression management method, system, device and storage medium. Background Art

[0002] Offline data compression mainly merges hudi avro type data log files to generate parquet data basic files, thereby improving data query efficiency. Currently, the compression of Hudi data files is mainly carried out through independent deployment and scheduling of table-level compression tasks, which relies on the joint management of the scheduling system and multiple modules of the task platform.

[0003] However, the current compression method of Hudi data files has the following problems: compression tasks require manual maintenance, and it cannot support automatic perception and timely construction of compression tasks for newly added table scenarios, resulting in data files that are too large and slow queries. This undoubtedly increases the cost of manual maintenance and makes the entire process cumbersome, and delayed compression affects the size of data files, which directly affects data query efficiency; the current scheduling method for compression tasks adopts a full-volume strategy, which leads to a large number of invalid tasks, resulting in unnecessary waste of resources and greatly reducing the operating efficiency of the system; for the status of compression tasks, the current system cannot provide refined failure situation distinction, which makes it difficult for us to locate and solve problems in a targeted manner when dealing with problems, increasing the difficulty of solving problems; because the modules of the task platform involve more content, task scheduling mainly relies on scripts and interface requests, which makes the cost relatively high when changes are needed, which is not conducive to flexible adjustment and optimization of the system. Summary of the invention

[0004] In view of this, the purpose of this application is to provide a data compression management method, system, device and storage medium to help improve the level of automated management of Hudi data table offline compression tasks, reduce the need for manual intervention, and make the management of Hudi offline compression tasks more efficient and reliable.

[0005] In a first aspect, an embodiment of the present application provides a data compression management method, including:

[0006] Scan the initial Hudi data tables and generate a compression plan for each of the initial Hudi data tables;

[0007] Detect candidate Hudi data tables with compression plans and filter them to obtain target Hudi data tables with valid compression plans;

[0008] Building a compression task based on the target Hudi data table, submitting the compression task to the cache queue to detect whether all the target Hudi data tables have duplicate names, and retaining a target Hudi data table with the duplicate name to obtain a filtered target Hudi data table;

[0009] Submit the compression task corresponding to the filtered target Hudi data table to the task submission controller, so as to submit the corresponding compression task to the task compression container through the task submission controller, thereby running the compression task corresponding to the filtered target Hudi data table.

[0010] In combination with the first aspect, an embodiment of the present application provides a first possible implementation of the first aspect, wherein the step of submitting the corresponding compression task to the task compression container through the task submission controller further includes:

[0011] Concurrently manage the plurality of compression tasks through a task concurrency controller, and detect whether the running environment of the task compression container of the compression task to be run is ready through the task submission controller;

[0012] If the operating environment of the task compression container is abnormal, a new task compression container is created to run the compression task to be run.

[0013] In combination with the first aspect, the embodiment of the present application provides a second possible implementation of the first aspect, wherein the task submission controller submits the corresponding compression task to the task compression container, and then further includes:

[0014] Configuring a task thread pool in the task compression container, and controlling the number of parallel operations of the compression task according to the number of core threads of the task thread pool;

[0015] Monitor the running status of the compression task through a task listener, and visualize the running status of the compression task;

[0016] When it is detected that the compression task fails to run, an alarm is triggered, and the compression task that fails to run is scheduled to the task concurrency controller.

[0017] In combination with the first aspect, the embodiment of the present application provides a third possible implementation of the first aspect, wherein the detecting a candidate Hudi data table with a compression plan includes:

[0018] Cache the candidate Hudi data tables with the compression plan, and determine the submission batch of the candidate Hudi data tables;

[0019] The compression plan detection is performed on the candidate Hudi data tables of a single batch submitted asynchronously triggered by the programming model, and the compression plan detection is performed on the candidate Hudi data tables of multiple batches submitted in parallel.

[0020] In combination with the first aspect, the embodiment of the present application provides a fourth possible implementation of the first aspect, wherein the filtering to obtain a target Hudi data table with a valid compression plan includes:

[0021] Filter the candidate Hudi data tables with valid compression plans according to the filtering rules, and delete the candidate Hudi data tables with invalid compression plans;

[0022] The filtering rules include:

[0023] The identifier of the compression plan is not empty, and the timestamp of the compression plan is not within the timestamp range of the rollback request.

[0024] In combination with the first aspect, an embodiment of the present application provides a fifth possible implementation of the first aspect, wherein the queue thread pool corresponding to the cache queue customizes the duplicate name detection logic and filtering logic of the target Hudi data table, and the detection of whether all the target Hudi data tables have duplicate names, and retaining a target Hudi data table with the duplicate name, includes:

[0025] The duplicate name detection logic and the filtering logic are called to detect and filter the target Hudi data table with the duplicate name, and only one target Hudi data table with the duplicate name is retained.

[0026] In combination with the first aspect, the embodiment of the present application provides a sixth possible implementation of the first aspect, wherein the compression task corresponding to the filtered target Hudi data table is submitted to the task submission controller, and then further includes:

[0027] The compression task cache operation of cleaning the queue thread pool is triggered to delete the compression tasks that have been submitted, while the compression tasks that have not been submitted are waiting to be submitted.

[0028] In a second aspect, an embodiment of the present application further provides a data compression management system, including:

[0029] A compression task coordination module, used to scan the initial Hudi data table and generate a compression plan for each of the initial Hudi data tables;

[0030] The compression task coordination module is also used to detect candidate Hudi data tables with compression plans and filter target Hudi data tables with valid compression plans;

[0031] A task management module is used to build a compression task based on the target Hudi data table, submit the compression task to the cache queue, detect whether all the target Hudi data tables have duplicate names, and retain a target Hudi data table with the duplicate name to obtain a filtered target Hudi data table;

[0032] The task management module is also used to submit the compression task corresponding to the filtered target Hudi data table to the task submission controller, so as to submit the corresponding compression task to the task compression container through the task submission controller, thereby running the compression task corresponding to the filtered target Hudi data table.

[0033] In a third aspect, an embodiment of the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program runs on the processor, the data compression management method is executed.

[0034] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is run on a processor, the data compression management method is executed.

[0035] The embodiments of the present application have the following beneficial effects:

[0036] A data compression management method, system, device and storage medium according to an embodiment of the present application, the data compression management method comprising: scanning a Hudi data table and generating a compression plan for each of the Hudi data tables; detecting candidate Hudi data tables with compression plans, and filtering to obtain a target Hudi data table with a valid compression plan; constructing a compression task based on the target Hudi data table, submitting the compression task to a cache queue to detect whether all the target Hudi data tables have duplicate names, and retaining a target Hudi data table with the duplicate name to obtain a filtered Hudi data table; submitting the compression task corresponding to the filtered Hudi data table to a task submission controller, so as to submit the corresponding compression task to a task compression container through the task submission controller, so as to run the compression task corresponding to the filtered Hudi data table.

[0037] Based on the above scheme, the data compression management method provided in this application can automatically scan the newly added Hudi data table and automatically trigger the construction of offline compression tasks to ensure that the new Hudi data table can be compressed in time; the data compression management method supports filtering Hudi data tables for which compression plans have not been generated, maximizes resource utilization and reduces task failure rate, and supports visualization of the running status of compression tasks, timely discovers and solves potential problems, and ensures compression stability; the data compression management method realizes convergence compression tasks, integrates compression tasks into the coordinator for unified management, automatically builds the entire process and automatically schedules execution, reduces the complexity of the module architecture, and reduces maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope of protection of the present invention. In each of the drawings, similar components are numbered similarly.

[0039] Figure 1 A first flow chart of the data compression management method provided in an embodiment of the present application is shown;

[0040] Figure 2 A second flow chart of the data compression management method provided in an embodiment of the present application is shown;

[0041] Figure 3 A third flow chart of the data compression management method provided in the embodiment of the present application is shown;

[0042] Figure 4 A structural schematic diagram of a data compression management system provided in an embodiment of the present application is shown.

[0043] Description of main component symbols:

[0044] 200 - data compression management system; 210 - compression task coordination module; 220 - task management module; 230 - timing scheduling module; 240 - alarm module. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0046] The components of the embodiments of the present invention generally described and shown in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0047] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present invention, are intended only to indicate specific features, numbers, steps, operations, elements, components or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing items or adding the possibility of one or more features, numbers, steps, operations, elements, components or combinations of the foregoing items.

[0048] Furthermore, the terms “first”, “second”, “third”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0049] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present invention belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present invention.

[0050] Example 1

[0051] In this embodiment, referring to FIG1 , the data compression management method includes the following steps:

[0052] S110: Scan the initial Hudi data table and generate a compression plan for each initial Hudi data table.

[0053] Demonstrably, in the field of Internet finance, the streaming data lake platform (Hudi) can be used to quickly import, clean, convert and process data. For example, the streaming data lake platform (Hudi) can import user transaction data into the Hudi data table, and use the data management and indexing functions provided by the streaming data lake platform (Hudi) to conveniently and quickly query and statistically analyze transaction orders. The client API of the distributed file system (HDFS) accesses and scans all Hudi data tables under the database storage directory, and can support automatic identification of additions and deletions to Hudi data tables without manual intervention. The system responsibility chain receives concurrently submitted requests for the scanned Hudi data table, and actively triggers the generation of a Hudi data table compression plan. The empty plan table of each Hudi data table executes the generated compression plan. Among them, the system responsibility chain is a behavioral design pattern, which is used to send the concurrently submitted requests of the Hudi data table along the processor chain of the compression plan. After receiving the request, the processor generates a compression plan for the Hudi data table.

[0054] It can be understood that the system responsibility chain actively triggers the generation of a compression plan for the Hudi data table, avoiding the scenario where no incremental data is written after the full synchronization of the Hudi data table, resulting in the inability to generate a compression plan in time, thereby minimizing the impact on query performance.

[0055] S120: Detect candidate Hudi data tables with compression plans, and filter to obtain target Hudi data tables with valid compression plans.

[0056] Demonstratively, the data compression management method starts an asynchronous task in the asynchronous programming model (CompletableFuture) to detect the compression plan of the Hudi data table. The data compression management method first creates an asynchronous programming model (CompletableFuture) object to start the asynchronous task; then writes code to detect the compression plan in the Hudi data table and return the plan result; finally, after completing the asynchronous task, the detection result of the compression plan is logged, and the callback function of the asynchronous programming model (CompletableFuture) will be triggered to further process the result of the asynchronous task. The callback function of the asynchronous programming model (CompletableFuture) is used to conveniently process the result of the asynchronous task, thereby realizing the function of asynchronously triggering the compression plan detection of the Hudi data table.

[0057] For example, in one embodiment, the data compression management method detects candidate Hudi data tables with compression plans, including: caching candidate Hudi data tables with compression plans and determining submission batches of candidate Hudi data tables; asynchronously triggering compression plan detection for a single batch of candidate Hudi data tables submitted through a programming model, and performing compression plan detection on multiple batches of submitted candidate Hudi data tables in parallel.

[0058] It can be understood that this data compression management method asynchronously triggers the Hudi data table compression plan detection through the asynchronous programming model (CompletableFuture) and synchronizes the blocking mode between batches to control the compression plan detection of a single batch of Hudi data tables, avoid serial blocking and waiting between batches, strictly control the compression plan detection of multiple batches of Hudi data tables concurrently, and reduce the service load.

[0059] Exemplarily, the data compression management method can set filtering rules for effective compression plans, filter out Hudi data tables with effective compression plans, delete data tables with invalid compression plans, and avoid submitting compression tasks corresponding to data tables with invalid compression plans. As a result, compression tasks are no longer started for Hudi data tables that do not need to be compressed, thereby avoiding waste of resources, reducing the failure of compression tasks caused by the status of Hudi data tables, and improving the stability of the overall system.

[0060] For example, in one embodiment, the data compression management method filters to obtain a target Hudi data table with a valid compression plan, including: filtering to obtain a candidate Hudi data table with a valid compression plan according to a filtering rule, and deleting a candidate Hudi data table with an invalid compression plan; wherein the filtering rule includes: the identifier of the compression plan is not empty, and the timestamp of the compression plan is not within the timestamp range of the rollback request.

[0061] S130: A compression task is constructed based on the target Hudi data table, and the compression task is submitted to the cache queue to detect whether all target Hudi data tables have duplicate names, and a target Hudi data table with the duplicate name is retained to obtain a filtered target Hudi data table.

[0062] Demonstratively, the task manager is used to monitor and consume custom deduplication task queues to prevent repeated submission and execution of tasks with the same name. The queue thread pool is used to cache compression plans and customize the Hudi data table duplicate name filtering logic in its own thread submission logic to detect whether there are compression tasks with the same name in the cache queue before submitting the compression task to the task submission controller. The queue thread pool is also used to trigger the cleanup of the internal compression task cache operation after the queue task thread (JobRunnable) successfully submits the compression task.

[0063] For example, in one embodiment, the duplicate name detection logic and filtering logic of the target Hudi data table are customized and added to the queue thread pool corresponding to the cache queue. The data compression management method detects whether all target Hudi data tables have duplicate names, and retains a target Hudi data table with a duplicate name, including: calling the duplicate name detection logic and filtering logic to detect and filter the target Hudi data tables with duplicate names, and only retaining one target Hudi data table with a duplicate name.

[0064] For example, in one embodiment, the data compression management method submits the compression task corresponding to the filtered Hudi data table to the task submission controller, and then also includes: triggering the compression task cache operation of the cleanup queue thread pool to delete the compression tasks that have been submitted, while the compression tasks that have not been submitted are waiting to be submitted.

[0065] Exemplarily, before submitting the compression task corresponding to the filtered Hudi data table to the task compression container, the task manager detects the status of the flink session window in the task compression container. The flink session window is used to record the status information in the compression task when a failure occurs during the execution of the compression task or the task needs to be interrupted, so that the task processing can be restarted after the compression task is restored, avoiding repeated compression processing of the Hudi data table.

[0066] For example, in one embodiment, before the task submission controller submits the corresponding compression task to the task compression container, Figure 2 As shown, the data compression management method also includes the following steps:

[0067] S131: concurrently manage multiple compression tasks through a task concurrency controller, and detect whether the running environment of the task compression container of the compression task to be run is ready through a task submission controller;

[0068] S132: If the running environment of the task compression container is abnormal, create a new task compression container to run the compression task to be run.

[0069] S140: Submit the compression task corresponding to the filtered target Hudi data table to the task submission controller, so as to submit the corresponding compression task to the task compression container through the task submission controller, thereby running the compression task corresponding to the filtered target Hudi data table.

[0070] Demonstratively, the task compression container softly controls the number of concurrent compression tasks by configuring the number of core threads in the task thread pool. The maximum number of tasks running in parallel is not greater than the number of core threads to prevent excessive resource consumption. After the compression task is successfully submitted, the task listener begins to monitor the running status of the compression task in the flink session window. The running status of the compression task includes the start, run, pause, and termination of the compression task. When the compression task fails abnormally, an alarm is triggered actively. The running status of the compression task can be visualized through the dashboard, monitoring system, or log records to timely discover the execution of the compression task, monitor the progress and status of the compression task, and handle abnormal situations in a timely manner to ensure the stability and reliability of the compression task.

[0071] For example, in one embodiment, the task submission controller submits the corresponding compression task to the task compression container, such as Figure 3 As shown, the data compression management method also includes the following steps:

[0072] S141: configuring a task thread pool in the task compression container, and controlling the number of parallel running compression tasks according to the number of core threads in the task thread pool;

[0073] S142: monitoring the running status of the compression task through the task listener, and visually displaying the running status of the compression task;

[0074] S143: When it is detected that the compression task fails to run, an alarm is triggered, and the compression task that failed to run is scheduled to the task concurrency controller.

[0075] Example 2

[0076] In this embodiment, see Figure 4 , Figure 4 A schematic diagram of the structure of a data compression management system 200 of the present invention is shown. The data compression management system 200 includes:

[0077] The compression task coordination module 210 is used to scan the initial Hudi data table and generate a compression plan for each initial Hudi data table;

[0078] The compression task coordination module 210 is also used to detect candidate Hudi data tables with compression plans, and filter to obtain target Hudi data tables with valid compression plans.

[0079] Exemplarily, the compression task coordination module 210 includes a data scanner, a concurrent controller, and a compression plan filter, wherein the data scanner is used to scan the Hudi data table under the database storage directory and update the data table in time, thereby realizing operations such as adding and deleting the Hudi data table without manual intervention. The data scanner is also used to determine whether the structure and data of the Hudi data table have changed by reading and parsing the metadata file of the Hudi data table, and automatically update the Hudi data table. The data scanner is also used to obtain information of all Hudi data tables by querying the data warehouse tool (Hive) metadata database, and determine whether the structure and data of the Hudi data table have changed, and automatically update the Hudi data table. The concurrent controller is used to effectively control the concurrency of the Hudi table compression plan detection operation and avoid excessive load on server resources, thereby improving the concurrent processing capability and stability of the data compression management system 200. The compression plan filter is used to filter out the Hudi data table with a valid compression plan according to the filtering rules, delete the Hudi data table with an invalid compression plan, and avoid the submission of compression tasks for invalid Hudi data tables.

[0080] The task management module 220 is used to build a compression task based on the target Hudi data table, submit the compression task to the cache queue to detect whether all target Hudi data tables have duplicate names, and retain a target Hudi data table with a duplicate name to obtain a filtered target Hudi data table;

[0081] The task management module 220 is also used to submit the compression task corresponding to the filtered target Hudi data table to the task submission controller, so as to submit the corresponding compression task to the task compression container through the task submission controller, thereby running the compression task corresponding to the filtered target Hudi data table.

[0082] Exemplarily, the task management module 220 includes a task concurrency controller, a task submission controller, and a task listener. The task concurrency controller is used to concurrently manage the execution of multiple compression tasks to improve the efficiency and accuracy of task execution. The task concurrency controller can set a concurrency upper limit, that is, the maximum number of compression tasks that can be processed simultaneously at the same time. When the number of compression tasks reaches the upper limit, new tasks will be placed in the queue for processing. The task submission controller is used to monitor the operating environment of the task compression container and submit the compression task to the task compression container. The task listener is used to monitor the operating status of the compression task in real time according to the compression task status information recorded in the flink session window.

[0083] Exemplarily, the data compression management system 200 further includes a timing scheduling module 230 and an alarm module 240, wherein the timing scheduling module 230 is used to implement timing management and execution of compression tasks, and the timing scheduling module 230 is also used to perform corresponding compression processing on the compression tasks by setting the execution time, execution cycle, triggering rules, etc. of the compression tasks. The alarm module 240 is used to actively trigger an alarm notification when a compression task fails or an abnormal situation occurs, so that the system administrator can take timely measures to solve the problem.

[0084] The present application also relates to a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program runs on the processor, the data compression management method of the above-mentioned embodiment 1 is executed.

[0085] The present application also relates to a computer-readable storage medium, on which a computer program is stored. When the computer program is run on a processor, the data compression management method of the above-mentioned embodiment 1 is executed.

[0086] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow chart, and the combination of boxes in the structure diagram and / or the flow chart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0087] In addition, the functional modules or units in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0088] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0089] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A data compression management method, characterized in that: include: Scan the initial Hudi data tables and generate a compression plan for each of the initial Hudi data tables; Detect candidate Hudi data tables with compression plans and filter them to obtain target Hudi data tables with valid compression plans; Building a compression task based on the target Hudi data table, submitting the compression task to the cache queue to detect whether all the target Hudi data tables have duplicate names, and retaining a target Hudi data table with the duplicate name to obtain a filtered target Hudi data table; Submit the compression task corresponding to the filtered target Hudi data table to the task submission controller, so as to submit the corresponding compression task to the task compression container through the task submission controller, thereby running the compression task corresponding to the filtered target Hudi data table.

2. The data compression management method according to claim 1, characterized in that: The step of submitting the corresponding compression task to the task compression container through the task submission controller also includes: Concurrently manage the plurality of compression tasks through a task concurrency controller, and detect whether the running environment of the task compression container of the compression task to be run is ready through the task submission controller; If the operating environment of the task compression container is abnormal, a new task compression container is created to run the compression task to be run.

3. The data compression management method according to claim 2, characterized in that: The step of submitting the corresponding compression task to the task compression container through the task submission controller further includes: Configuring a task thread pool in the task compression container, and controlling the number of parallel operations of the compression task according to the number of core threads of the task thread pool; Monitor the running status of the compression task through a task listener, and visualize the running status of the compression task; When it is detected that the compression task fails to run, an alarm is triggered, and the compression task that fails to run is scheduled to the task concurrency controller.

4. The data compression management method according to claim 1, characterized in that: The detection of candidate Hudi data tables with compression plans includes: Cache the candidate Hudi data tables with the compression plan, and determine the submission batch of the candidate Hudi data tables; The compression plan detection is performed on the candidate Hudi data tables of a single batch submitted asynchronously triggered by the programming model, and the compression plan detection is performed on the candidate Hudi data tables of multiple batches submitted in parallel.

5. The data compression management method according to claim 1 or 4, characterized in that: The filtering obtains a target Hudi data table with a valid compression plan, including: Filter the candidate Hudi data tables with valid compression plans according to the filtering rules, and delete the candidate Hudi data tables with invalid compression plans; The filtering rules include: The identifier of the compression plan is not empty, and the timestamp of the compression plan is not within the timestamp range of the rollback request.

6. The data compression management method according to claim 1, characterized in that: The queue thread pool corresponding to the cache queue customizes the duplicate name detection logic and filtering logic of the target Hudi data table, detects whether all the target Hudi data tables have duplicate names, and retains a target Hudi data table with the duplicate name, including: The duplicate name detection logic and the filtering logic are called to detect and filter the target Hudi data table with the duplicate name, and only one target Hudi data table with the duplicate name is retained.

7. The data compression management method according to claim 6, characterized in that: The compression task corresponding to the filtered target Hudi data table is submitted to the task submission controller, which then includes: The compression task cache operation of cleaning the queue thread pool is triggered to delete the compression tasks that have been submitted, while the compression tasks that have not been submitted are waiting to be submitted.

8. A data compression management system, characterized in that: include: A compression task coordination module, used to scan the initial Hudi data table and generate a compression plan for each of the initial Hudi data tables; The compression task coordination module is also used to detect candidate Hudi data tables with compression plans and filter target Hudi data tables with valid compression plans; A task management module is used to build a compression task based on the target Hudi data table, submit the compression task to the cache queue, detect whether all the target Hudi data tables have duplicate names, and retain a target Hudi data table with the duplicate name to obtain a filtered target Hudi data table; The task management module is also used to submit the compression task corresponding to the filtered target Hudi data table to the task submission controller, so as to submit the corresponding compression task to the task compression container through the task submission controller, thereby running the compression task corresponding to the filtered target Hudi data table.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is run on the processor, the data compression management method according to any one of claims 1 to 7 is executed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is run on a processor, the data compression management method according to any one of claims 1 to 7 is executed.

Citation Information

Patent Citations

  • Data integrity monitoring method and device and computer readable storage medium

    CN109684877A

  • Query task execution method and device, computer equipment and storage medium

    CN116881299A