Data optimization method, device, storage medium, equipment and program product

Abstract

This application discloses a data optimization method, apparatus, storage medium, device, and program product, applied to data optimization scenarios. The method includes: when a metadata change event of a target data table in a data lake is detected, acquiring snapshot information of the metadata change event; receiving data optimization configuration information, which is generated by a terminal responding to configuration operations input on a data management interface for a configuration target, where the configuration target is at least one of a data directory, database, and data table; generating a data optimization task corresponding to the metadata change event based on the snapshot information and the data optimization configuration information; and executing the data optimization task. This application generates and executes data optimization tasks by monitoring metadata change events of data tables, without manual intervention, enabling real-time response to data changes, reducing the cost of data lake maintenance, and improving the timeliness and accuracy of data governance.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data optimization method, apparatus, storage medium, device, and program product. Background Technology

[0002] With the continuous development of big data technology, data lakes, as a key infrastructure for processing massive amounts of data, have become a focus of industry attention in terms of performance and efficiency. Data lake storage formats support multiple big data processing engines processing the same data table simultaneously and provide the reliability and simplicity of Structured Query Language (SQL) tables, leading to their widespread application in the big data field.

[0003] However, in the actual use of data tables in data lake storage formats, especially in high-frequency data write scenarios, data governance becomes a challenging task. Taking the MergeOnRead (MOR) mode of some data lake storage formats as an example, updated data files are written to both data files and deletion files. Subsequent queries require merging these two types of files to obtain the final data. While this mechanism improves write performance, it also increases the merging overhead during queries. Furthermore, each write operation in a data lake storage table generates a new snapshot, which, over time, leads to a significant increase in storage space, further impacting query performance. Traditional data governance solutions, such as manual or time-triggered methods, are insufficient to meet the data optimization needs of high-frequency write scenarios, easily leading to resource waste or data governance backlog. Summary of the Invention

[0004] This application provides a data optimization method, apparatus, storage medium, device, and program product, which generates and executes data optimization tasks by monitoring metadata change events of data tables, thereby improving the timeliness and accuracy of data governance.

[0005] On one hand, embodiments of this application provide a data optimization method, the method comprising: when a metadata change event of a target data table in a data lake is detected, acquiring snapshot information of the metadata change event; receiving data optimization configuration information, the data optimization configuration information being generated by a terminal in response to a configuration operation input on a data management interface for a configuration target, the configuration target being at least one of a data directory, a database, and a data table; generating a data optimization task corresponding to the metadata change event based on the snapshot information and the data optimization configuration information; and executing the data optimization task.

[0006] On the other hand, embodiments of this application provide a data optimization method, the method comprising: displaying a data management interface; generating data optimization configuration information in response to a configuration operation input on the data management interface for a configuration target, wherein the configuration target is at least one of a data directory, a database, and a data table; sending the data optimization configuration information to a server, so that the server generates a data optimization task corresponding to the metadata change event based on snapshot information of a metadata change event and the data optimization configuration information, and executes the data optimization task, wherein the snapshot information of the metadata change event is obtained by the server when it detects a metadata change event of a target data table in a data lake; receiving task execution data corresponding to the data optimization task sent by the server, and visually displaying the data optimization task on a graphical user interface based on the task execution data.

[0007] On the other hand, embodiments of this application provide a data optimization apparatus, the apparatus comprising:

[0008] The acquisition unit is used to acquire snapshot information of the metadata change event when a metadata change event of a target data table in the data lake is detected.

[0009] A receiving unit is used to receive data optimization configuration information, which is generated by the terminal in response to a configuration operation input on the data management interface for a configuration target. The configuration target is at least one of a data directory, a database, and a data table.

[0010] The first generation unit is used to generate a data optimization task corresponding to the metadata change event based on the snapshot information and the data optimization configuration information.

[0011] An execution unit is used to perform the data optimization task.

[0012] In some embodiments, the data optimization configuration information includes preset task generation conditions;

[0013] The first generation unit is used to generate a data optimization task corresponding to the metadata change event when the preset task generation conditions are met, based on historical data optimization task data and the snapshot information.

[0014] In some embodiments, when the first generation unit generates a data optimization task corresponding to the metadata change event based on historical data optimization task data and the snapshot information, and determines that the preset task generation conditions are met, it can be configured to: obtain the previous data optimization task corresponding to the target data table and the historical snapshot information corresponding to the previous data optimization task based on the historical data optimization task data; obtain a first number of data files generated within the time interval between the end time of the previous data optimization task and the trigger time of the metadata change event based on the historical snapshot information and the snapshot information; obtain a second number of incremental files generated within the time interval between the end time of the previous data optimization task and the trigger time of the metadata change event based on the historical snapshot information and the snapshot information; obtain the number of snapshots corresponding to the target data table; and generate a data optimization task corresponding to the metadata change event when it is determined that the data optimization task generation conditions are met based on at least one of the previous data optimization task, the first number, the second number, the number of snapshots, and the snapshot information.

[0015] In some embodiments, the acquisition unit may be used to: when the unified metadata system on the server receives snapshot information of a metadata change event of a target data table in the data lake, push the snapshot information to the event queue of the event messaging platform on the server through the unified metadata system; when the data optimization service on the server detects that there is snapshot information of the metadata change event in the event queue of the event messaging platform, acquire the snapshot information from the event queue.

[0016] In some embodiments, the execution unit may be configured to: after generating a data optimization task corresponding to the metadata change event through a data optimization service on a server, mark the task status of the data optimization task as initialization state; after successfully obtaining the data optimization task from the data optimization service through a data optimizer on the server, the data optimization task waits to be executed within the data optimizer, and the task status is marked as scheduling state; when the data optimizer confirms the task to the data optimization task and starts execution, the task status is marked as executing state; when the data optimization task finishes execution, the data optimizer reports the task execution result to the data optimization service, and if the task execution result is successful, the task status is marked as analysis state; after clearing the task record of the data optimization task or determining the next execution action of the target data table where the data optimization task is located according to the task type and task execution result of the data optimization task, the task status is marked as execution successful state.

[0017] In some embodiments, the execution unit can also be used to: for the data optimization task in the initialization state, if the data optimizer fails to acquire and execute the data optimization task when the first waiting time exceeds a first preset time threshold, the task status is marked as expired.

[0018] In some embodiments, the execution unit can also be used to: for the data optimization task that is in a scheduling state or an execution state, if the data optimization service does not receive the task execution result reported by the data optimizer when the second waiting time exceeds the second preset time threshold, the task status is marked as an execution timeout state.

[0019] In some embodiments, the acquisition unit may also be used to: obtain database information related to the metadata change event by parsing the data snapshot information, and store the snapshot information into the target database indicated by the database information.

[0020] In some embodiments, the first generation unit may also be used to: after generating the data optimization task corresponding to the metadata change event, store the data optimization task in the target database.

[0021] In some embodiments, the execution unit may also be used to: send task execution data corresponding to the data optimization task to the terminal, so that the terminal can visualize the data optimization task based on the task execution data.

[0022] On the other hand, embodiments of this application provide a data optimization apparatus, the apparatus comprising:

[0023] The display unit is used to display the data management interface;

[0024] The second generation unit is used to generate data optimization configuration information in response to the configuration operation input for the configuration target in the data management interface, wherein the configuration target is at least one of a data directory, a database, and a data table;

[0025] The sending unit is used to send the data optimization configuration information to the server, so that the server generates a data optimization task corresponding to the metadata change event based on the snapshot information of the metadata change event and the data optimization configuration information, and executes the data optimization task. The snapshot information of the metadata change event is obtained by the server when it detects the metadata change event of the target data table in the data lake.

[0026] The processing unit is used to receive the task execution data corresponding to the data optimization task sent by the server, and to visualize the data optimization task on a graphical user interface based on the task execution data.

[0027] On the other hand, an embodiment of this application provides a computer-readable storage medium storing a computer program adapted for loading by a processor to execute the data optimization method as described in any of the above embodiments.

[0028] On the other hand, an embodiment of this application provides a computer device, which includes a processor and a memory. The memory stores a computer program, and the processor executes the data optimization method as described in any of the above embodiments by calling the computer program stored in the memory.

[0029] On the other hand, an embodiment of this application provides a computer program product, including computer instructions that, when executed by a processor, implement the data optimization method as described in any of the above embodiments.

[0030] In this embodiment, when a metadata change event of a target data table in the data lake is detected, a snapshot of the metadata change event is obtained; data optimization configuration information is received, which is generated by the terminal in response to configuration operations input on the data management interface for a configuration target, where the configuration target is at least one of a data directory, database, and data table; based on the snapshot information and the data optimization configuration information, a data optimization task corresponding to the metadata change event is generated; and the data optimization task is executed. This embodiment, by monitoring metadata change events of data tables, generates and executes data optimization tasks without manual intervention, enabling immediate response to data changes. This avoids the delayed processing caused by data changes in traditional methods, reduces the cost of data lake maintenance, and improves the timeliness and accuracy of data governance. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the architecture of the data optimization system provided in the embodiments of this application.

[0032] Figure 2 This is a first flowchart illustrating the data optimization method provided in an embodiment of this application.

[0033] Figure 3 This is a schematic diagram of the task status of the data optimization task provided in the embodiments of this application.

[0034] Figure 4 This is a schematic diagram of the second process of the data optimization method provided in the embodiments of this application.

[0035] Figure 5 This is a schematic diagram of the data management interface provided in an embodiment of this application.

[0036] Figure 6 This is a first structural schematic diagram of the data optimization device provided in an embodiment of this application.

[0037] Figure 7 This is a second structural schematic diagram of the data optimization device provided in an embodiment of this application.

[0038] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] This application provides a data optimization method, apparatus, storage medium, device, and program product. Exemplarily, the data optimization method of this application can be executed by a computer device, which can be a terminal or server, etc. The terminal can be a smartphone, tablet, laptop, desktop computer, smart TV, smart speaker, wearable smart device, personal computer (PC), smart vehicle terminal, etc. The terminal may also include a client, which can be a data management client, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0041] The embodiments of this application can be applied to scenarios such as data governance and data optimization.

[0042] First, the following explanations are given for some of the nouns or terms that appear in the embodiments of this application:

[0043] Iceberg is a high-performance, open data table format suitable for large analytical tables. As an open-source data lake storage format, Iceberg tables bring the reliability and simplicity of Structured Query Language (SQL) tables to big data, enabling multiple engines (Spark / Flink / Trino / Presto, etc.) to safely process the same data table simultaneously.

[0044] Merge On Read (MOR): A data organization method in the big data field that combines original data files with incremental modification files. This method is often used in data update scenarios. The updated data is written into a new incremental file, thus providing extremely high data write performance. However, when querying, the original file and the incremental file need to be merged to obtain the final data.

[0045] Upsert: A write method provided by Iceberg based on primary key updates, using MOR as the data organization mode.

[0046] Data File: Iceberg stores the complete data in a data file.

[0047] Incremental Files (Delete Files): Iceberg stores incremental files for changed data, which can be further divided into position delete files and equality delete files. The position delete file stores the complete storage path of the data file containing the deleted data, as well as the location of the deleted data within that data file. The equality delete file stores the primary key of the deleted data.

[0048] Data Lake Compute (DLC) is an open cloud-native lake warehouse analytics service with advantages such as storage-compute separation, Iceberg lake format storage, and lake format governance.

[0049] Data Optimization Service: The backend service of DLC for optimizing Iceberg format data. It is mainly responsible for managing Iceberg write events, data optimization tasks and data optimizers, and provides users with a user-friendly management interface.

[0050] Data optimization tasks: The DLC performs data optimization actions for the Iceberg Lake format, including data merging tasks, snapshot expiration tasks, orphan file removal tasks, data lifecycle tasks, and smart indexing tasks.

[0051] Data Optimizer: The component of DLC responsible for executing data optimization tasks. It is responsible for obtaining data optimization tasks from the data optimization service, executing the tasks, and then reporting the execution results to the data optimization task.

[0052] Metadata change events: Events in which DLC-related metadata changes, including the creation, modification, and deletion of data catalogs, the creation, modification, and deletion of databases, and the creation, modification, and deletion of data tables.

[0053] Taking Iceberg as an example, traditional data governance solutions using the Iceberg data lake format include manual triggering and time-based triggering. For instance, manual triggering involves manually generating and executing Iceberg stored procedure tasks when a merge is needed. However, manual triggering cannot handle high-frequency Upsert writes, which typically occur on a minute-by-minute basis. This means data optimization tasks need to be generated every minute, and if these tasks are not generated promptly, data governance backlog will occur, requiring more resources to complete the governance process, or even failing to complete it and necessitating data re-import. Time-based triggering involves configuring and scheduling tasks to periodically generate and execute Iceberg stored procedure tasks. However, with time-based triggering, when a large number of tables require data governance, the inconsistent write frequencies of each table make it impossible to accurately estimate the governance trigger time. Estimating the trigger time based on high write frequencies leads to resource waste for tables with low write frequencies, while estimating based on low write frequencies results in data backlog for tables with high write frequencies.

[0054] Traditional data governance solutions, such as manual or time-triggered methods, are difficult to meet the data optimization needs in high-frequency write scenarios, and can easily lead to resource waste or data governance backlog.

[0055] Therefore, this application provides a data optimization method that generates and executes data optimization tasks by monitoring metadata change events of data tables, thereby improving the timeliness and accuracy of data governance.

[0056] The solutions provided in this application involve technologies such as data optimization, data governance, and data lake management, which are specifically illustrated in the following embodiments. These embodiments are described in detail below. It should be noted that the order of description of the following embodiments is not intended to limit the priority of the embodiments.

[0057] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of the data optimization system provided in the embodiments of this application.

[0058] The data optimization system includes a server 10 and a terminal 20, etc.; the server 10 and the terminal 20 are connected via a network, such as a wired or wireless network.

[0059] The terminal 20 can be used to display a graphical user interface (GUI). This terminal interacts with the user through the GUI, for example, by downloading and installing a client application, running a corresponding app, or accessing a website. In this embodiment, the terminal 20 can be a terminal device with a client application installed that can manage data.

[0060] Server 10 provides background services to clients of applications (such as applications capable of data management) in terminal 20. For example, server 10 can be a background server for the aforementioned applications (such as applications capable of data management). Server 10 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center.

[0061] During data management, a client application for data management is displayed on terminal 20. This client displays the data management interface and, in response to configuration operations input on the data management interface targeting a configuration goal, generates data optimization configuration information. The configuration goal can be at least one of a data directory, database, or data table. The data optimization configuration information is then sent to server 10, which, based on the snapshot information of the metadata change event and the data optimization configuration information, generates and executes the data optimization task corresponding to the metadata change event. The snapshot information of the metadata change event is obtained by server 10 when it detects a metadata change event in the target data table of the data lake. Finally, terminal 20 receives the task execution data corresponding to the data optimization task sent by server 10 and visualizes the data optimization task on a graphical user interface based on the task execution data.

[0062] Server 10 may be configured with a unified metadata system 11, an event messaging platform 12, a data optimization service 13, and a data optimizer 14.

[0063] The unified metadata system 11 can be used to manage the metadata information of Iceberg's data tables. When data is successfully written, the latest snapshot information of this write will be submitted to the unified metadata system 11.

[0064] Among them, the event message platform 12 can be used to temporarily store metadata change events of Iceberg's data tables, including database information addition, deletion and modification, data table information addition, modification and addition, and snapshot information of data write and commit.

[0065] Among them, the data optimization service 13 can be used to process metadata change events and generate corresponding data optimization tasks based on the metadata change events; it can also be used to manage data optimization tasks and data optimizers.

[0066] The data optimizer 14 can be used to execute data optimization tasks, obtain data optimization tasks from the data optimization service, and report the results of executing the data optimization tasks to the data optimization task.

[0067] The terminal 20 may be configured with a management and control system 21, which is used to provide users with a data optimization control entry point and to provide a visual display of information such as data optimization tasks and data optimizers.

[0068] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.

[0069] Please see Figures 2 to 3 , Figure 2 This is a first flowchart illustrating the data optimization method provided in an embodiment of this application. Figure 3 This is a schematic diagram of the task status for a data optimization task provided in an embodiment of this application. The method may include the following steps:

[0070] Step 110: When a metadata change event of the target data table in the data lake is detected, obtain the snapshot information of the metadata change event.

[0071] For example, target tables in a data lake can be monitored in real time to promptly detect any metadata change events. This can be achieved through polling, event triggering, or log analysis. Once a metadata change event is detected, the server will obtain a snapshot of that event. The snapshot information can include key information such as the state before and after the metadata change, timestamps, and the type of change (e.g., addition, update, or deletion).

[0072] In some embodiments, when a metadata change event of a target data table in the data lake is detected, snapshot information of the metadata change event is obtained, including: when the unified metadata system on the server receives snapshot information of the metadata change event of the target data table in the data lake, the snapshot information is pushed to the event queue of the event messaging platform on the server through the unified metadata system; when the data optimization service on the server detects that there is snapshot information of the metadata change event in the event queue of the event messaging platform, the snapshot information is obtained from the event queue.

[0073] The unified metadata system 11, acting as the metadata management center, is responsible for receiving and recording metadata change information of the data lake and generating snapshots. For example, the unified metadata system 11 on server 10 is responsible for real-time monitoring of metadata changes in target data tables within the data lake, including but not limited to creation, modification, and deletion events of data directories, databases, and data tables. When a metadata change event is detected, the unified metadata system 11 immediately generates snapshot information for that event. The snapshot information records detailed information such as the metadata status before and after the change, the change timestamp, and the change type (e.g., addition, modification, or deletion).

[0074] Then, the unified metadata system 11 pushes the snapshot information of the generated metadata change events to the event messaging platform 12 on the server 10. The event messaging platform 12 acts as an information relay station, temporarily storing the snapshot information to ensure that the data optimization service 13 can obtain the latest metadata change information in a timely manner. The event messaging platform 12 will temporarily store this snapshot information in an event queue so that the data optimization service 13 can efficiently process and respond to these metadata change events.

[0075] The data optimization service 13 is responsible for handling metadata change events and serves as a bridge connecting metadata change detection and data optimization task generation. For example, the data optimization service 13 on server 10 periodically or in real-time monitors the event queue of the event message platform 12. Once it detects snapshot information of metadata change events in the event queue, the data optimization service 13 immediately retrieves this snapshot information from the event queue.

[0076] In some embodiments, the method further includes: obtaining database information related to the metadata change event by parsing the data snapshot information, and storing the snapshot information in the target database indicated by the database information.

[0077] For example, when a new message (i.e., snapshot information of a metadata change event) is written to the event queue of the event messaging platform 12, a message callback mechanism is triggered. During the message callback process, the server retrieves the latest queue message from the event queue. Subsequently, it calls the event management module 1301 in the data optimization service 13 to parse the queue message. By parsing the queue message, the event management module 1301 can extract the specific database and table information involved in this metadata change event. Based on the extracted database and table information, it further reads the corresponding data storage information to obtain the complete snapshot information of this metadata change event. The snapshot information may include detailed information such as changes in the structure of the data table and the addition or deletion of data content. Then, the obtained snapshot information is stored in the target database indicated by the database information. This process ensures that the complete record of the metadata change event is properly preserved, providing a reliable data foundation for subsequent data optimization and analysis.

[0078] Step 120: Receive data optimization configuration information. The data optimization configuration information is generated by the terminal in response to the configuration operation input on the data management interface for the configuration target. The configuration target is at least one of the following: data directory, database, and data table.

[0079] For example, data optimization configuration information is generated by the terminal in response to configuration operations performed by the user on the data management interface of terminal 20 regarding configuration targets. The configuration target can be at least one of a data directory, database, or data table, depending on the user's optimization needs.

[0080] like Figure 5 As shown, the data management interface 50 provides users with an intuitive and easy-to-use configuration environment. The interface includes configuration components 51 corresponding to multiple configuration attribute fields. These components allow users to input or select specific configuration parameters and conditions. Users can configure data optimization parameters by performing corresponding operations on the configuration components 51 (such as numerical input, option selection, etc.). These operations are reflected in real time on the data management interface and generate corresponding data optimization configuration information.

[0081] For example, users can select the object to optimize based on their actual needs; this is called the configuration target. The configuration target can be a single data directory, database, or table, or a combination of them.

[0082] In addition to configuring the goals, users can also set corresponding optimization parameters and conditions. These parameters and conditions will directly affect the specific execution method and effect of the data optimization task.

[0083] For example, the data management interface can also display management configuration prompts. These prompts are designed to help users better understand configuration options, avoid misconfigurations, and provide suggested configuration solutions. Users can use these prompts to determine the final data optimization configuration, thereby ensuring the accuracy and effectiveness of the configuration.

[0084] Server 10 receives data optimization configuration information from terminal 20 and stores it in a suitable storage medium for subsequent access and use. After receiving the data optimization configuration information, server 10 performs necessary processing and analysis to ensure that it meets the requirements of the data optimization task.

[0085] Step 130: Based on the snapshot information and data optimization configuration information, generate the data optimization task corresponding to the metadata change event.

[0086] For example, based on snapshot information and data optimization configuration information, combined with the execution status of the previous data optimization task, the data changes of the data file and the incremental file (deleted file) within the time period from the last data optimization task to the current event trigger are calculated. Then, based on the calculation results, it is determined whether a data optimization task needs to be generated. Depending on the strategy of the snapshot data inspection items, different types of data optimization tasks will be generated. For example, possible task types include, but are not limited to: data merging tasks (to reduce the number of small files and improve query performance), snapshot expiration tasks (to clean up expired snapshot data and free up storage space), orphan file removal tasks (to delete files that are no longer referenced to avoid resource waste), data lifecycle tasks (to automatically archive or delete data according to the data lifecycle strategy), and intelligent indexing tasks (to dynamically adjust the index structure according to data access patterns and optimize query performance), etc.

[0087] In some embodiments, the data optimization configuration information includes preset task generation conditions; generating a data optimization task corresponding to a metadata change event based on snapshot information and data optimization configuration information includes: generating a data optimization task corresponding to a metadata change event when it is determined that the preset task generation conditions are met based on historical data optimization task data and snapshot information.

[0088] For example, preset task generation conditions enable the server to automatically and intelligently determine whether a new optimization task needs to be executed based on the current data status and historical optimization records. This reduces the need for manual intervention and improves the efficiency and accuracy of data management. By reasonably setting preset task generation conditions, tasks can be generated promptly when the data status truly requires optimization, avoiding unnecessary waste of computing resources. Simultaneously, it ensures that more important optimization tasks are prioritized when resources are scarce.

[0089] For example, first collect snapshot information and historical data optimization task data. Snapshot information records details of metadata change events, while historical data optimization task data provides the execution status and effects of historical data optimization tasks. Then, this information is comprehensively analyzed according to preset task generation conditions to determine whether the current data state meets the requirements for generating a new task.

[0090] If the task generation conditions are met, data optimization tasks corresponding to metadata change events will be automatically generated. These tasks may include various types such as data merging, snapshot expiration cleanup, orphan file removal, and data lifecycle management.

[0091] In some embodiments, after optimizing task data and snapshot information based on historical data and determining that preset task generation conditions are met, a data optimization task corresponding to a metadata change event is generated, including:

[0092] Based on historical data optimization task data, obtain the previous data optimization task corresponding to the target data table, as well as the historical snapshot information corresponding to the previous data optimization task;

[0093] Based on historical snapshot information and snapshot information, obtain the first number of data files generated within the time interval between the end time of the last data optimization task and the trigger time of the metadata change event;

[0094] Based on historical snapshot information and snapshot information, obtain the second number of incremental files generated within the time interval between the end time of the last data optimization task and the trigger time of the metadata change event;

[0095] Get the number of snapshots corresponding to the target data table;

[0096] If the conditions for generating a data optimization task are met based on at least one of the previous data optimization task, the first quantity, the second quantity, the number of snapshots, and the snapshot information, a data optimization task corresponding to the metadata change event is generated.

[0097] During the processing, the first step is to locate the most recently executed data optimization task for the target data table by optimizing historical data. This typically involves a task log or database record that stores detailed information about each data optimization task, including the task execution time, the data tables involved, the task type (such as full optimization, incremental optimization, etc.), and the snapshot information corresponding to the task execution (i.e., a snapshot of the data state at the start or end of the task).

[0098] Then, by comparing the snapshot information at the end of the previous data optimization task with the snapshot information at the time the current metadata change event is triggered, the number of data files added or modified in the target data table between the end of the previous task and the trigger of the current metadata change event is calculated (first count). At the same time, the number of incremental files generated during this time interval is also counted (second count).

[0099] The number of snapshots of a target data table reflects how frequently its data state is recorded and saved. A higher number of snapshots indicates stronger data state monitoring and recovery capabilities, but may also increase storage and management costs. When deciding whether to generate a new data optimization task, the number of snapshots can serve as a factor in assessing the stability of the current data state and the necessity of optimization.

[0100] After considering the above information, the server will make a decision based on the preset data optimization conditions for task generation.

[0101] In some embodiments, the data optimization task includes a data merging task, and the preset task generation conditions include data merging task generation conditions. When the preset task generation conditions are met based on historical data optimization task data and snapshot information, a data optimization task corresponding to the metadata change event is generated, including: obtaining the previous data merging task corresponding to the target data table and the historical snapshot information corresponding to the previous data merging task based on the historical data optimization task data; obtaining a first number of data files generated within the time interval between the end time of the previous data merging task and the trigger time of the metadata change event based on the historical snapshot information and the snapshot information; obtaining a second number of incremental files generated within the time interval between the end time of the previous data merging task and the trigger time of the metadata change event based on the historical snapshot information and the snapshot information; and generating a data merging task corresponding to the metadata change event when the data merging task generation conditions are met based on the previous data merging task, the first number, the second number, and the snapshot information.

[0102] The conditions for generating the data merging task include any of the following:

[0103] The last data merge task was executed successfully;

[0104] The interval between the end time of the last data merging task and the start time of the current data merging task to be generated is greater than the task running cycle, and the first quantity is greater than the merging quantity threshold. The start time of the current data merging task to be generated is determined based on snapshot information.

[0105] The interval between the end time of the previous data merging task and the start time of the current data merging task to be generated is greater than the task running cycle, and the second quantity is greater than the merging quantity threshold.

[0106] For example, data merging tasks aim to reduce the number of small files and improve query performance. In big data processing scenarios, an excessive number of small files can significantly impact query efficiency, making it essential to perform data merging tasks regularly. For instance, the triggering of a data merging task can be determined by calculating the following four formulas:

[0107]

[0108] Formula (1) is used to check the previous data merging task Result corresponding to the target data table. lasted_task Whether the data merge was successful (SUCCESS) is the basis for ensuring the continuity and effectiveness of the task. If the previous data merge task failed to execute successfully, the server may pause subsequent data merge tasks to avoid data inconsistency or excessive server load.

[0109] Formula (2) defines the end time T of the previous data merging task. lasted_task The interval between the start time (Current_time) of the merge task with the currently generated data is greater than the task execution period (max(task_execute_time)); this is to ensure that the server has enough time to process the newly generated data and to avoid resource waste caused by too frequent task execution.

[0110] Formula (3) defines the end time T of the previous data merging task. lasted_task The trigger time T of the metadata change event current_task The first number of data files generated within the time interval between them A value greater than the merge quantity threshold Val indicates that enough small files have accumulated on the server and need to be merged to improve query performance.

[0111] Formula (4) defines the end time T of the previous data merging task. lasted_task The trigger time T of the metadata change event current_task The second number of incremental files (Delete Files) generated within the time interval between them A value greater than the merge quantity threshold Val indicates that enough small files have accumulated on the server and need to be merged to improve query performance.

[0112] When formula (1) is met, or both formulas (2) and (3) are met, or both formulas (2) and (4) are met, the conditions for generating a data merging task are determined, and the data merging task is triggered. This mechanism ensures that the server can automatically execute the data merging task at the appropriate time and under appropriate conditions, thereby improving the efficiency and performance of data management.

[0113] In some embodiments, the data optimization task includes a snapshot expiration task, and the preset task generation conditions include snapshot expiration task generation conditions. When the preset task generation conditions are met based on historical data optimization task data and snapshot information, a data optimization task corresponding to the metadata change event is generated, including: obtaining the previous snapshot expiration task corresponding to the target data table based on historical data optimization task data; obtaining the number of snapshots corresponding to the target data table; and generating the snapshot expiration task corresponding to the metadata change event when the snapshot expiration task generation conditions are met based on the previous snapshot expiration task, the number of snapshots, and snapshot information.

[0114] The conditions for generating a snapshot expiration task include any of the following:

[0115] The previous snapshot expiration task was executed successfully;

[0116] The interval between the end time of the last snapshot expiration task and the start time of the current snapshot expiration task to be generated is greater than the task execution cycle. The start time of the current snapshot expiration task to be generated is determined based on the snapshot information.

[0117] The number of snapshots corresponding to the target data table has reached the snapshot data threshold.

[0118] For example, a snapshot expiration task is responsible for cleaning up expired snapshot data in a target data table. In data lakes or distributed storage systems, as data is continuously updated, old snapshot data can occupy a large amount of storage space, thus requiring periodic snapshot expiration tasks to free up this space. For example, the triggering of a snapshot expiration task can be determined by calculating the following three formulas:

[0119]

[0120] Formula (1) is used to check the previous snapshot expiration task Result corresponding to the target data table. lasted_task Whether the execution was successful (SUCCESS) is the basis for ensuring the continuity and effectiveness of the task. If the previous snapshot expiration task failed to execute successfully, the server may pause subsequent snapshot expiration tasks to avoid data loss or excessive system load.

[0121] Formula (2) defines the end time T of the last snapshot expiration task. lasted_taskThe interval between the current snapshot expiration task and the start time of the current snapshot task to be generated is greater than the task execution period max(task_execute_time); this is to ensure that the server has enough time to process the newly generated snapshot data and to avoid resource waste caused by too frequent task execution.

[0122] Formula (3) defines that when the number of snapshots corresponding to the target data table ∑(Sn) reaches the snapshot data threshold Val, it indicates that enough snapshot data has been accumulated in the server and needs to be cleaned up to release storage space.

[0123] When formula (1) is met, or both formulas (2) and (3) are met simultaneously, the conditions for generating a snapshot expiration task are determined, and the snapshot expiration task is triggered. This mechanism ensures that the server can automatically execute the snapshot expiration task at the appropriate time and under appropriate conditions, thereby freeing up storage space and improving system performance.

[0124] In some embodiments, the data optimization task includes an orphan file removal task, and the preset task generation conditions include orphan file removal generation conditions. When the preset task generation conditions are met based on historical data optimization task data and snapshot information, a data optimization task corresponding to a metadata change event is generated, including: obtaining the previous orphan file removal task corresponding to the target data table based on historical data optimization task data; and generating the orphan file removal task corresponding to the metadata change event when the preset task generation conditions are met based on the previous orphan file removal task and snapshot information.

[0125] The conditions for generating the task of removing orphaned files include any of the following:

[0126] The last task to remove orphaned files was completed successfully;

[0127] The interval between the end time of the last orphan removal task and the start time of the current orphan removal task to be generated is greater than the task execution cycle. The start time of the current orphan removal task to be generated is determined based on snapshot information.

[0128] For example, the Remove Orphaned Files task is responsible for cleaning up orphaned files that are no longer referenced in the target data table. In data lakes or distributed storage systems, as data is continuously updated and deleted, some orphaned files may appear that are no longer referenced by any data table or index. These orphaned files occupy storage space and are no longer valuable, so it is necessary to periodically execute the Remove Orphaned Files task to clean them up. For example, the triggering of the Remove Orphaned Files task can be determined by calculating the following two formulas:

[0129]

[0130] Formula (1) is used to check the previous orphan file removal task Result corresponding to the target data table. lasted_task Whether the task was successfully executed (SUCCESS) is the basis for ensuring the continuity and effectiveness of the task. If the previous task to remove orphaned files failed to execute successfully, the server may pause subsequent tasks to remove orphaned files to avoid accidentally deleting important files or causing excessive system load.

[0131] Formula (2) defines the end time T of the last orphan file removal task. lasted_task The interval between the start time (Current_time) of the currently pending orphan removal task is greater than the task execution cycle.

[0132] max(task_execute_time); This is to ensure that the server has enough time to detect new orphaned files and to avoid resource waste caused by overly frequent task execution;

[0133] When either formula (1) or formula (2) is met, the conditions for generating the orphan file removal task are determined, and the orphan file removal task is triggered. This mechanism ensures that the server can automatically execute the orphan file removal task at the appropriate time and under appropriate conditions, thereby cleaning up orphan files that are no longer referenced and avoiding resource waste and data corruption.

[0134] For example, data lifecycle tasks primarily target data that needs to be archived or deleted according to the data's lifecycle policy. For instance, the server periodically evaluates the data lifecycle policy to determine which data has met the criteria for archiving or deletion. For example, some data may be marked as expired based on creation date, last access time, or business rules. For example, by combining changes to data files and incremental files from the last data optimization task to the current event trigger, the server analyzes the actual usage of the data; if data has not been accessed or modified during this period, it may be considered suitable for archiving or deletion. For example, when data meets the lifecycle policy and the server determines that the data no longer requires frequent access, a data lifecycle task will be generated. This task may include moving the data to archive storage, deleting old data, or exporting the data to other systems.

[0135] For example, smart indexing tasks are generated to dynamically adjust the index structure based on data access patterns to optimize query performance. For instance, the server analyzes data access patterns, particularly query frequency and type, to determine which indexes are effective and which may need adjustment; if certain indexes have been inactive for a long time or query performance degrades, the server may consider rebuilding or deleting these indexes. For example, the current state of the indexes can also be assessed in conjunction with changes to data files and incremental files; if the number of data files increases significantly or the data distribution changes, the indexes may need to be updated to reflect these changes. When the server determines that an index needs optimization, it generates a smart indexing task, which may include creating new indexes, deleting unnecessary indexes, merging multiple indexes, or rebuilding existing indexes.

[0136] In some embodiments, the method further includes: after generating a data optimization task corresponding to a metadata change event, storing the data optimization task in the target database.

[0137] For example, storing data optimization tasks in the target database ensures persistent storage of the tasks. Even if the system restarts or fails, this task information will not be lost, thus guaranteeing the continuity and reliability of the data optimization process. Data optimization tasks stored in the target database can be identified and scheduled for execution by the data optimization service 13 in server 10. This helps to automate task execution and rationally allocate resources, improving data optimization efficiency. After generating a data optimization task, the server encapsulates relevant task information (such as task type, target data table, execution parameters, etc.) to form a task record. The server establishes a connection with the target database through predefined database connection information. Then, it uses appropriate database operation statements (such as INSERT statements) to store the task record in a specified table or collection in the target database. During task execution, the server also periodically updates the task's status information in the target database. This helps users and other system components understand the task's execution status in real time.

[0138] To ensure data security and privacy, the server adheres to strict security protocols and access control mechanisms when operating on the target database. Only authorized users or system components can access and modify task information.

[0139] Step 140: Perform the data optimization task.

[0140] In some embodiments, executing a data optimization task includes: after generating a data optimization task corresponding to a metadata change event through a data optimization service on the server, the task status of the data optimization task is marked as initialization; after successfully obtaining the data optimization task from the data optimization service through the data optimizer on the server, the data optimization task waits for execution within the data optimizer, and the task status is marked as scheduling; when the data optimizer confirms the task to the data optimization task and starts execution, the task status is marked as executing; when the data optimization task finishes execution, the data optimizer reports the task execution result to the data optimization service, and if the task execution result is successful, the task status is marked as analysis; after cleaning the task record of the data optimization task or determining the next execution action of the target data table where the data optimization task is located according to the task type and task execution result of the data optimization task, the task status is marked as successful execution.

[0141] In some embodiments, the method further includes: if the task execution result is a failure, then the task status is marked as an execution failure status.

[0142] In some embodiments, the method further includes: for a data optimization task in the initialization state, if the data optimizer fails to acquire and execute the data optimization task when the first waiting time exceeds a first preset time threshold, the task status is marked as expired.

[0143] In some embodiments, the method further includes: for a data optimization task that is in a scheduling state or an execution state, if the data optimization service does not receive the task execution result reported by the data optimizer when the second waiting time exceeds a second preset time threshold, the task status is marked as an execution timeout state.

[0144] For example, the server will execute the generated data optimization tasks. The execution of these tasks can be handled by a dedicated component, such as a data optimizer. The data optimizer will perform corresponding operations according to the task requirements, such as merging files, cleaning up expired snapshots, and removing orphaned files. Figure 3 The task status diagram shown below illustrates the specific implementation scheme for task execution as follows:

[0145] (1) Initialization state: When a metadata change event is triggered and a data optimization task is generated and stored in the database, the task is marked as initialization state in the data optimization service 13. This means that the task has been generated but has not yet been acquired and executed by the data optimizer 14.

[0146] (2) Scheduling Status: After the data optimizer 14 successfully obtains the data optimization task from the data optimization service 13, the task waits for execution within the data optimizer 14. At this time, the task status is updated to the scheduling status. This indicates that the task has been received by the data optimizer 14, but has not yet started execution.

[0147] (3) In-process status: When the data optimizer 14 confirms and begins execution of the data optimization task, the task status is marked as in-process. During execution, the data optimizer 14 will perform corresponding operations according to the task requirements. For example, when executing a data merging task, it checks the threshold for the number of data files (Delete File) and can execute it again. For example, when executing a snapshot expiration task, it removes expired snapshots and refreshes snapshot count statistics, etc.

[0148] (4) In-Analysis Status: When the data optimization task finishes execution, the data optimizer 14 reports the task execution result to the data optimization service 13. If the task execution result is successful, the task status is marked as In-Analysis. In this status, the server will clean up the task record or determine the next execution action based on the task type and execution result.

[0149] (5) Successful execution status: After completing the corresponding cleanup or determining the next execution action based on the task type and execution result, the task status is marked as successful execution. This indicates that the task has been successfully executed and all subsequent processing has been completed.

[0150] (6) Execution Failure Status: If the task execution result is failure, the task status is marked as execution failure. The server may need to take additional measures to handle failed tasks, such as retrying execution or notifying the administrator.

[0151] (7) Expired Status: For data optimization tasks in the initialization state, if the data optimizer fails to acquire and execute the task within the first preset time threshold after the first waiting time, the task status is marked as expired. This indicates that the task has expired due to prolonged inactivity, which may require administrator intervention or adjustments to system resources. For example, users can enable further elastic resource usage for the data optimizer. When there are too many initialization tasks, more resources can be allocated to optimize tasks, thereby reducing task expiration. The system supports elastic resource usage. When a large number of expired or timed-out tasks are detected, the server can automatically increase resources, such as starting more data optimizer instances, to improve processing capacity.

[0152] (8) Execution Timeout Status: For data optimization tasks in the scheduling or execution state, if the data optimization service 13 does not receive the task execution result reported by the data optimizer after the second waiting time exceeds the second preset time threshold, the task status is marked as execution timeout status. This indicates that a problem may have occurred during task execution, such as insufficient resources or excessive task complexity, requiring administrator intervention or system configuration adjustments. For example, the server may issue an alarm message to the user, indicating insufficient resources and suggesting that the user increase data governance resources. For example, the user can schedule the task to a shared governance resource hosted by the system with larger resources based on the alarm message.

[0153] In some embodiments, the method further includes: sending task execution data corresponding to the data optimization task to the terminal, so that the terminal can visualize the data optimization task based on the task execution data.

[0154] For example, during the execution of data optimization tasks, various relevant task execution data can be collected and recorded in real time, including but not limited to task status (such as initialization, scheduling, execution, completion, etc.), task progress (such as the amount of data processed, remaining processing time, etc.), task logs (such as error messages, warning messages, etc.), and task performance metrics (such as execution speed, resource utilization, etc.). Task execution data can be obtained through various methods such as log recording, database queries, and system monitoring. The server will automatically collect and integrate this data at different stages of task execution for subsequent visualization.

[0155] Task execution data is sent to designated terminals, which can be personal computers, tablets, smartphones, etc., depending on the actual use case and requirements. Data can be sent to the terminals via network transmission. The server selects an appropriate transmission protocol based on the terminal's network environment and access method to ensure data real-time performance and accuracy. Before transmission, task execution data is converted into a format suitable for terminal parsing and display, such as JSON or XML. This helps the terminal quickly parse the data and generate corresponding visual interfaces.

[0156] The terminal utilizes various visualization technologies to display task execution data, such as charts, graphs, and animations. These technologies intuitively display task status, progress, and performance metrics, enabling users to quickly understand the task's execution status. The terminal provides a user-friendly graphical interface, allowing users to easily view and manipulate task execution data. Users can browse data across different time periods and dimensions, switch task statuses, and query logs by clicking, dragging, and zooming. The terminal receives and updates task execution data in real time, ensuring users always see the latest task status and information. This helps users promptly identify problems and take appropriate measures for adjustment and optimization. By visually displaying task execution data, users can more intuitively understand the execution status and performance metrics of data optimization tasks, thereby improving process transparency. Users can monitor and manage task execution data through the terminal, promptly identify and resolve problems, and improve the efficiency and reliability of the data optimization process. Visualization also provides users with data support, helping them better understand the execution effects and existing problems of data optimization tasks, thus enabling more informed decision-making.

[0157] By sending the task execution data corresponding to the data optimization task to the terminal and displaying it visually, users can more easily understand the task execution status, identify problems in a timely manner, and make adjustments and optimizations, thereby improving the efficiency and quality of data optimization.

[0158] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0159] In this embodiment, when a metadata change event of a target data table in the data lake is detected, a snapshot of the metadata change event is obtained; data optimization configuration information is received, which is generated by the terminal in response to configuration operations input on the data management interface for a configuration target, where the configuration target is at least one of a data directory, database, and data table; based on the snapshot information and the data optimization configuration information, a data optimization task corresponding to the metadata change event is generated; and the data optimization task is executed. This embodiment, by monitoring metadata change events of data tables, generates and executes data optimization tasks without manual intervention, enabling immediate response to data changes. This avoids the delayed processing caused by data changes in traditional methods, reduces the cost of data lake maintenance, and improves the timeliness and accuracy of data governance.

[0160] Please see Figures 4 to 5 , Figure 4 This is a schematic diagram of the second process of the data optimization method provided in the embodiments of this application. Figure 5 This is a schematic diagram of a data management interface provided in an embodiment of this application. The method may include the following steps:

[0161] Step 210: Display the data management interface.

[0162] For example, on the terminal, after the system starts, it displays an intuitive and easy-to-use data management interface. This interface typically includes multiple functional areas, such as data catalog browsing, database management, and data table configuration, to facilitate various data management and configuration operations for users. The interface design prioritizes user experience, providing clear navigation and interaction guidance so that users can easily find and operate the configuration targets they need.

[0163] like Figure 5 As shown, the data management interface 50 provides users with an intuitive and easy-to-use configuration environment. The interface contains configuration components 51 corresponding to multiple configuration attribute fields, which allow users to input or select specific configuration parameters and conditions.

[0164] Step 220: In response to the configuration operation input on the data management interface for the configuration target, generate data optimization configuration information. The configuration target is at least one of the following: data directory, database, and data table.

[0165] For example, when a user performs configuration operations through the data management interface displayed on the terminal, the terminal captures the user's input in real time and generates corresponding data optimization configuration information based on the user's configuration requirements. Configuration targets can include at least one of a data directory, database, and data table. For instance, a user might need to set data compression, partitioning, or indexing strategies for a specific data table. The terminal will generate data optimization configuration information containing these configuration requirements based on the user's input.

[0166] like Figure 5 The data management interface 50 shown contains configuration components 51 corresponding to multiple configuration attribute fields. Users can configure data optimization parameters by performing corresponding operations on the configuration components 51 (such as entering values, selecting options, etc.). These operations are reflected in the data management interface in real time and generate corresponding data optimization configuration information.

[0167] For example, users can select the object to optimize based on their actual needs; this is called the configuration target. The configuration target can be a single data directory, database, or table, or a combination of them.

[0168] In addition to configuring the goals, users can also set corresponding optimization parameters and conditions. These parameters and conditions will directly affect the specific execution method and effect of the data optimization task.

[0169] For example, the data management interface can also display management configuration prompts. These prompts are designed to help users better understand configuration options, avoid misconfigurations, and provide suggested configuration solutions. Users can use these prompts to determine the final data optimization configuration, thereby ensuring the accuracy and effectiveness of the configuration.

[0170] Step 230: Send the data optimization configuration information to the server so that the server can generate a data optimization task corresponding to the metadata change event based on the snapshot information of the metadata change event and the data optimization configuration information, and execute the data optimization task. The snapshot information of the metadata change event is obtained by the server when it detects the metadata change event of the target data table in the data lake.

[0171] For example, once the data optimization configuration information is generated, the terminal immediately sends it to the server. Upon receiving this data optimization configuration information, the server performs a comprehensive analysis, combining it with snapshot information of previously detected metadata change events. Based on the analysis results, the server generates data optimization tasks corresponding to the metadata change events. After generating these data optimization tasks, the server immediately executes them to ensure that the data in the data lake remains in optimal condition.

[0172] For example, the server can calculate the data changes in data files (DataFile) and incremental files (Delete File) within the time period from the last data optimization task to the current event trigger, based on snapshot information, data optimization configuration information, and the execution status of the previous data optimization task. Then, it decides whether to generate a data optimization task based on the calculation results. Depending on the strategy for checking the snapshot data, different types of data optimization tasks will be generated. Possible task types include, but are not limited to: data merging tasks (to reduce the number of small files and improve query performance), snapshot expiration tasks (to clean up expired snapshot data and free up storage space), orphan file removal tasks (to delete files that are no longer referenced to avoid resource waste), data lifecycle tasks (to automatically archive or delete data according to the data lifecycle strategy), and intelligent indexing tasks (to dynamically adjust the index structure based on data access patterns to optimize query performance), etc.

[0173] Step 240: Receive the task execution data corresponding to the data optimization task sent by the server, and visualize the data optimization task on the graphical user interface based on the task execution data.

[0174] During the execution of data optimization tasks, the server continuously generates task execution data, including but not limited to task status (such as initialization, scheduling, execution, completion, etc.), task progress (such as the amount of data processed, remaining processing time, etc.), task logs (such as error messages, warning messages, etc.), and task performance metrics (such as execution speed, resource utilization, etc.). This data is sent back to the terminal in real time. After receiving this task execution data, the terminal visualizes it on a graphical user interface. The terminal uses various visualization techniques to display the task execution data, such as charts, graphs, and animations. These techniques can intuitively display the task status, progress, and performance metrics, allowing users to quickly understand the task's execution status. The terminal provides a user-friendly graphical user interface, enabling users to easily view and manipulate task execution data. Users can browse data from different time periods and dimensions, as well as switch task statuses and query logs, by clicking, dragging, zooming, etc. The terminal receives and updates task execution data in real time, ensuring that users always see the latest task status and information. This helps users promptly identify problems and take corresponding measures for adjustment and optimization.

[0175] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0176] This application embodiment displays a data management interface; responds to configuration operations input on the data management interface for a configuration target, generating data optimization configuration information. The configuration target can be at least one of a data directory, database, and data table. The data optimization configuration information is sent to a server, which generates a data optimization task corresponding to the metadata change event based on the snapshot information of the metadata change event and the data optimization configuration information, and executes the data optimization task. The snapshot information of the metadata change event is obtained by the server when it detects a metadata change event in a target data table in the data lake. The application receives task execution data corresponding to the data optimization task sent by the server and visualizes the data optimization task on a graphical user interface based on the task execution data. This application embodiment generates and executes data optimization tasks by monitoring metadata change events of data tables, without manual intervention. It can respond to data changes instantly, avoiding the delayed processing caused by data changes in traditional methods, reducing the cost of data lake maintenance, and improving the timeliness and accuracy of data governance. This application embodiment provides a data management interface on a graphical user interface, allowing users to generate data optimization configuration information by inputting configuration operations through the interface. Simultaneously, it receives task execution data corresponding to the data optimization task sent by the server and visualizes it on the interface. This intuitive presentation allows users to clearly understand the execution status and progress of data optimization tasks, facilitating data management.

[0177] To facilitate better implementation of the data optimization method of this application embodiment, this application embodiment also provides a data optimization apparatus. Please refer to... Figure 6 , Figure 6 This is a first structural schematic diagram of a data optimization device provided in an embodiment of this application. The data optimization device 300 may include:

[0178] The acquisition unit 310 is used to acquire snapshot information of the metadata change event when a metadata change event of the target data table in the data lake is detected;

[0179] The receiving unit 320 is used to receive data optimization configuration information. The data optimization configuration information is generated by the terminal in response to the configuration operation input on the data management interface for the configuration target. The configuration target is at least one of the data directory, database and data table.

[0180] The first generation unit 330 is used to generate data optimization tasks corresponding to metadata change events based on snapshot information and data optimization configuration information.

[0181] Execution unit 340 is used to perform data optimization tasks.

[0182] In some embodiments, the data optimization configuration information includes preset task generation conditions;

[0183] The first generation unit 330 is used to generate a data optimization task corresponding to a metadata change event when optimizing task data and snapshot information based on historical data and determining that the preset task generation conditions are met.

[0184] In some embodiments, when the first generation unit 330 generates a data optimization task corresponding to a metadata change event based on historical data optimization task data and snapshot information, and determines that the preset task generation conditions are met, it can be used to: obtain the previous data optimization task corresponding to the target data table and the historical snapshot information corresponding to the previous data optimization task based on historical data optimization task data; obtain a first number of data files generated within the time interval between the end time of the previous data optimization task and the trigger time of the metadata change event based on historical snapshot information and snapshot information; obtain a second number of incremental files generated within the time interval between the end time of the previous data optimization task and the trigger time of the metadata change event based on historical snapshot information and snapshot information; obtain the number of snapshots corresponding to the target data table; and generate a data optimization task corresponding to the metadata change event when it is determined that the data optimization task generation conditions are met based on at least one of the previous data optimization task, the first number, the second number, the number of snapshots, and the snapshot information.

[0185] In some embodiments, the data optimization task includes a data merging task, and the preset task generation conditions include data merging task generation conditions. When the first generation unit 330 generates a data optimization task corresponding to a metadata change event based on historical data optimization task data and snapshot information, and determines that the preset task generation conditions are met, it can be used to: obtain the previous data merging task corresponding to the target data table and the historical snapshot information corresponding to the previous data merging task based on historical data optimization task data; obtain a first number of data files generated within the time interval between the end time of the previous data merging task and the trigger time of the metadata change event based on historical snapshot information and snapshot information; obtain a second number of incremental files generated within the time interval between the end time of the previous data merging task and the trigger time of the metadata change event based on historical snapshot information and snapshot information; and generate a data merging task corresponding to the metadata change event based on the previous data merging task, the first number, the second number, and the snapshot information, and determine that the data merging task generation conditions are met.

[0186] The conditions for generating the data merging task include any of the following:

[0187] The last data merge task was executed successfully;

[0188] The interval between the end time of the last data merging task and the start time of the current data merging task to be generated is greater than the task running cycle, and the first quantity is greater than the merging quantity threshold. The start time of the current data merging task to be generated is determined based on snapshot information.

[0189] The interval between the end time of the previous data merging task and the start time of the current data merging task to be generated is greater than the task running cycle, and the second quantity is greater than the merging quantity threshold.

[0190] In some embodiments, the data optimization task includes a snapshot expiration task, and the preset task generation conditions include snapshot expiration task generation conditions. When the first generation unit 330 generates a data optimization task corresponding to a metadata change event based on historical data optimization task data and snapshot information, and determines that the preset task generation conditions are met, it can be used to: obtain the previous snapshot expiration task corresponding to the target data table based on historical data optimization task data; obtain the number of snapshots corresponding to the target data table; and generate a snapshot expiration task corresponding to the metadata change event based on the previous snapshot expiration task, the number of snapshots, and snapshot information, and determine that the snapshot expiration task generation conditions are met.

[0191] The conditions for generating a snapshot expiration task include any of the following:

[0192] The previous snapshot expiration task was executed successfully;

[0193] The interval between the end time of the last snapshot expiration task and the start time of the current snapshot expiration task to be generated is greater than the task execution cycle. The start time of the current snapshot expiration task to be generated is determined based on the snapshot information.

[0194] The number of snapshots corresponding to the target data table has reached the snapshot data threshold.

[0195] In some embodiments, the data optimization task includes a task to remove orphaned files, and the preset task generation conditions include conditions for removing orphaned files. When the first generation unit 330 generates a data optimization task corresponding to a metadata change event based on historical data optimization task data and snapshot information, and determines that the preset task generation conditions are met, it can be used to: obtain the previous task to remove orphaned files corresponding to the target data table based on historical data optimization task data; and generate a task to remove orphaned files corresponding to the metadata change event based on the previous task to remove orphaned files and snapshot information, and determine that the conditions for removing orphaned files are met.

[0196] The conditions for generating the task of removing orphaned files include any of the following:

[0197] The last task to remove orphaned files was completed successfully;

[0198] The interval between the end time of the last orphan removal task and the start time of the current orphan removal task to be generated is greater than the task execution cycle. The start time of the current orphan removal task to be generated is determined based on snapshot information.

[0199] In some embodiments, the acquisition unit 310 can be used to: when the unified metadata system on the server receives snapshot information of metadata change events of a target data table in the data lake, push the snapshot information to the event queue of the event message platform on the server through the unified metadata system; when the data optimization service on the server detects that there is snapshot information of metadata change events in the event queue of the event message platform, acquire the snapshot information from the event queue.

[0200] In some embodiments, the execution unit 340 can be used to: after generating a data optimization task corresponding to a metadata change event through the data optimization service on the server, mark the task status of the data optimization task as initialization state; after successfully obtaining the data optimization task from the data optimization service through the data optimizer on the server, the data optimization task waits for execution within the data optimizer, and the task status is marked as scheduling state; when the data optimizer confirms the task to the data optimization task and starts execution, the task status is marked as execution state; when the data optimization task finishes execution, the data optimizer reports the task execution result to the data optimization service, and if the task execution result is successful, the task status is marked as analysis state; after cleaning the task record of the data optimization task or determining the next execution action of the target data table where the data optimization task is located according to the task type and task execution result of the data optimization task, the task status is marked as execution successful state.

[0201] In some embodiments, the execution unit 340 can also be used to: mark the task status as an execution failure status if the task execution result is a failure.

[0202] In some embodiments, the execution unit 340 can also be used to: for a data optimization task in the initialization state, if the data optimizer fails to acquire and execute the data optimization task when the first waiting time exceeds a first preset time threshold, the task status is marked as expired.

[0203] In some embodiments, the execution unit 340 can also be used to: for a data optimization task that is in a scheduling state or an execution state, if the data optimization service does not receive the task execution result reported by the data optimizer when the second waiting time exceeds the second preset time threshold, the task status is marked as an execution timeout state.

[0204] In some embodiments, the acquisition unit 310 can also be used to: obtain database information related to the metadata change event by parsing the data snapshot information, and store the snapshot information into the target database indicated by the database information.

[0205] In some embodiments, the first generation unit 330 can also be used to: after generating the data optimization task corresponding to the metadata change event, store the data optimization task in the target database.

[0206] In some embodiments, the execution unit 340 can also be used to: send the task execution data corresponding to the data optimization task to the terminal, so that the terminal can visualize the data optimization task based on the task execution data.

[0207] Please see Figure 7 , Figure 7This is a second structural schematic diagram of the data optimization device provided in an embodiment of this application. The data optimization device 400 may include:

[0208] Display unit 410 is used to display the data management interface;

[0209] The second generation unit 420 is used to generate data optimization configuration information in response to a configuration operation input on the data management interface for a configuration target, wherein the configuration target is at least one of a data directory, a database, and a data table.

[0210] Sending unit 430 is used to send data optimization configuration information to the server so that the server can generate a data optimization task corresponding to the metadata change event based on the snapshot information of the metadata change event and the data optimization configuration information, and execute the data optimization task. The snapshot information of the metadata change event is obtained by the server when it detects the metadata change event of the target data table in the data lake.

[0211] The processing unit 440 is used to receive the task execution data corresponding to the data optimization task sent by the server, and to visualize the data optimization task on the graphical user interface based on the task execution data.

[0212] It should be noted that the functions of each module in the data optimization device 300 or data optimization device 400 in the embodiments of this application can be referred to the specific implementation of any of the above method embodiments, and will not be repeated here.

[0213] Each unit in the above-described device can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each unit.

[0214] For example, the data optimization device 300 can be integrated into a server that has storage and a processor and thus computing power, or the data optimization device 300 can be a server.

[0215] For example, the data optimization device 400 can be integrated into a terminal that has storage and a processor and thus computing capabilities, or the data optimization device 400 can be the terminal itself.

[0216] In some embodiments, this application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0217] Figure 8 A schematic structural diagram of the computer device provided in the embodiments of this application, such as Figure 8As shown, the computer device 500 may include: a communication interface 501, a memory 502, a processor 503, and a communication bus 504. The communication interface 501, memory 502, and processor 503 communicate with each other via the communication bus 504. The communication interface 501 is used for data communication between the device 500 and external devices. The memory 502 can be used to store software programs and modules, and the processor 503 runs the software programs and modules stored in the memory 502, such as the software programs for the corresponding operations in the aforementioned method embodiments.

[0218] In some embodiments, the computer device 500 can be a server, and the processor 503 can call software programs and modules stored in the memory 502 to perform the following operations: when a metadata change event of a target data table in the data lake is detected, obtain snapshot information of the metadata change event; receive data optimization configuration information, which is generated by the terminal in response to the configuration operation input for the configuration target in the data management interface, and the configuration target is at least one of the data directory, database and data table; generate a data optimization task corresponding to the metadata change event based on the snapshot information and the data optimization configuration information; and execute the data optimization task.

[0219] In some embodiments, the computer device 500 can be a terminal, and the processor 503 can call software programs and modules stored in the memory 502 to perform the following operations: displaying a data management interface; generating data optimization configuration information in response to configuration operations input on the data management interface for a configuration target, wherein the configuration target is at least one of a data directory, a database, and a data table; sending the data optimization configuration information to a server, so that the server generates a data optimization task corresponding to the metadata change event based on the snapshot information of the metadata change event and the data optimization configuration information, and executes the data optimization task, wherein the snapshot information of the metadata change event is obtained by the server when it detects a metadata change event of a target data table in the data lake; receiving task execution data corresponding to the data optimization task sent by the server, and visually displaying the data optimization task on a graphical user interface based on the task execution data.

[0220] In some embodiments, the computer device 500 may be integrated into a terminal or server that has storage and a processor and thus computing power, or the computer device 500 may be the terminal or server.

[0221] This application also provides a computer-readable storage medium for storing a computer program. This computer-readable storage medium can be applied to a computer device, and the computer program causes the computer device to execute the corresponding processes in the methods described above in the embodiments of this application; for brevity, further details are omitted here.

[0222] This application also provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding processes in the methods described above in the embodiments of this application. For brevity, these details will not be elaborated further here.

[0223] This application also provides a computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding processes in the methods described above in the embodiments of this application. For brevity, these details will not be elaborated further here.

[0224] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0225] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0226] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0227] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0228] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0229] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0230] In addition, the functional units in the embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0231] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer or a server) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0232] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data optimization method, characterized in that, The method includes: When a metadata change event of a target data table in the data lake is detected, a snapshot of the metadata change event is obtained; Receive data optimization configuration information, which is generated by the terminal in response to a configuration operation input on the data management interface targeting a configuration goal. The configuration goal is at least one of a data directory, a database, and a data table. Based on the snapshot information and the data optimization configuration information, a data optimization task corresponding to the metadata change event is generated; Perform the data optimization task.

2. The data optimization method as described in claim 1, characterized in that, The data optimization configuration information includes preset task generation conditions; The step of generating a data optimization task corresponding to the metadata change event based on the snapshot information and the data optimization configuration information includes: Based on historical data, optimize task data and snapshot information, and determine that the preset task generation conditions are met, generate the data optimization task corresponding to the metadata change event.

3. The data optimization method as described in claim 2, characterized in that, The step of generating a data optimization task corresponding to the metadata change event, based on historical data optimization task data and the snapshot information, and determining that the preset task generation conditions are met, includes: Based on the historical data optimization task data, obtain the previous data optimization task corresponding to the target data table, and the historical snapshot information corresponding to the previous data optimization task; Based on the historical snapshot information and the snapshot information, obtain the first number of data files generated within the time interval between the end time of the last data optimization task and the trigger time of the metadata change event; Based on the historical snapshot information and the snapshot information, obtain the second number of incremental files generated within the time interval between the end time of the last data optimization task and the trigger time of the metadata change event; Obtain the number of snapshots corresponding to the target data table; If the conditions for generating the data optimization task are met based on at least one of the previous data optimization task, the first quantity, the second quantity, the number of snapshots, and the snapshot information, a data optimization task corresponding to the metadata change event is generated.

4. The data optimization method according to any one of claims 1-3, characterized in that, When a metadata change event of a target data table in the data lake is detected, the snapshot information of the metadata change event is obtained, including: When the unified metadata system on the server receives snapshot information of metadata change events of the target data table in the data lake, it pushes the snapshot information to the event queue of the event message platform on the server through the unified metadata system. When the data optimization service on the server detects that there is a snapshot of the metadata change event in the event queue of the event message platform, the snapshot information is retrieved from the event queue.

5. The data optimization method according to any one of claims 1-3, characterized in that, The execution of the data optimization task includes: After the data optimization task corresponding to the metadata change event is generated through the data optimization service on the server, the task status of the data optimization task is marked as the initialization state; After successfully obtaining the data optimization task from the data optimization service through the data optimizer on the server, the data optimization task waits to be executed inside the data optimizer, and the task status is marked as scheduled. When the data optimizer confirms the task to the data optimization task and starts execution, the task status is marked as in execution. When the data optimization task is completed, the data optimizer reports the task execution result to the data optimization service. If the task execution result is successful, the task status is marked as "analysis in progress". After clearing the task record of the data optimization task or determining the next execution action of the target data table where the data optimization task is located, based on the task type and task execution result of the data optimization task, the task status is marked as successful execution.

6. The data optimization method as described in claim 5, characterized in that, The method further includes: For the data optimization task in the initialization state, if the data optimizer fails to acquire and execute the data optimization task when the first waiting time exceeds the first preset time threshold, the task status is marked as expired.

7. The data optimization method as described in claim 5, characterized in that, The method further includes: For the data optimization task that is in the scheduling or execution state, if the data optimization service does not receive the task execution result reported by the data optimizer when the second waiting time exceeds the second preset time threshold, the task status is marked as execution timeout state.

8. The data optimization method according to any one of claims 1-3, characterized in that, The method further includes: By parsing the data snapshot information, the database information involved in the metadata change event is obtained, and the snapshot information is stored in the target database indicated by the database information.

9. The data optimization method as described in claim 8, characterized in that, The method further includes: After generating the data optimization task corresponding to the metadata change event, the data optimization task is stored in the target database.

10. The data optimization method according to any one of claims 1-3, characterized in that, The method further includes: The task execution data corresponding to the data optimization task is sent to the terminal so that the terminal can visualize the data optimization task based on the task execution data.

11. A data optimization method, characterized in that, The method includes: Displays the data management interface; In response to a configuration operation input on the data management interface targeting a configuration objective, data optimization configuration information is generated, wherein the configuration objective is at least one of a data directory, a database, and a data table; The data optimization configuration information is sent to the server so that the server generates a data optimization task corresponding to the metadata change event based on the snapshot information of the metadata change event and the data optimization configuration information, and executes the data optimization task. The snapshot information of the metadata change event is obtained by the server when it detects the metadata change event of the target data table in the data lake. Receive the task execution data corresponding to the data optimization task sent by the server, and visualize the data optimization task on the graphical user interface based on the task execution data.

12. A data optimization device, characterized in that, The device includes: The acquisition unit is used to acquire snapshot information of the metadata change event when a metadata change event of a target data table in the data lake is detected. A receiving unit is used to receive data optimization configuration information, which is generated by the terminal in response to a configuration operation input on the data management interface for a configuration target. The configuration target is at least one of a data directory, a database, and a data table. The first generation unit is used to generate a data optimization task corresponding to the metadata change event based on the snapshot information and the data optimization configuration information. An execution unit is used to perform the data optimization task.

13. A data optimization device, characterized in that, The device includes: The display unit is used to display the data management interface; The second generation unit is used to generate data optimization configuration information in response to the configuration operation input for the configuration target in the data management interface, wherein the configuration target is at least one of a data directory, a database, and a data table; The sending unit is used to send the data optimization configuration information to the server, so that the server generates a data optimization task corresponding to the metadata change event based on the snapshot information of the metadata change event and the data optimization configuration information, and executes the data optimization task. The snapshot information of the metadata change event is obtained by the server when it detects the metadata change event of the target data table in the data lake. The processing unit is used to receive the task execution data corresponding to the data optimization task sent by the server, and to visualize the data optimization task on a graphical user interface based on the task execution data.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the data optimization method as described in any one of claims 1-11.

15. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, and the processor executing the data optimization method as described in any one of claims 1-11 by calling the computer program stored in the memory.

16. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the data optimization method according to any one of claims 1-11.

Images