A method, apparatus, and storage medium for optimizing ETL data waiting time.

Abstract

This invention discloses a method, apparatus, and storage medium for optimizing ETL data waiting. The method includes: receiving multiple sets of data content; obtaining the number of tasks currently awaiting verification for each data content based on the historical tasks triggered by each data content for associated verification; obtaining the data content after completing the verification of the corresponding number of tasks; and determining whether the data content after completing the verification of the corresponding number of tasks can cover the data content required for the ETL process by comparing it with the data content after completing the verification of the corresponding number of tasks. If it cannot cover the data content, the process proceeds to wait for the ETL process to be executed; if it can cover the data content, the ETL process is executed. This invention sets a preset threshold for priority verification, prioritizing the processing of tasks whose current number of tasks to be verified is less than the preset threshold, thereby reducing data processing waiting time. When the total amount of data information of the tasks currently awaiting verification and the data information after the ETL operation is less than the original data information, the ETL operation is performed, reducing the waiting time before the ETL operation and improving the efficiency of ETL.

Description

Technical Field

[0001] This invention relates to the field of data processing optimization technology, and in particular to a method, apparatus and storage medium for optimizing ETL data waiting. Background Technology

[0002] Power data primarily originates from the various stages of power generation and consumption, including generation, transformation, distribution, consumption, and dispatch. It can be broadly categorized into three types: first, power grid operation and equipment testing or monitoring data; second, power company marketing data, such as transaction prices, electricity sales volume, and customer data; and third, power company management data. Analyzing power grid data, from generation and transmission to consumption, requires real-time data processing. By leveraging power big data analytics, potential patterns and dynamics can be identified from the massive amounts of data in the power system. For my country's extensive power grid system, due to the sheer volume and complexity of power grid data, it is typically stored regionally, with each region storing its data on its own local server. For example, taking a province as an example, the power grid data of different cities are usually not interconnected. Each city's power grid data is stored on its own local server. When it is necessary to summarize a certain aspect of the power grid data from each city, it is necessary to perform an Extract-Transform-Load (ETL) operation on the data from each city. This involves extracting, transforming, and loading the data from each city, and finally storing the useful power grid data from each city on the provincial server in a specific manner, while deleting the power grid data information that is not needed from each city from the provincial server.

[0003] Provincial-level servers upload power grid data to various municipal-level servers. By understanding the municipal-level servers' requests for raw data, the specific power grid data required for the ETL process can be determined. ETL is typically processed on a task-by-task basis within the server, corresponding to the power grid data requests from each municipal-level server. If historical tasks within the power grid data are not verified to ensure their data has been processed before performing ETL, computational resources may be wasted, potentially leading to the data being deemed problematic after ETL, resulting in data waste. Furthermore, theoretically, data after ETL cannot be used for normal verification. In addition, data transmission between different servers is primarily determined by network speed, transmission distance, and server processing capabilities. Since power grid data sources for different municipalities come from different servers, time differences exist when the provincial level receives power grid data from various municipalities, causing data waiting issues. Therefore, overcoming the shortcomings of this existing technology is a pressing problem to be solved in this technical field. Summary of the Invention

[0004] The technical problem this invention aims to solve is how to reduce data waiting time and improve the efficiency of the ETL process by optimizing ETL while ensuring data validity.

[0005] The present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a method for optimizing ETL data waiting, comprising:

[0007] Based on the historical tasks that triggered associated verifications for each set of data content that have been received, obtain the number of tasks currently waiting for verification for each data content;

[0008] Obtain the data content after verifying the number of completed tasks. By comparing the data content after verifying the number of completed tasks, determine whether it can cover the data content required for the ETL process. If it cannot cover the data content, proceed to wait for the ETL process to be executed. If it can cover the data content, execute the ETL process.

[0009] During the ETL execution process, if the sum of the remaining data content awaiting verification and the result data obtained from the ETL execution is less than or equal to the original data content during the ETL execution process, then the ETL process is executed, and the ETL result and the remaining data content awaiting verification are saved. Preferably, if the sum of the remaining data content awaiting verification and the result data obtained from the ETL execution is greater than the original data content during the ETL execution process, then the ETL execution process is maintained.

[0010] Preferably, the historical task triggered by the association verification based on each data content specifically includes:

[0011] The received data content is used to determine the historical task that triggered the associated verification based on one or more of the following: the source of the data content, the attributes of the data content, and the size of the data content.

[0012] The historical task requires multiple sets of data to be received before verification can be completed. If any data is missing, the verification process for the corresponding historical task cannot be performed.

[0013] Preferably, when entering a waiting period for an ETL process, and the ETL process includes at least a first ETL process and a second ETL process, the method further includes:

[0014] Obtain the total number of remaining tasks awaiting verification for each data item during the first ETL process, and the total number of remaining tasks awaiting verification for each data item during the second ETL process;

[0015] If the total number of remaining tasks awaiting verification for each data content during the first ETL process is less than or equal to a preset threshold, then this server or another server will be scheduled to prioritize the verification tasks for each data content during the first ETL process.

[0016] Preferably, when the total number of remaining tasks awaiting verification for each data content during the ETL process is greater than a preset threshold, the tasks awaiting verification are processed in the original order of this server or other servers.

[0017] Preferably, the execution of the ETL process and the saving of the ETL results and the remaining data currently awaiting task verification specifically include:

[0018] Before executing the ETL process, a copy of the remaining data content currently awaiting verification by the task is generated; after executing the ETL process, the ETL result and the copy of the remaining data content currently awaiting verification by the task are saved.

[0019] Once all verification tasks corresponding to the copy of the remaining data content currently awaiting verification are completed, delete the copy of the remaining data content currently awaiting verification.

[0020] Preferably, during the ETL process, the relationship between the sum of the remaining data content awaiting task verification and the result data obtained from the ETL process and the original data content during the ETL process is analyzed. Specifically, this includes:

[0021] After the analysis is triggered for the first time, record the first difference between the sum of the remaining data content currently waiting for task verification and the result data obtained from the ETL process and the original data content waiting for the ETL process to be executed;

[0022] The ETL compression ratio is obtained by comparing the result data obtained during the ETL process with the original data content during the ETL process. After the analysis is triggered for the first time, the number of tasks associated with a newly acquired set or more sets of data content is cleared to zero. If the result of the newly acquired set or more sets of data content after being weighted by the ETL compression ratio is greater than or equal to the first difference, the corresponding analysis content is triggered for the second time.

[0023] Preferably, the ETL compression ratio is obtained by calculation of the corresponding ETL process executed in the past; or by calculation through simulation operation.

[0024] In a second aspect, the present invention also provides an apparatus for optimizing ETL data waiting, used to implement the method for optimizing ETL data waiting described in the first aspect, the apparatus comprising:

[0025] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the method for optimizing ETL data waiting as described in the first aspect.

[0026] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the method for optimizing ETL data waiting as described in the first aspect.

[0027] This invention sets a preset threshold for priority verification during the data validity verification stage before ETL operations. When the number of tasks waiting to be verified is less than the preset threshold, it provides a mechanism to schedule the current server or other servers to prioritize the processing of the currently waiting tasks, enabling the invention to complete data verification as early as possible and reducing the waiting time for the currently waiting task data. In addition, by adjusting the server's logic for triggering ETL operations, if the sum of the remaining data content of the currently waiting task verification and the result data obtained from the ETL process is less than or equal to the original data content waiting to be executed in the ETL process, then the ETL process is executed and the ETL result is saved. This reduces the time spent verifying the validity of the currently waiting tasks, thereby reducing the server's ETL operation time and improving the efficiency of the ETL process. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0029] Figure 1 This is a flowchart of a method for optimizing ETL data waiting provided by an embodiment of the present invention;

[0030] Figure 2 This is a flowchart of step 202 provided in an embodiment of the present invention;

[0031] Figure 3 This is a flowchart of the method for performing the ETL process in step 203 of the present invention.

[0032] Figure 4 This is a flowchart of the method for waiting to execute the ETL process in step 203, provided in an embodiment of the present invention.

[0033] Figure 5This is a schematic flowchart of a device for optimizing ETL data waiting provided in an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0035] In the description of this invention, the terms "inner", "outer", "longitudinal", "lateral", "upper", "lower", "top", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and do not require that this invention must be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0036] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0037] Example 1:

[0038] Embodiment 1 of the present invention provides an optimization method for data ETL, such as... Figure 1 As shown, it includes:

[0039] Step 201: Based on the historical tasks that triggered the associated verification of each set of data content, obtain the number of tasks currently waiting for verification for each set of data content.

[0040] ETL primarily involves extracting, transforming, and loading data. It extracts useful data from otherwise messy and complex datasets and stores it to facilitate direct utilization of the data and save server storage space. Taking electricity data as an example, the useful data content differs for different power sectors. For instance, for the marketing department of a power company, useful data often includes transaction prices, electricity sales volume, and customer data; for the equipment repair department, useful data might include the distribution of power supply equipment, its specific location, and its usage status. After receiving the relevant data, power companies typically categorize it according to actual needs (e.g., departmental needs). Different actual needs lead to different data categorizations, and the number of categories corresponds to the number of sets of data content. Each set of data corresponds to multiple tasks. For ease of understanding, let's take electricity sales as a set of data for explanation. For example, if the electricity sales department obtains the monthly electricity sales data for 50 households in a certain community, then the data content at this time is the electricity sales data for those 50 households in that month. The corresponding task can be understood as the execution action of each household purchasing electricity that month, and the number of tasks at this time is 50. The historical tasks in this embodiment of the invention refer to the tasks corresponding to all data content before verification. By using the tasks corresponding to all data content, as well as the currently verified tasks, the number of tasks currently awaiting verification for each data content can be obtained. For example, if the number of tasks corresponding to all data content is 50, and the number of verified tasks is 30, then the number of tasks currently awaiting verification is 20. It is worth noting that tasks and task quantities need to be strictly distinguished. For a specific task, a strict correspondence is required. For example, if A purchases 30 units of electricity and B purchases 40 units of electricity, then A purchasing 30 units of electricity is one task. A needs to be matched with 30 units of electricity to constitute one task, and the number of tasks at this time is 2.

[0041] Step 202: Obtain the data content after verifying the number of completed tasks. By comparing the data content after verifying the number of completed tasks, determine whether it can cover the data content required for the ETL process. If it cannot cover the data content, proceed to wait for the ETL process to be executed. If it can cover the data content, execute the ETL process.

[0042] The purpose of verifying data content in this embodiment of the invention is actually to validate the corresponding tasks and whether the corresponding tasks have processed the original data. Using the current time as a node, the data content after the verification of the corresponding number of tasks is obtained. By comparing the data content after the verification of the corresponding number of tasks with the data content required for the ETL process, it is determined whether to perform the ETL operation. The data content required for ETL is determined according to actual needs (e.g., electricity sales data from an electricity sales department). If the data content after the verification of the corresponding number of tasks completely covers the data content required for ETL, it means that all the required data has been verified, and the data content corresponding to the tasks currently awaiting verification is non-essential (this part of the data will be deleted later), and the ETL process is executed. When the data content cannot be covered, it indicates that there are omissions in the verification of the required data, and the process enters a waiting state for the ETL process to be executed.

[0043] Step 203: During the ETL process, if the sum of the remaining data content waiting for the current task verification and the result data obtained from the ETL process is less than or equal to the original data content during the ETL process, then execute the ETL process and save the ETL result and the remaining data content waiting for the current task verification.

[0044] The ETL (Extended Time To Live) waiting step ensures the accuracy of the ETL results on previously validated data. ETL transforms useful information according to ETL rules and stores it on the server. If the data corresponding to the currently pending verification task cannot be directly used for verification after ETL, a copy of the data needs to be stored on the server for verification by the server itself or other servers. Theoretically, with fixed ETL rules (i.e., fixed required information) and identical data, the memory usage after ETL transformation is constant. To reduce server storage pressure, the decision to execute ETL is made by analyzing the sum of the remaining data and the ETL results, compared to the memory usage of the original data during the ETL process. When the sum of the remaining data content awaiting task verification and the result data obtained from the ETL process is less than or equal to the original data content awaiting ETL execution, it indicates that the storage space occupied by the overall data content after ETL execution is gradually decreasing, and the memory consumption of the data content on the server is decreasing. At this point, performing ETL can save server memory. Furthermore, this embodiment of the invention does not perform ETL operations after all data content corresponding to all tasks has been verified, and it ensures the accuracy of the ETL operation, thereby reducing the ETL waiting time for verification and improving the server's ETL efficiency.

[0045] This invention adjusts the logic of the server triggering the ETL operation. After ensuring the validity of the data content (validity means that the task corresponding to the data content to be ETL'd has been verified before ETL, indicating that the data content is valid) is completed, if the sum of the remaining data content currently awaiting verification and the result data obtained from the ETL process is less than or equal to the original data content waiting to be executed, the ETL operation is performed directly, and the ETL result and the remaining data content currently awaiting verification are saved. This reduces the time spent verifying the validity of the currently awaiting verification task, thereby reducing the server's ETL operation time and improving the efficiency of the ETL process.

[0046] To fully illustrate the solution of this invention, the following detailed description of the embodiments of this invention is provided. Furthermore, if, during the waiting period for ETL execution, the sum of the remaining data content awaiting verification of the current task and the result data obtained from the ETL execution process is found to be greater than the original data content during the waiting period for ETL execution, the waiting period for ETL execution is maintained.

[0047] When the sum of the remaining data awaiting task verification and the result data obtained from the ETL process exceeds the original data content awaiting ETL execution, it indicates that the storage space occupied by the overall data content after ETL execution is gradually increasing, the data content is consuming more memory on the server, and thus increasing the server's pressure. In this case, the waiting period for ETL execution continues. The principle here is the same as that in step 203, and will not be elaborated further.

[0048] Furthermore, the historical task triggered by the association verification based on each data content specifically involves: for the received data content, determining the historical task triggered by the association verification based on one or more of the following: the source of the data content, the attributes of the data content, and the size of the data content; wherein, the historical task requires multiple sets of data content to be received in order to complete the verification, and if any data content is missing, the verification process of the corresponding historical task cannot be performed.

[0049] Depending on the requirements, each set of data content will be different, and the corresponding tasks will also differ. Based on actual needs, for example, the historical tasks triggered by the associated verification of data content are determined according to the data content's source (e.g., electricity usage in different cities and counties), data content attributes (electricity transaction price, electricity sales volume, electricity customers, etc.), and data content dimensions. To ensure that the saved ETL results contain the required data and that no necessary information is missing, the historical tasks described in this embodiment of the invention require multiple sets of data content to be received to complete the verification. If any data content is missing, the verification process for the corresponding historical task cannot be performed.

[0050] Furthermore, in this embodiment of the invention, the process enters a waiting state for the execution of an ETL process. When the ETL process includes at least a first ETL process and a second ETL process, such as... Figure 2 As shown, the method also includes:

[0051] Step 301: Obtain the total number of remaining tasks awaiting verification for each data content during the first ETL process, and the total number of remaining tasks awaiting verification for each data content during the second ETL process.

[0052] In this invention, multiple sets of data content are simultaneously stored on a central server. For each different branch server, the ETL results obtained after ETL vary depending on the requirements. The first and second ETL operations in this invention are merely markers used to distinguish ETL operations performed on data content with different requirements. Within the central server, multiple parallel ETL operations may be performed simultaneously based on the needs of different branch servers.

[0053] Step 302: If the total number of remaining tasks waiting to be verified for each data content during the first ETL process is less than or equal to a preset threshold, then schedule this server or other servers to prioritize the verification tasks for each data content during the first ETL process.

[0054] The preset threshold for the total number of remaining tasks in this invention is typically set considering server performance. When this preset threshold is set, if the total number of remaining tasks awaiting verification for each data content during the first ETL process is less than or equal to the preset threshold, the server or another server is scheduled to prioritize processing the verification tasks for each data content during the first ETL process. This allows the first ETL to quickly complete the verification tasks and perform the ETL operation, reducing the pressure on the server to execute ETL in parallel. By comparing the total number of remaining tasks awaiting verification with the preset threshold, the server is triggered to determine whether to prioritize processing the verification tasks for each data content during the ETL process. By reasonably setting the corresponding preset threshold for the total number of remaining tasks to be prioritized for verification, the tasks to be verified in the ETL process are optimized, reducing the waiting time for tasks currently awaiting verification in the ETL process. For the data content corresponding to the currently pending verification task, it may be verified by the ETL server itself or by other servers. However, for the server, it is not only necessary to process this data content, but also to process other data content (data unrelated to the data content of this invention). In the server's process order, the data content that needs to be verified may be processed later, resulting in a longer waiting time. By triggering the server with the corresponding comparison command, the server can prioritize the data tasks that need to be compared, thereby saving the waiting time of the currently pending task.

[0055] To reduce server requirements and avoid prioritizing the processing of multiple data verification tasks in the ETL process simultaneously, when the total number of remaining tasks awaiting verification for each data content in the ETL process exceeds a preset threshold, the tasks awaiting verification will be processed according to the original order of this server or other servers.

[0056] Furthermore, the ETL process is executed, and the ETL results and the remaining data currently awaiting task verification are saved, such as... Figure 3 As shown, it specifically includes:

[0057] Step 401: Before executing the ETL process, generate a copy of the remaining data content that is currently waiting for task verification.

[0058] ETL is essentially a process of extracting, transforming, and loading useful data content. This process usually involves compressing the data content before storage. The valid data content after ETL processing cannot be used as the data content currently awaiting verification. In order to ensure the validity of the data content in this embodiment of the invention, all useful information needs to be verified one by one. Before performing ETL, the remaining data content currently awaiting verification by the task is copied into a duplicate for subsequent tasks to verify the data.

[0059] Step 402: After performing the ETL process, save a copy of the ETL result and the remaining data content currently awaiting verification by the task.

[0060] After ETL is performed, the corresponding ETL operation is accurate for the data content that has already been verified. The remaining data content that is currently waiting for verification can continue to be used as data content waiting for verification.

[0061] Step 403: After all verification tasks corresponding to the copy of the remaining data content currently awaiting verification are completed, delete the copy of the remaining data content currently awaiting verification.

[0062] To conserve server memory, once all copies of the remaining data content awaiting verification by the current task have been verified, we can assume that the ETL result obtained before verifying the copy is accurate and valid. If the ETL result is confirmed to be valid, the copy's data content is no longer useful, and deleting it alleviates the pressure on server storage space occupied by the copy. This invention effectively reduces the time spent on data content verification and improves the efficiency of data ETL by performing ETL operations on the data content in advance. By generating copies of the data content awaiting verification and verifying each task within those copies, the validity of the ETL result is ensured. Furthermore, during the ETL execution process, the relationship between the sum of the remaining data content awaiting verification and the result data obtained from the ETL process and the original data content awaiting ETL execution is analyzed. Figure 4 As shown, it specifically includes:

[0063] Step 501: After the analysis is triggered for the first time, record the first difference between the sum of the remaining data content currently waiting for task verification and the result data obtained from the ETL process and the original data content waiting to be executed in the ETL process.

[0064] The conditions for triggering the analysis in this embodiment can be set according to the actual situation. The first difference in this embodiment changes according to the conditions for triggering the analysis for the first time. For example, when the data content after completing the verification of the corresponding number of tasks accounts for 90% of the original data content, the first analysis is triggered to obtain the first difference. Through the first analysis, the first difference between the sum of the remaining data content waiting for task verification and the result data obtained from the ETL process and the original data content waiting to be executed in the ETL process is obtained.

[0065] Step 502: Based on the ratio of the result data obtained from the ETL process to the original data content during the ETL process, the ETL compression ratio is obtained. After the analysis is triggered for the first time, the number of tasks associated with a newly acquired set or more sets of data content is cleared to zero, and the result of the newly acquired set or more sets of data content after being weighted by the ETL compression ratio is greater than or equal to the first difference, the corresponding analysis content is triggered for the second time.

[0066] In this embodiment, after the first trigger analysis, the difference between the result data obtained from the ETL process and the original data content during the ETL process is obtained to determine the ETL compression ratio. The compression ratio indicates the progress of the corresponding data content verification. By comparing the weighted compression ratio of each set of data content with the first difference, a second analysis of the corresponding data content is triggered until an ETL result with acceptable error is output. The acceptable error range is determined based on the actual situation. The ETL compression ratio is obtained through calculations based on historical executions of the corresponding ETL process; or through simulation calculations.

[0067] In this embodiment, a preset threshold for priority verification is set during the data validity verification stage before ETL operations. When the number of tasks waiting to be verified is less than the preset threshold, the system schedules the current server or other servers to prioritize processing the tasks currently awaiting verification. This allows the invention to complete data verification as early as possible, reducing the waiting time for the data of the tasks currently awaiting verification. In addition, by adjusting the logic of the server triggering the ETL operation, when the total amount of data information of the tasks currently awaiting verification and the data information after the ETL operation is less than the memory occupied by the original data information, the ETL operation is performed directly, and the ETL result is saved. This reduces the time spent verifying the validity of the tasks currently awaiting verification, thereby reducing the time the server spends on the ETL operation and improving the efficiency of the ETL process.

[0068] Example 2:

[0069] like Figure 5The diagram shown is an architectural schematic of an apparatus for optimizing ETL data waiting according to an embodiment of the present invention. The apparatus for optimizing ETL data waiting in this embodiment includes one or more processors 21 and a memory 22. Figure 5 Take a processor 21 as an example.

[0070] Processor 21 and memory 22 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0071] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the method for optimizing ETL data wait in Embodiment 1. The processor 21 executes the method for optimizing ETL data wait by running the non-volatile software program and instructions stored in the memory 22.

[0072] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0073] The program instructions / modules are stored in the memory 22 and, when executed by one or more processors 21, perform the optimized ETL data waiting method described in Embodiment 1.

[0074] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the optimized ETL data waiting method provided in Embodiment 1 of this invention.

[0075] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0076] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for optimizing ETL data waiting, characterized in that, include: The system will receive multiple sets of data from the power sector and, based on the historical tasks that trigger associated verification for each data set, obtain the number of tasks currently awaiting verification for each data set. Obtain the data content after verifying the number of completed tasks. By comparing the data content after verifying the number of completed tasks, determine whether the verified data content contains all the data content required for the ETL process. If it does not contain the data content, proceed to wait for the ETL process to be executed. If it does contain the data content, execute the ETL process. If, during the ETL process, the sum of the size of the remaining data content awaiting verification by the current task and the size of the result data obtained from the ETL process is less than or equal to the size of the original data content during the ETL process, then before executing the ETL process, a copy of the remaining data content awaiting verification by the current task is generated, the ETL process is executed, and the ETL result and the copy of the remaining data content awaiting verification by the current task are saved.

2. The method for optimizing ETL data waiting according to claim 1, characterized in that, This also includes maintaining the ETL process in the waiting-to-execute state if, during the ETL process, the sum of the remaining data content awaiting verification of the current task and the result data obtained from the ETL process is greater than the original data content during the waiting-to-execute-ETL process.

3. The method for optimizing ETL data waiting according to claim 1, characterized in that, The historical tasks triggered by the association verification based on each data content are specifically as follows: The received data content is used to determine the historical task that triggered the associated verification based on one or more of the following: the source of the data content, the attributes of the data content, and the size of the data content. The historical task requires multiple sets of data to be received before verification can be completed. If any data is missing, the verification process for the corresponding historical task cannot be performed.

4. The method for optimizing ETL data waiting according to claim 1, characterized in that, When entering a waiting period for ETL execution, and the ETL process includes at least a first ETL process and a second ETL process, the method further includes: Obtain the total number of remaining tasks awaiting verification for each data item during the first ETL process, and the total number of remaining tasks awaiting verification for each data item during the second ETL process; If the total number of remaining tasks awaiting verification for each data content during the first ETL process is less than or equal to a preset threshold, then this server or another server will be scheduled to prioritize the verification tasks for each data content during the first ETL process.

5. The method for optimizing ETL data waiting according to claim 4, characterized in that, During ETL, if the total number of remaining tasks awaiting verification for each data content exceeds a preset threshold, the tasks awaiting verification will be processed according to the original order of this server or other servers.

6. A method for optimizing ETL data waiting according to claim 1, characterized in that, During the ETL process, the relationship between the sum of the remaining data content awaiting task verification and the result data obtained from the ETL process, and the original data content awaiting ETL execution, is analyzed. Specifically, this includes: After the analysis is triggered for the first time, record the first difference between the sum of the remaining data content currently waiting for task verification and the result data obtained from the ETL process and the original data content waiting for the ETL process to be executed; The ETL compression ratio is obtained by comparing the result data obtained during the ETL process with the original data content during the ETL process. After the analysis is triggered for the first time, the number of tasks associated with a newly acquired set or more sets of data content is cleared to zero. If the result of the newly acquired set or more sets of data content after being weighted by the ETL compression ratio is greater than or equal to the first difference, the corresponding analysis content is triggered for the second time.

7. The method for optimizing ETL data waiting according to claim 6, characterized in that, The ETL compression ratio is obtained by calculation of the corresponding ETL process executed in the past; or by calculation through simulation operation.

8. An apparatus for optimizing ETL data waiting, characterized in that, include: At least one processor; At least one memory; The at least one processor and the at least one memory are communicatively connected to each other. The at least one memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the optimized ETL data waiting method provided by any one of claims 1-7 of the present invention.

9. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions that are executed by one or more processors to perform the method for optimizing ETL data waiting provided in any one of claims 1-7.

Images