Data warehouse service optimization method, device and equipment and storage medium

Abstract

The present application relates to a kind of data warehouse service optimization method, by dividing data warehouse service into data warehouse service (DW) and data application service (DA) two independent modules, and introduce task driving mechanism, realize decoupling between two. Data warehouse service when processing data task in preset time period, after completion, will store processing result to database, and pass task completion information by task driving mechanism. Data application service is processed by listening to task driving mechanism, after receiving task completion information, to the data result stored, generates the final output of data warehouse service. The present application significantly improves the flexibility and efficiency of system when processing large-scale data task, especially in the face of data adjustment and historical data backtracking scene, can effectively optimize resource utilization and task scheduling.

Description

Technical Field

[0001] This invention relates to the field of data processing and optimization technology, and in particular to a data warehouse service optimization method, apparatus, equipment and storage medium. Background Technology

[0002] Data warehouse systems play a vital role in modern enterprise information management. Following the data inflow and outflow process, a typical data warehouse architecture is usually divided into three layers: the source data layer (ODS), the data warehouse layer (DW), and the data application layer (DA).

[0003] The Source Data Layer (ODS) is the input layer of the data warehouse, responsible for collecting and integrating data from multiple data sources. The primary task of the ODS is to provide temporary storage and initial cleaning of raw data. This layer is designed to ensure that the raw data received by the data warehouse is up-to-date and structured, facilitating subsequent processing and analysis.

[0004] The Data Warehouse (DW) layer is responsible for thoroughly cleaning, processing, and transforming the data from the ODS layer. Data processed by the DW layer is validated, formatted, and of high quality, serving as the basis for critical business decisions. The key function of the DW layer is to ensure data consistency and accuracy, providing reliable foundational data for the data application layer.

[0005] The Data Application (DA) layer is the output of the data warehouse, extracting, filtering, and applying data to meet specific business needs. The output of the DA layer is typically used to generate reports, perform data analysis, or other business applications. The design goal of this layer is to refine data processing according to different business requirements to support diverse application scenarios.

[0006] In offline data warehouse systems, while immediate responses are not required, data processing tasks must be completed within a specified timeframe to support subsequent task chains. For example, in a regulatory reporting scenario within an investment management system, the system must complete the processing of upstream data and generate corresponding reports before a specific time point for the regulatory system to access and submit.

[0007] When upstream data changes, especially when processing data spanning a quarter or longer, the system needs to backtrack and reprocess the data for the entire period. In such cases, the performance of the data warehouse system often faces severe challenges, particularly when processing large amounts of historical data. Traditional data warehouse systems typically employ a batch processing model: data is extracted from the ODS (Operational Data Store), processed by the DW (Data Warehouse) layer, stored in the database, and finally extracted and processed by the DA (Data Analyzer) layer according to requirements. When data adjustments occur, the system needs to backtrack and process the data day by day, repeating the same data processing flow. This approach is often inefficient when dealing with large-scale data processing. Summary of the Invention

[0008] The main objective of this invention is to provide a data warehouse service optimization method, apparatus, device, and storage medium, aiming to solve the technical problem of low efficiency in processing large-scale data tasks caused by the tight coupling between data warehouse services and data application services in the prior art.

[0009] To achieve the above objectives, the present invention provides a data warehouse service optimization method, which includes the following steps:

[0010] The data warehouse service is split into a data warehouse service and a data application service, and a task-driven mechanism is constructed to decouple the data warehouse service and the data application service.

[0011] Data tasks with a preset time period are processed through a data warehouse service. The preset time period includes multiple time periods. After the data task of a certain time period is processed, the data result of the current time period is stored in the database, and the data task completion information of the current time period is sent to the task-driven mechanism.

[0012] By monitoring the task-driven mechanism through the data application service, when it is detected that the data warehouse service has completed the data task for the current time period, the data application service processes the data results for the current time period to obtain the data warehouse service processing result.

[0013] Optionally, after sending the data task completion information for the current time period to the task-driven mechanism, the method further includes:

[0014] Determine whether the data task in the current time period is the data task in the last time period of the preset time cycle;

[0015] If so, terminate the processing task of the data warehouse service;

[0016] If not, the data warehouse service continues to process the data tasks for the next time period until all data tasks for the preset time period have been processed.

[0017] Optionally, data tasks with a preset time period are processed through a data warehouse service, including:

[0018] The data warehouse service includes multiple data warehouse sub-services, and the load status of each data warehouse sub-service is determined.

[0019] The data tasks for the preset time period are divided according to the load of each data warehouse sub-service to obtain the same number of sub-data tasks as the data warehouse sub-services;

[0020] Each sub-data task is assigned to a data warehouse sub-service with the corresponding load, and each data warehouse sub-service processes the assigned sub-data task independently and in parallel.

[0021] Optionally, the processed data for the current time period is stored in a database, including:

[0022] The data results for the current time period are divided into blocks, and each data block is compressed using a compression algorithm to obtain compressed data blocks. The compressed data blocks are then stored in the database.

[0023] Optionally, before processing the data results for the current time period through the data application service, the method further includes:

[0024] Task events are generated through the task-driven mechanism and received through the data application service. The task events include processing data results for the current time period.

[0025] Optionally, generating task events through the task-driven mechanism includes:

[0026] Define task identifiers and task triggering conditions, wherein the task triggering conditions include the completion of data tasks for the current time period by the data warehouse service;

[0027] When the task triggering condition is met, the task-driven mechanism automatically generates the task event, which includes the task identifier and task type, for the data application service to listen to, receive and process.

[0028] Optionally, after constructing the task-driven mechanism, the following may also be included:

[0029] The task-driven mechanism records detailed execution logs for each data task. When an abnormal state occurs during the execution of a data task, the task-driven mechanism generates an error log and handles the abnormal state according to a pre-defined strategy. The error log is available for viewing by the system administrator.

[0030] Furthermore, to achieve the above objectives, the present invention also provides a data warehouse service optimization device, the data warehouse service optimization device comprising:

[0031] The task coordination and decoupling module splits the data warehouse service into a data warehouse service and a data application service, and constructs a task-driven mechanism to achieve decoupling between the data warehouse service and the data application service.

[0032] The data processing and storage module processes data tasks within a preset time period through a data warehouse service. The preset time period includes multiple time periods. After processing the data task for a certain time period, the module stores the data result of the current time period in the database and sends the data task completion information for the current time period to the task-driven mechanism.

[0033] The data application and execution module monitors the task-driven mechanism through the data application service. When it detects that the data warehouse service has completed the data task for the current time period, it processes the data results for the current time period through the data application service to obtain the data warehouse service processing result.

[0034] Furthermore, to achieve the above objectives, the present invention also provides a data warehouse service optimization device, which includes a memory, a processor, and a data warehouse service optimization program stored in the memory and executable on the processor. When the data warehouse service optimization program is executed by the processor, it implements the steps of the data warehouse service optimization method as described above.

[0035] Furthermore, to achieve the above objectives, the present invention also provides a computer storage medium storing a data warehouse service optimization program, which, when executed by a processor, implements the steps of the data warehouse service optimization method as described above.

[0036] This invention relates to a data warehouse service optimization method. By dividing the data warehouse service into two independent modules—a data warehouse service (DW) and a data application service (DA)—and introducing a task-driven mechanism, the two modules are decoupled. When processing data tasks within a preset time period, the data warehouse service stores the processing results in a database upon completion and transmits task completion information through the task-driven mechanism. The data application service, by listening to the task-driven mechanism, processes the stored data results upon receiving the task completion information, generating the final output of the data warehouse service. This invention significantly improves the system's flexibility and efficiency when handling large-scale data tasks, especially in scenarios involving data adjustments and historical data backtracking, effectively optimizing resource utilization and task scheduling. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the hardware operating environment of the device involved in the embodiment of the data warehouse service optimization device of the present invention;

[0038] Figure 2 This is a flowchart illustrating the first embodiment of the data warehouse service optimization method of the present invention;

[0039] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the data warehouse service optimization device of the present invention.

[0040] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0041] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0042] It should be noted that data warehouse systems play a vital role in modern enterprise information management. Following the data inflow and outflow process, a typical data warehouse architecture is usually divided into three layers: the source data layer (ODS), the data warehouse layer (DW), and the data application layer (DA).

[0043] The Source Data Layer (ODS) is the input layer of the data warehouse, responsible for collecting and integrating data from multiple data sources. The primary task of the ODS is to provide temporary storage and initial cleaning of raw data. This layer is designed to ensure that the raw data received by the data warehouse is up-to-date and structured, facilitating subsequent processing and analysis.

[0044] The Data Warehouse (DW) layer is responsible for thoroughly cleaning, processing, and transforming the data from the ODS layer. Data processed by the DW layer is validated, formatted, and of high quality, serving as the basis for critical business decisions. The key function of the DW layer is to ensure data consistency and accuracy, providing reliable foundational data for the data application layer.

[0045] The Data Application (DA) layer is the output of the data warehouse, extracting, filtering, and applying data to meet specific business needs. The output of the DA layer is typically used to generate reports, perform data analysis, or other business applications. The design goal of this layer is to refine data processing according to different business requirements to support diverse application scenarios.

[0046] In offline data warehouse systems, while immediate responses are not required, data processing tasks must be completed within a specified timeframe to support subsequent task chains. For example, in a regulatory reporting scenario within an investment management system, the system must complete the processing of upstream data and generate corresponding reports before a specific time point for the regulatory system to access and submit.

[0047] When upstream data changes, especially when processing data spanning a quarter or longer, the system needs to backtrack and reprocess the data for the entire period. In such cases, the performance of the data warehouse system often faces severe challenges, particularly when processing large amounts of historical data. Traditional data warehouse systems typically employ a batch processing model: data is extracted from the ODS (Operational Data Store), processed by the DW (Data Warehouse) layer, stored in the database, and finally extracted and processed by the DA (Data Analyzer) layer according to requirements. When data adjustments occur, the system needs to backtrack and process the data day by day, repeating the same data processing flow. This approach is often inefficient when dealing with large-scale data processing.

[0048] To address the aforementioned shortcomings, this invention provides a data warehouse service optimization device, referring to... Figure 1 , Figure 1 This is a schematic diagram of the hardware operating environment of the device involved in the embodiment of the data warehouse service optimization device of the present invention.

[0049] like Figure 1 As shown, the data warehouse service optimization device may include: a processor 1001, such as a CPU; a communication bus 1002; a user interface 1003; a network interface 1004; and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or stable non-volatile memory, such as disk storage. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0050] Those skilled in the art will understand that Figure 1 The hardware structure of the data warehouse service optimization device shown in the figure does not constitute a limitation on the data warehouse service optimization device. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0051] like Figure 1As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a data warehouse service optimization program. The operating system is a program that manages and controls the data warehouse service optimization equipment and software resources, supporting the operation of the network communication module, the user interface module, the data warehouse service optimization program, and other programs or software. The network communication module manages and controls the network interface 1004; the user interface module manages and controls the user interface 1003.

[0052] exist Figure 1 In the hardware structure of the data warehouse service optimization device shown, the network interface 1004 is mainly used to connect to the backend server and communicate with it; the user interface 1003 is mainly used to connect to the client and communicate with it; the processor 1001 can call the data warehouse service optimization program stored in the memory 1005 and perform the following operations:

[0053] The data warehouse service is split into a data warehouse service and a data application service, and a task-driven mechanism is constructed to decouple the data warehouse service and the data application service.

[0054] Data tasks with a preset time period are processed through a data warehouse service. The preset time period includes multiple time periods. After the data task of a certain time period is processed, the data result of the current time period is stored in the database, and the data task completion information of the current time period is sent to the task-driven mechanism.

[0055] By monitoring the task-driven mechanism through the data application service, when it is detected that the data warehouse service has completed the data task for the current time period, the data application service processes the data results for the current time period to obtain the data warehouse service processing result.

[0056] Furthermore, after sending the data task completion information for the current time period to the task-driven mechanism, the method further includes:

[0057] Determine whether the data task in the current time period is the data task in the last time period of the preset time cycle;

[0058] If so, terminate the processing task of the data warehouse service;

[0059] If not, the data warehouse service continues to process the data tasks for the next time period until all data tasks for the preset time period have been processed.

[0060] Furthermore, data tasks with preset time periods are processed through data warehouse services, including:

[0061] The data warehouse service includes multiple data warehouse sub-services, and the load status of each data warehouse sub-service is determined.

[0062] The data tasks for the preset time period are divided according to the load of each data warehouse sub-service to obtain the same number of sub-data tasks as the data warehouse sub-services;

[0063] Each sub-data task is assigned to a data warehouse sub-service with the corresponding load, and each data warehouse sub-service processes the assigned sub-data task independently and in parallel.

[0064] Furthermore, the processed data for the current time period is stored in the database, including:

[0065] The data results for the current time period are divided into blocks, and each data block is compressed using a compression algorithm to obtain compressed data blocks. The compressed data blocks are then stored in the database.

[0066] Furthermore, before processing the data results for the current time period through the data application service, the method further includes:

[0067] Task events are generated through the task-driven mechanism and received through the data application service. The task events include processing data results for the current time period.

[0068] Furthermore, generating task events through the task-driven mechanism includes:

[0069] Define task identifiers and task triggering conditions, wherein the task triggering conditions include the completion of data tasks for the current time period by the data warehouse service;

[0070] When the task triggering condition is met, the task-driven mechanism automatically generates the task event, which includes the task identifier and task type, for the data application service to listen to, receive and process.

[0071] Furthermore, after constructing the task-driven mechanism, it also includes:

[0072] The task-driven mechanism records detailed execution logs for each data task. When an abnormal state occurs during the execution of a data task, the task-driven mechanism generates an error log and handles the abnormal state according to a pre-defined strategy. The error log is available for viewing by the system administrator.

[0073] The specific implementation of the data warehouse service optimization device of the present invention is basically the same as the embodiments of the data warehouse service optimization method described below, and will not be repeated here.

[0074] The present invention also provides a data warehouse service optimization method based on the above-mentioned data warehouse service optimization equipment.

[0075] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the data warehouse service optimization method of the present invention.

[0076] This invention provides an embodiment of a data warehouse service optimization method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0077] In various embodiments of the data warehouse service optimization method, the executing entity is the data warehouse service optimization device.

[0078] The data warehouse service optimization method is applied to a data warehouse service optimization device, and the data warehouse service optimization method includes:

[0079] Step S10: The data warehouse service is split into a data warehouse service and a data application service, and a task-driven mechanism is constructed. The task-driven mechanism is used to decouple the data warehouse service and the data application service.

[0080] Data warehouse systems play a vital role in modern enterprise information management. Following the data inflow and outflow process, a typical data warehouse architecture is usually divided into three layers: the source data layer (ODS), the data warehouse layer (DW), and the data application layer (DA).

[0081] The Source Data Layer (ODS) is the input layer of the data warehouse, responsible for collecting and integrating data from multiple data sources. The primary task of the ODS is to provide temporary storage and initial cleaning of raw data. This layer is designed to ensure that the raw data received by the data warehouse is up-to-date and structured, facilitating subsequent processing and analysis.

[0082] The Data Warehouse (DW) layer is responsible for thoroughly cleaning, processing, and transforming the data from the ODS layer. Data processed by the DW layer is validated, formatted, and of high quality, serving as the basis for critical business decisions. The key function of the DW layer is to ensure data consistency and accuracy, providing reliable foundational data for the data application layer.

[0083] The Data Application (DA) layer is the output of the data warehouse, extracting, filtering, and applying data to meet specific business needs. The output of the DA layer is typically used to generate reports, perform data analysis, or other business applications. The design goal of this layer is to refine data processing according to different business requirements to support diverse application scenarios.

[0084] In offline data warehouse systems, while immediate responses are not required, data processing tasks must be completed within a specified timeframe to support subsequent task chains. For example, in a regulatory reporting scenario within an investment management system, the system must complete the processing of upstream data and generate corresponding reports before a specific time point for the regulatory system to access and submit.

[0085] When upstream data changes, especially when processing data spanning a quarter or longer, the system needs to backtrack and reprocess the data for the entire period. In such cases, the performance of the data warehouse system often faces severe challenges, particularly when processing large amounts of historical data. Traditional data warehouse systems typically employ a batch processing model: data is extracted from the ODS (Operational Data Store), processed by the DW (Data Warehouse) layer, stored in the database, and finally extracted and processed by the DA (Data Analyzer) layer according to requirements. When data adjustments occur, the system needs to backtrack and process the data day by day, repeating the same data processing flow. This approach is often inefficient when dealing with large-scale data processing.

[0086] In this embodiment, the data warehouse service is divided into two independent modules: Data Warehouse Service (DW) and Data Application Service (DA). To decouple these two modules, a task-driven mechanism is constructed. The main function of this mechanism is to coordinate the task transfer between DW and DA, allowing them to work independently without directly depending on each other's state. By monitoring and managing the state of tasks, the task-driven mechanism passes the processing results of the Data Warehouse Service to the Data Application Service, thereby making the entire system more flexible and scalable.

[0087] In practice, after the data warehouse service completes processing data tasks within a certain time period, it stores the processing results in the database and generates task completion information through a task-driven mechanism. This information is then passed to the data application service, notifying it that it can begin processing the data. Upon receiving the task completion information, the data application service, by listening to the task-driven mechanism, begins further processing of the stored data, such as analysis, summarization, or report generation.

[0088] This implementation ensures the independent operation of the data warehouse service and the data application service. Even if the data application service is temporarily unable to process data for some reason, the data warehouse service can still continue its data processing tasks without being affected by the status of the data application service. This decoupling significantly improves the efficiency and stability of the system when handling large-scale data tasks.

[0089] Example explanation:

[0090] In a financial system, the data warehouse service is responsible for processing daily transaction data. Whenever the data warehouse service finishes processing the transaction data for a certain day, it stores the results in the database and generates a task completion message. The task-driven mechanism captures this message and notifies the data application service. After receiving the notification, the data application service can start further analyzing the transaction data, such as generating end-of-day reports or conducting risk assessments. If the transaction volume on a certain day is particularly large, the data application service may need more time to process the data, while at this time the data warehouse service has already started processing the data tasks for the next day. This decoupling mechanism ensures the efficient operation of the system and does not affect the overall progress due to delays in a certain link.

[0091] Through the above steps, the system can more flexibly respond to the large-scale data processing requirements and effectively improve the task scheduling and resource utilization efficiency.

[0092] Step S20, process the data tasks for a preset time period through the data warehouse service. The preset time period includes multiple time segments. When the data tasks for a certain time segment are processed, store the data results obtained after processing for the current time segment in the database, and send the task completion message for the current time segment to the task-driven mechanism;

[0093] In this embodiment, the data warehouse service is responsible for processing the data tasks within a preset time period. Here, the "preset time period" refers to the time range associated with the data tasks themselves. The time period consists of multiple time segments, and each time segment can be one day, several hours, or any predefined time range. The data tasks are processed based on these time segments, thus ensuring that the data processing can proceed in an orderly manner according to the specified time nodes.

[0094] After the data warehouse service finishes processing the data tasks within a specific time segment, it stores the data results obtained within that time segment in the database. This step ensures that the data for each time segment is saved after processing for subsequent access or application.

[0095] After finishing processing the data tasks for a certain time segment and storing the results, the data warehouse service generates the task completion message for the current time segment and sends this message to the task-driven mechanism. The task-driven mechanism will then, based on this message, initiate the subsequent task processing process, such as notifying the data application service to start processing the stored data.

[0096] A specific implementation method could be as follows: The data warehouse service processes data tasks for each time period according to a preset time cycle. These tasks may involve operations such as data extraction, cleaning, transformation, and loading. Once a data task for a time period is completed, the resulting data is stored in the database. When a data task for a time period is completed, the data warehouse service generates a task completion signal and transmits it to other services or modules through a task-driven mechanism. Based on the received signal, the task-driven mechanism notifies the data application service to process the data results for that time period, ensuring the continuity and timeliness of the data processing flow.

[0097] Example explanation:

[0098] Suppose there is a system for real-time monitoring and analysis of financial market data. This system needs to process large amounts of transaction data and generate analytical reports. To meet business requirements, the system employs two parts: a data warehouse service (DW) and a data application service (DA), decoupled through a task-driven mechanism.

[0099] In financial markets, transaction data is dynamic, with millions of transaction records potentially generated every hour. To effectively process and analyze this data, the system divides the day into 24 time periods, each lasting one hour. Within each time period, the data warehouse service needs to process all transaction data from that period.

[0100] The preset time period here is a whole day, which includes a 24-hour period, with each hour being an independent period.

[0101] At the start of each hour, the data warehouse service automatically extracts all transaction records for that time period from transaction data sources (such as exchange APIs, internal system logs, etc.). The data warehouse service then cleans this data, removing duplicate records, correcting errors, and converting the data into a standard format for easier subsequent processing.

[0102] After cleaning and transforming the data for a given hour, the data warehouse service stores this processed data in a database. The database uses a columnar storage format to optimize query performance and storage efficiency.

[0103] Once the data warehouse service has completed processing and storing data for a given hour, it generates a task completion message. This message includes the hour's time stamp, processing status, and data volume, indicating that the data for that time period is ready for subsequent operations.

[0104] After generating task completion information, the data warehouse service sends it to the task-driven mechanism. Upon receiving this information, the task-driven mechanism immediately passes it to the data application service and initiates the corresponding task processing flow.

[0105] By following the steps above, we can flexibly and efficiently process large-scale daily data while maintaining the independence and real-time response capability of data tasks in different time periods. This not only improves the system's processing capacity but also greatly enhances its adaptability and stability in the face of sudden increases in data volume or abnormal situations.

[0106] Step S30: By monitoring the task-driven mechanism through the data application service, when it is detected that the data warehouse service has completed the data task for the current time period, the data application service processes the data result for the current time period to obtain the data warehouse service processing result.

[0107] In this embodiment, the data application service continuously monitors the task status of the data warehouse service through a task-driven monitoring mechanism. When the task-driven mechanism detects that the data warehouse service has completed the data processing task for the current time period, the data application service will receive a notification or signal indicating task completion.

[0108] Once the data application service receives the task completion information, it will immediately perform subsequent processing on the results of the current time period. This processing may include data analysis, summarization, report generation, and other operations, depending on business needs.

[0109] By processing the data results for the current time period, the data application service generates the final data warehouse service processing result. This result may be used to generate business reports, data analysis reports, or as input data for other systems to further support the enterprise's decision-making process.

[0110] A specific implementation could be as follows: The data application service is configured with a monitoring mechanism to monitor the signals of the task-driven mechanism in real time. Whenever the data warehouse service completes a data task for a given time period and updates the task-driven mechanism, the data application service receives a notification. Upon receiving the task completion information, the data application service initiates the corresponding data processing flow. This processing may include data aggregation and analysis, anomaly detection, trend analysis, etc., and generates the required processing results based on business logic. After processing, the data application service generates corresponding results, such as daily, weekly, monthly reports, or other forms of analytical reports. These results may be automatically pushed to decision-makers, archived in the system, or used to drive other business processes.

[0111] Example explanation:

[0112] Suppose there is a data warehouse system used to monitor and analyze sales data from an e-commerce platform. This system needs to process hourly transaction data in real time and generate reports and analysis results. The system includes data warehouse services (DW) and data application services (DA), which are coordinated through a task-driven mechanism.

[0113] Data processing task: At the end of each hour, the data warehouse service extracts all order data for that hour from the e-commerce platform's transaction database. This data includes detailed information such as order number, product ID, quantity purchased, price, user ID, and transaction time.

[0114] Data cleaning and transformation: The extracted raw data may contain erroneous, duplicate, or incomplete information. Data warehouse services clean this data (such as removing duplicate records and correcting erroneous data) and transform the data into a standardized format to facilitate subsequent processing and analysis.

[0115] Data storage: The cleaned and transformed data is stored in a database to ensure that all transaction data is properly preserved and available for subsequent querying and processing.

[0116] Generate task completion information: Whenever the data warehouse service completes a data processing task for a certain hour, it will generate task completion information. This information includes the processing time period (e.g., "August 7, 2024, 14:00-15:00"), the processing status (success or failure), the amount of data processed (e.g., the total number of transaction records), and other relevant metadata.

[0117] Task-driven mechanism transmission: Task completion information is transmitted to the data application service through a task-driven mechanism. This mechanism records task completion information for each time period and ensures that the data application service receives this information promptly to initiate the next data processing operation.

[0118] Listening task-driven mechanism: The data application service is configured to use a real-time listening task-driven mechanism. When a data processing task for a certain time period is completed, the data application service will immediately receive a notification and begin processing the data results for that time period stored in the database.

[0119] Data Analysis and Report Generation: The data application service performs in-depth analysis of the received transaction data. For example, the system can calculate key performance indicators (KPIs) such as total sales, number of orders, and average order value for the specified time period. Furthermore, the data application service may also conduct user behavior analysis to identify which products are most popular and which promotional activities are most effective.

[0120] Real-time decision support: The analytics results are used to generate real-time sales reports, which can be automatically pushed to management to help them make business decisions. For example, if a product sells exceptionally well within a given hour, management can decide whether to increase inventory or extend the promotion.

[0121] Managing peak periods: During shopping festivals or promotional events, transaction volumes may increase significantly. Data warehouse services can dynamically allocate more computing resources to process this data, ensuring that data tasks for each time period are completed on time. A task-driven mechanism ensures that data application services can begin analysis immediately after the data warehouse service completes processing, maintaining the efficient operation of the entire system.

[0122] Anomaly Handling and Recovery: If an anomaly occurs during data processing within a certain time period (such as data loss or processing failure due to system failure), the task-driven mechanism will record the anomaly information and initiate an error handling process. The system may automatically retry the processing task or notify the administrator for manual intervention. This minimizes the risk of data loss and ensures the continuity and reliability of data processing.

[0123] Real-time Marketing and Inventory Adjustment: The analytical reports generated by the data application service not only provide services to management but can also directly drive other business systems. For example, the inventory management system can automatically adjust inventory levels based on the analysis results, while the marketing system can launch new promotional campaigns to respond to current market demands.

[0124] Long-term analysis and trend forecasting: Hourly data results are aggregated and used for long-term trend analysis, helping companies predict future sales trends and develop more effective strategic plans.

[0125] By using a task-driven mechanism, the data warehouse service and data application service are decoupled, ensuring that the system can efficiently and flexibly process and utilize large amounts of data.

[0126] This invention relates to a data warehouse service optimization method. By dividing the data warehouse service into two independent modules—a data warehouse service (DW) and a data application service (DA)—and introducing a task-driven mechanism, the two modules are decoupled. When processing data tasks within a preset time period, the data warehouse service stores the processing results in a database upon completion and transmits task completion information through the task-driven mechanism. The data application service, by listening to the task-driven mechanism, processes the stored data results upon receiving the task completion information, generating the final output of the data warehouse service. This invention significantly improves the system's flexibility and efficiency when handling large-scale data tasks, especially in scenarios involving data adjustments and historical data backtracking, effectively optimizing resource utilization and task scheduling.

[0127] Furthermore, a second embodiment of the data warehouse service optimization method of the present invention is proposed. In step S20 above, after sending the data task completion information for the current time period to the task-driven mechanism, the method further includes:

[0128] Step a1: Determine whether the data task in the current time period is the data task of the last time period in the preset time cycle;

[0129] Step a2: If yes, end the processing task of the data warehouse service;

[0130] Step a3: If not, continue processing the data tasks for the next time period through the data warehouse service until all data tasks for the preset time period have been processed.

[0131] In this embodiment, after the data warehouse service completes processing data tasks for a certain time period and sends task completion information, the system checks whether the currently processed time period is the last time period within a preset time cycle. For example, if the preset time cycle is one day (24 hours), the system will determine whether the currently processed time period is the last hour of that day. If so, it means that all data tasks within the entire preset time cycle have been completed.

[0132] If the determination result indicates that the current time period is the last time period, the data warehouse service will end all data processing tasks for the day or period. This means that the data warehouse service's processing flow is complete, all data tasks for this time period have been processed, no new data tasks need to be processed, and the system enters the waiting phase for the next period's task processing.

[0133] If the result indicates that the current time period is not the last time period, the data warehouse service will continue processing data tasks for the next time period. This process will repeat until all data tasks within the preset time period have been processed. For example, the system will process data from 9:00 AM to 9:00 PM (assuming a period of 12 hours) until all data tasks for all time periods have been processed.

[0134] Example explanation:

[0135] Suppose there is a financial analysis system that needs to process the previous day's transaction data daily to generate reports and perform trend analysis. The system's preset time period is 24 hours per day, but it actually processes all the data from the previous day, including data from 24 time periods.

[0136] System Initialization: At 0:00 each day, the system begins a new daily task processing cycle. The Data Warehouse Service (DW) starts, ready to process the previous day's transaction data. The system begins processing transaction data from 0:00 to 24:00 of the previous day.

[0137] Data Extraction and Storage: The data warehouse service extracts all transaction data from the previous day from historical databases or transaction record storage systems. This data includes detailed information such as transaction time, transaction type, amount, and user ID. The system then processes the data according to time periods, performing tasks such as cleaning and transformation, and stores the processed data in the database.

[0138] The role of the task-driven mechanism: The task-driven mechanism receives task completion information from the data warehouse service, confirming that the data processing for the first time period has been completed (e.g., from 0:00 to 1:00 the previous day). Then, the system continues to process the data tasks for the next time period, such as from 1:00 to 2:00, until all data tasks for all time periods have been processed.

[0139] Continuous data processing: Once the data task for one time period is completed, the system immediately begins processing the data task for the next time period. This ensures the continuity of data processing, even if the current data being processed is from the previous day.

[0140] Processing data for the last time period: When the system processes data from 23:00 to 24:00 the previous day, it will process the data according to the normal process and send the task completion information to the task-driven mechanism.

[0141] End of task processing: The task-driven mechanism detects that this is the last time period of the previous day's data task, and the system ends the previous day's data processing task to prepare for the new task of the day.

[0142] Through the above steps, this embodiment can process all data tasks within the preset time period in an orderly and efficient manner, avoiding any omission of processing time periods, while reducing unnecessary waste of system resources.

[0143] Furthermore, a third embodiment of the data warehouse service optimization method of the present invention is proposed. In step S20 above, processing data tasks with a preset time period through the data warehouse service includes:

[0144] Step b1: The data warehouse service includes multiple data warehouse sub-services; determine the load status of each data warehouse sub-service.

[0145] Step b2: Divide the data tasks of the preset time period according to the load of each data warehouse sub-service to obtain the same number of sub-data tasks as the data warehouse sub-services;

[0146] Step b3: Assign each sub-data task to the corresponding data warehouse sub-service under the corresponding load condition, and each data warehouse sub-service processes the assigned sub-data task independently and in parallel.

[0147] In this embodiment, the data warehouse service is decomposed into multiple data warehouse sub-services (such as different computing nodes or processing modules), each of which can independently handle data tasks. The system monitors the current load of each sub-service, including its processing capacity and resource usage, in order to allocate data tasks appropriately.

[0148] The system divides the overall data task within a preset time period into multiple subtasks based on the load of each data warehouse subservice. The number of these subtasks equals the number of data warehouse subservices to ensure that each subservice can handle one subtask. During the division, the system considers the load of each subservice, ensuring that the task allocation balances the workload of the subservices and optimizes processing efficiency.

[0149] The system assigns each sub-data task to a sub-service that matches its load. Each sub-service processes its assigned task independently and in parallel, thereby improving the efficiency of the entire data processing process. The advantage of parallel processing is that it can make full use of system resources, speed up task processing, and reduce overall processing time.

[0150] A possible implementation method is as follows: The system first monitors the current load of each data warehouse sub-service, assessing the processing capacity, memory usage, CPU utilization, etc., of each sub-service. This information will be used to determine how to divide the data tasks.

[0151] Based on the load analysis results, the system divides data tasks within a preset time period into multiple sub-tasks. For example, if there are 24 hours of transaction data within a time period, and the system has four data warehouse sub-services, then the system may divide these 24 hours of data into four sub-tasks, each containing six hours of data. During the division, the system considers the load of each sub-service to ensure a balanced distribution of workload.

[0152] After the data is partitioned, the system assigns each subtask to its corresponding sub-service. For example, a less loaded sub-service might be assigned a larger data task, while a more loaded sub-service might be assigned a smaller task. Each sub-service, upon receiving its task, begins processing its assigned data task independently and in parallel. These sub-services are independent of each other and can process different data sub-tasks simultaneously.

[0153] After processing their assigned subtasks, each subservice stores the results data in the central database or its own storage system. The system periodically checks the processing status of each subservice to ensure that all subtasks are processed and stored correctly.

[0154] Example explanation:

[0155] In the data warehouse system of a large e-commerce platform:

[0156] The platform's data warehouse system has 5 data warehouse sub-services. The system monitors the load of each sub-service and found that 3 of the sub-services are currently under heavy load, while the other 2 sub-services are under light load.

[0157] Based on these load conditions, the system divides the previous day's transaction data task into 5 subtasks. Larger subtasks (such as time periods containing more transaction records) are assigned to lighter-loaded subservices, while smaller subtasks are assigned to heavier-loaded subservices.

[0158] Each sub-service processes its assigned data tasks independently and in parallel. One sub-service might be processing transaction data from 0:00 to 6:00, while another processes data from 6:00 to 12:00. All processing tasks run simultaneously in the background, and after completion, each sub-service stores the results in a unified database.

[0159] This embodiment optimizes the use of system resources and ensures rapid processing of data tasks through the above steps, making it particularly suitable for complex systems that need to process large-scale data.

[0160] Furthermore, a fourth embodiment of the data warehouse service optimization method of the present invention is proposed. In step S20 above, storing the processed data results for the current time period into the database includes:

[0161] Step c1: Divide the data results of the current time period into blocks, compress each data block using a compression algorithm to obtain compressed data blocks, and store the compressed data blocks in the database.

[0162] In this embodiment, after the data task for the current time period is completed, the system performs data segmentation. The purpose of segmentation is to break down a large dataset into smaller data blocks to facilitate subsequent compression, storage, and access operations. This segmentation can be based on data size, structure, or other specific rules.

[0163] After processing into blocks, the system compresses each data block using a specific compression algorithm. This algorithm can be common ones like ZIP, GZIP, or other specialized compression algorithms. The purpose of compression is to reduce storage space usage while improving the efficiency of data storage and transmission. Compressed data blocks are more compact, which helps optimize database storage performance.

[0164] After the data blocks are compressed, the system stores these compressed data blocks in the database. During storage, the database system performs appropriate storage management based on the size and format of the data blocks to ensure that the data can be effectively accessed and managed.

[0165] A specific implementation method could be: the system divides the data results into blocks based on the size of the data, the complexity of the data structure, or specific business logic. For example, the processed data may be divided into blocks by time, geographical location, or transaction type to ensure that each data block has a reasonable size for subsequent operations. If a data block is found to be too large during processing, the system can further subdivide it into smaller blocks to ensure that the size of each data block is within a reasonable range for subsequent compression processing.

[0166] Depending on the data type and storage requirements, the system selects an appropriate compression algorithm to compress data blocks. For example, GZIP might be used for text data, while a dedicated binary compression algorithm might be used for binary data. If the data volume is large, the system can compress multiple data blocks simultaneously, which can accelerate the compression process and improve the overall processing efficiency of the system.

[0167] The compressed data blocks are stored in the database. The system indexes each data block for easy retrieval and access. The database system may also perform additional management operations on these data blocks, such as backups and redundant storage. During storage, the system performs integrity checks on each compressed data block to ensure that the data is not damaged or lost during compression and storage.

[0168] Example explanation:

[0169] In a large e-commerce platform, tens of thousands of transaction records need to be processed and stored every hour:

[0170] The system divides the processed transaction data into blocks based on the hash value of the order number, with each block containing a certain number of transaction records. Assume the system is processing transaction data from 8:00 AM to 9:00 AM, totaling millions of transaction records.

[0171] After being divided into blocks, the system uses the GZIP algorithm to compress each data block, significantly reducing the size of the compressed data blocks. Original data that originally occupied 100MB may be compressed to 20MB, saving considerable storage space.

[0172] The system stores the compressed data blocks in a distributed database and creates an index for each data block to facilitate fast querying and access. During storage, the system also performs integrity checks on the data blocks to ensure that the data is not corrupted during storage.

[0173] This embodiment, through the above steps, not only improves the efficiency of data storage but also ensures the reliability and stability of large-scale data processing, making it particularly suitable for systems that need to process large amounts of historical data.

[0174] Furthermore, a fifth embodiment of the data warehouse service optimization method of the present invention is proposed. In step S30 above, before processing the data results of the current time period through the data application service, the method further includes:

[0175] Step d1: Generate a task event through the task-driven mechanism, and receive the task event through the data application service. The task event includes processing the data results for the current time period.

[0176] In this embodiment, after the data warehouse service completes its data processing task for the current time period, the task-driven mechanism automatically generates a task event. This task event contains the necessary information to instruct the data application service to perform subsequent data processing. The generation of the task event is a crucial step in the communication between the task-driven mechanism and the data application service, ensuring the continuity of the data processing flow.

[0177] The data application service is configured to listen for a task-driven mechanism and receive the task event immediately after it is generated. Upon receiving the task event, the data application service will initiate processing operations on the data results for the current time period.

[0178] The data application service executes specified processing tasks on the data results for the current time period based on the instructions in the task event. For example, this may involve analyzing data, generating reports, or further transferring the data to other systems or modules for more in-depth analysis.

[0179] Example explanation:

[0180] In a financial statement generation system:

[0181] Data warehouse service completion: Assume the data warehouse service has completed processing the transaction data from 9:00 to 10:00 the previous day and stored the results in the database. The task-driven mechanism detects task completion and automatically generates a task event. This task event contains instructions for the data application service to analyze the data results from this period and generate reports.

[0182] Data application service monitoring and startup: The data application service continuously monitors the task-driven mechanism. Upon receiving the task event, it immediately begins processing the data results from 9:00 to 10:00. The system may need to analyze transaction data, calculate the total transaction amount, generate risk analysis reports, etc.

[0183] Task completed: Based on the instructions in the task event, the data application service processed the transaction data and generated corresponding reports. These reports may be pushed to financial management or used for further decision support systems.

[0184] This embodiment ensures smooth interaction between data warehouse services and data application services through the above steps. The task-driven mechanism ensures that each step of the data processing flow can proceed in an orderly manner by generating and transmitting task events, thereby improving the automation and efficiency of the system.

[0185] Furthermore, a sixth embodiment of the data warehouse service optimization method of the present invention is proposed. In step d1 above, generating task events through the task-driven mechanism includes:

[0186] Step e1: Define the task identifier and task triggering conditions, wherein the task triggering conditions include the data warehouse service completing the data task for the current time period;

[0187] Step e2: When the task triggering condition is met, the task-driven mechanism automatically generates the task event. The task event includes the task identifier and task type, which are then monitored, received, and processed by the data application service.

[0188] In this embodiment, the task-driven mechanism requires defining an identifier (such as a task ID) for each task and the conditions for triggering that task. Task triggering conditions typically include the data warehouse service completing its data processing task for the current time period. This means that after the data warehouse service completes its processing, the system will generate a task event based on pre-defined conditions.

[0189] Once the triggering condition (such as the data warehouse service completing a task) is met, the task-driven mechanism will automatically generate a task event. This task event contains a task identifier (such as a task ID) and a task type (such as data analysis, report generation, etc.), which will guide the data application service on how to process the data results for the current time period.

[0190] After a task event is generated, the data application service receives the task event through the task-driven listening mechanism and performs corresponding data processing operations according to the instructions in the task event.

[0191] A specific implementation could be as follows: The system assigns a unique task identifier to each data processing task to ensure that the tasks can be accurately tracked and managed. For example, the task identifier can be defined based on factors such as timestamp, data type, and task priority. The task triggering conditions are set based on the status of the data warehouse service. For example, the task-driven mechanism monitors the status of the data warehouse service, and the triggering condition is met when it is detected that a data task for a certain time period has been completed and the data has been successfully stored in the database.

[0192] When the task triggering conditions are met, the task-driven mechanism automatically generates task events. These events contain a task identifier and a task type. The task type can be a specific data processing operation, such as data aggregation, trend analysis, or report generation. In addition to the task identifier and type, task events can also contain other necessary information, such as data storage location, processing parameters, or related dependent tasks.

[0193] The data application service is designed to continuously monitor task events driven by the task mechanism. Once a new task event is detected, it immediately receives and parses the event, initiating the corresponding data processing flow. Based on the instructions in the task event, the data application service processes the data for the current time period, performing the required task type, such as generating reports, performing analysis, or transmitting results to downstream systems.

[0194] Example explanation:

[0195] In an online advertising analytics system:

[0196] Define the task identifier and trigger condition: Assume the system needs to analyze the previous day's ad click data daily. The system defines a task identifier for each ad click data analysis task, such as "AD_ANALYSIS_20240807". The task trigger condition is set to "Completion of the previous day's ad click data processing task".

[0197] Task Event Generation: After the data warehouse service completes the processing of the previous day's ad click data, the task-driven mechanism automatically detects that the triggering conditions have been met. At this time, the task-driven mechanism generates a task event, the content of which includes the task identifier "AD_ANALYSIS_20240807" and the task type "Click Data Analysis".

[0198] Data application service processing: Upon detecting this task event, the data application service immediately initiates the data processing flow. The system may analyze ad click-through rates, conversion rates, user behavior patterns, etc., and generate detailed ad performance reports. These reports will be used to optimize ad placement strategies or provided to clients for marketing decisions.

[0199] This embodiment, through the above steps, can automatically process a large number of complex data tasks, ensuring that each data processing step can be triggered and executed in a timely manner, thereby improving the system's efficiency and responsiveness.

[0200] Furthermore, a seventh embodiment of the data warehouse service optimization method of the present invention is proposed, wherein, in step S10 above, after constructing the task-driven mechanism, the method further includes:

[0201] Step f1: The task-driven mechanism records detailed execution logs for each data task. When an abnormal state occurs during the execution of a data task, the task-driven mechanism generates an error log and handles the abnormal state according to a pre-defined strategy. The error log is available for viewing by the system administrator.

[0202] In this embodiment, the task-driven mechanism records detailed execution logs for each data task during its execution. These logs include the task start time, execution steps, resource usage, and execution status (e.g., success, failure). These logs help track the task execution process, provide system auditability, and offer a basis for future troubleshooting.

[0203] If an error or exception occurs during the execution of a data task, the task-driven mechanism will generate an error log. These exceptions may include task failure, insufficient resources, data inconsistency, etc. The task-driven mechanism will handle these abnormal states according to predefined strategies, such as retrying the task, skipping the task, or notifying the administrator. The error log records in detail the time of occurrence, the type of exception, and the handling process.

[0204] After error logs are generated, system administrators can view them to analyze and understand the reasons for task failures. Administrators can then take further action based on the error logs, such as manual intervention or adjusting system configurations, to ensure more stable and efficient task execution in the future.

[0205] A specific implementation method may be:

[0206] Detailed execution log records:

[0207] Task execution log content: Whenever a data task begins execution, the task-driven mechanism activates the logging function. The log content includes the task's unique identifier, the task's execution time, the execution steps, the system resources used (such as CPU, memory, etc.), and the task's execution result (success or failure).

[0208] Log storage and management: These logs are usually stored in a dedicated log management system, which can classify and index this data for subsequent querying and analysis.

[0209] Error log generation and processing:

[0210] Anomaly detection: If any abnormal state is detected during task execution (such as task timeout, insufficient resources, etc.), the task-driven mechanism will immediately generate an error log.

[0211] Execution of pre-defined strategies: When an error occurs, the task-driven mechanism will handle these exceptions according to the system's pre-defined strategies. For example, if the strategy is set to "retry 3 times", the system will attempt to re-execute the task up to three times; if the problem persists, the system may abort the task and notify the administrator.

[0212] Error log content: The error log records in detail the type of exception, the time of occurrence, the data tasks affected, the attempted handling methods, and the final handling result.

[0213] Administrators can view and analyze:

[0214] Error log visualization: Error logs are stored in the system and made available for system administrators to view. System administrators can use dedicated log analysis tools or dashboards to view detailed error information and analyze the root cause of the anomaly.

[0215] Follow-up measures: Administrators can decide whether to adjust the system configuration, optimize the task execution strategy, or take other measures to prevent similar problems from happening again in the future, based on the information in the logs.

[0216] Example explanation:

[0217] In a financial data processing system:

[0218] Log Recording and Management: Each data task in the system (such as daily transaction data processing) generates a detailed execution log. These logs record every step of the task from start to finish, resource usage, and task status. For example, the system might record that a transaction data processing task was executed on August 7, 2024, at 08:00, using 50% CPU and 70% memory, and that the task was successfully completed.

[0219] Error Log Generation: If the system discovers data file corruption while processing a batch of transaction data, the task-driven mechanism will generate an error log to record this anomaly. Assuming the system's pre-defined policy is "retry once and notify the administrator," the system will retry processing the task. If the retry fails, the system will generate a second error log entry and send a notification to the administrator.

[0220] Administrators can view logs: System administrators can use the log management system to view these error logs and understand the specific reasons for task failures. By analyzing the error logs, administrators may discover problems with the data source, and then contact the relevant departments to fix the data source issue and adjust system configurations to prevent similar failures in the future.

[0221] This embodiment, through the above steps, not only improves the robustness of the system, but also provides administrators with detailed diagnostic information so that they can respond quickly and take appropriate measures when problems occur.

[0222] The present invention also provides a data warehouse service optimization device.

[0223] Reference Figure 3 , Figure 3 This is a functional module diagram of a first embodiment of the data warehouse service optimization device of the present invention. The data warehouse service optimization device includes:

[0224] The task coordination and decoupling module splits the data warehouse service into a data warehouse service and a data application service, and constructs a task-driven mechanism to achieve decoupling between the data warehouse service and the data application service.

[0225] The data processing and storage module processes data tasks within a preset time period through a data warehouse service. The preset time period includes multiple time periods. After processing the data task for a certain time period, the module stores the data result of the current time period in the database and sends the data task completion information for the current time period to the task-driven mechanism.

[0226] The data application and execution module monitors the task-driven mechanism through the data application service. When it detects that the data warehouse service has completed the data task for the current time period, it processes the data results for the current time period through the data application service to obtain the data warehouse service processing result.

[0227] Furthermore, embodiments of the present invention also propose a computer storage medium.

[0228] The storage medium stores a data warehouse service optimization program, which, when executed by the processor, implements the steps of the data warehouse service optimization method as described above.

[0229] The specific implementation of the storage medium of the present invention is basically the same as the embodiments of the above-described data warehouse service optimization method, and will not be repeated here.

[0230] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many forms under the guidance of the present invention without departing from the spirit and scope of the claims. All equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are within the protection scope of the present invention.

[0231] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0232] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. If the function is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0233] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0234] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data warehouse service optimization method, characterized in that, Includes the following steps: The data warehouse service is split into a data warehouse service and a data application service, and a task-driven mechanism is constructed to decouple the data warehouse service and the data application service. Data tasks with a preset time period are processed through a data warehouse service. The preset time period includes multiple time periods. After the data task of a certain time period is processed, the data result of the current time period is stored in the database, and the data task completion information of the current time period is sent to the task-driven mechanism. By monitoring the task-driven mechanism through the data application service, when it is detected that the data warehouse service has completed the data task for the current time period, the data application service processes the data results for the current time period to obtain the data warehouse service processing result.

2. The data warehouse service optimization method as described in claim 1, characterized in that, After sending the data task completion information for the current time period to the task-driven mechanism, the method further includes: Determine whether the data task in the current time period is the data task of the last time period in the preset time cycle; If so, terminate the processing task of the data warehouse service; If not, the data warehouse service continues to process the data tasks for the next time period until all data tasks for the preset time period have been processed.

3. The data warehouse service optimization method as described in claim 1, characterized in that, Data warehouse services process data tasks within preset time periods, including: The data warehouse service includes multiple data warehouse sub-services, and the load status of each data warehouse sub-service is determined. The data tasks for the preset time period are divided according to the load of each data warehouse sub-service to obtain the same number of sub-data tasks as the data warehouse sub-services; Each sub-data task is assigned to a data warehouse sub-service with the corresponding load, and each data warehouse sub-service processes the assigned sub-data task independently and in parallel.

4. The data warehouse service optimization method as described in claim 1, characterized in that, The processed data for the current time period is stored in the database, including: The data results for the current time period are divided into blocks, and each data block is compressed using a compression algorithm to obtain compressed data blocks. The compressed data blocks are then stored in the database.

5. The data warehouse service optimization method as described in claim 1, characterized in that, Before processing the data results for the current time period through the data application service, the process also includes: Task events are generated through the task-driven mechanism and received through the data application service. The task events include processing data results for the current time period.

6. The data warehouse service optimization method as described in claim 5, characterized in that, The task-driven mechanism generates task events, including: Define task identifiers and task triggering conditions, wherein the task triggering conditions include the completion of data tasks for the current time period by the data warehouse service; When the task triggering condition is met, the task-driven mechanism automatically generates the task event, which includes the task identifier and task type, for the data application service to listen to, receive and process.

7. The data warehouse service optimization method as described in claim 1, characterized in that, After constructing the task-driven mechanism, it also includes: The task-driven mechanism records detailed execution logs for each data task. When an abnormal state occurs during the execution of a data task, the task-driven mechanism generates an error log and handles the abnormal state according to a pre-defined strategy. The error log is available for viewing by the system administrator.

8. A data warehouse service optimization device, characterized in that, The data warehouse service optimization device includes: The task coordination and decoupling module splits the data warehouse service into a data warehouse service and a data application service, and constructs a task-driven mechanism to achieve decoupling between the data warehouse service and the data application service. The data processing and storage module processes data tasks within a preset time period through a data warehouse service. The preset time period includes multiple time periods. After processing the data task for a certain time period, the module stores the data result of the current time period in the database and sends the data task completion information for the current time period to the task-driven mechanism. The data application and execution module monitors the task-driven mechanism through the data application service. When it detects that the data warehouse service has completed the data task for the current time period, it processes the data results for the current time period through the data application service to obtain the data warehouse service processing result.

9. A data warehouse service optimization device, characterized in that, The data warehouse service optimization device includes a memory, a processor, and a data warehouse service optimization program stored in the memory and executable on the processor. When the data warehouse service optimization program is executed by the processor, it implements the steps of the data warehouse service optimization method as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The storage medium stores a data warehouse service optimization program, which, when executed by a processor, implements the steps of the data warehouse service optimization method as described in any one of claims 1-7.

Images