Warehouse counting service optimization method and device, equipment and storage medium

Abstract

The invention relates to a data warehouse service optimization method, and the method comprises the steps: dividing a data warehouse service into two independent modules, namely, a data warehouse service (DW) and a data application service (DA), introducing a task driving mechanism, and achieving the decoupling between the DW and the DA. When the data warehouse service processes the data task in the preset time period, the processing result is stored in the database after the data task is processed, and task completion information is transmitted through a task driving mechanism. And the data application service processes the stored data result by monitoring a task driving mechanism after receiving task completion information, and generates final output of the data warehouse service. According to the method and the system, the flexibility and the efficiency of processing large-scale data tasks by the system are remarkably improved, and particularly, resource utilization and task scheduling can be effectively optimized in a scene of facing data adjustment and historical data backtracking.

Description

Technical Field

[0001] The present invention relates to the field of data processing and optimization technology, and in particular to a data warehouse service optimization method, device, equipment and storage medium. Background Art

[0002] Data warehouse systems play an important role in modern enterprise information management. According to the process of data inflow and outflow, the typical data warehouse architecture is usually divided into three layers: source data layer (ODS), data warehouse layer (DW) and data application layer (DA).

[0003] The source data layer (ODS) is the input end of the data warehouse, responsible for collecting and integrating data from multiple data sources. The main task of ODS is to provide temporary storage and preliminary cleaning of raw data. This layer is designed to ensure that the raw data received by the data warehouse is up-to-date and structured, which is convenient for subsequent processing and analysis.

[0004] The data warehouse layer (DW) is responsible for in-depth cleaning, processing and conversion of the data in the ODS layer. The data processed by the DW layer is high-quality data that has been verified and in a unified format. These data are the basis for enterprises to make important business decisions. The key function of the DW layer is to ensure the consistency and accuracy of the data and provide reliable basic data for the data application layer.

[0005] The data application layer (DA) is the output end of the data warehouse, which extracts, filters and applies data for specific business needs. The output of the DA layer is usually used to generate reports, data analysis or other business applications. The design goal of this layer is to refine the data according to different business needs to support a variety of application scenarios.

[0006] In offline data warehouse systems, although immediate response is not required, data processing tasks must be completed within a specified time to support subsequent task chains. For example, in the regulatory reporting scenario in an investment management system, the system must complete the processing of upstream data before a specific time point and generate corresponding reports for the regulatory system to obtain and submit.

[0007] When upstream data changes, especially when processing data for a quarter or longer, the system needs to backtrack and reprocess the data for the entire time period. In this case, the performance of the data warehouse system is often severely challenged, especially when processing large amounts of historical data. Traditional data warehouse systems usually use a batch processing model to extract data from the ODS, store it in the database after processing at the DW layer, and finally extract and process the data according to demand at the DA layer. When encountering data adjustments, the system needs to backtrack and process data day by day, repeating the same data processing process. This approach often has the problem of inefficiency when dealing with large-scale data processing. Summary of the invention

[0008] The main purpose of the present invention is to provide a data warehouse service optimization method, device, equipment and storage medium, aiming to solve the technical problem in the prior art that the tight coupling of data warehouse services and data application services leads to low efficiency in processing large-scale data tasks.

[0009] To achieve the above object, the present invention provides a data warehouse service optimization method, which comprises the following steps:

[0010] Split the data warehouse service into data warehouse service and data application service, and build a task-driven mechanism to achieve decoupling between the data warehouse service and the data application service;

[0011] Processing data tasks of a preset time period through a data warehouse service, wherein the preset time period includes multiple time periods. After processing the data tasks of a certain time period, the data results of the current time period obtained after processing are stored in the database, and the completion information of the data tasks of the current time period is sent to the task driving mechanism;

[0012] The task driving mechanism is monitored by the data application service. When it is detected that the data warehouse service has completed the data task of the current time period, the data application service processes the data result of the current time period to obtain the data warehouse service processing result.

[0013] Optionally, after sending the data task completion information of the current time period to the task driving mechanism, the method further includes:

[0014] Determine whether the data task of the current time period is the data task of the last time period in the preset time period;

[0015] If so, terminate the processing task of the data warehouse service;

[0016] If not, the data tasks of the next time period continue to be processed through the data warehouse service until all the data tasks of the preset time period are processed.

[0017] Optionally, the data warehouse service processes data tasks of a preset time period, including:

[0018] The data warehouse service includes a plurality of data warehouse sub-services, and the load condition of each data warehouse sub-service is determined;

[0019] Dividing the data tasks of the preset time period according to the load of each data warehouse sub-service to obtain sub-data tasks with the same number as the data warehouse sub-services;

[0020] Each sub-data task is assigned to a data warehouse sub-service corresponding to the load condition, and each data warehouse sub-service processes the assigned sub-data task independently and in parallel.

[0021] Optionally, the processed data results for the current time period are stored in a database, including:

[0022] The data results of the current time period are processed in blocks, each obtained data block is compressed by a compression algorithm to obtain a compressed data block, and the compressed data block is stored in the database.

[0023] Optionally, before processing the data result of the current time period by the data application service, the method further includes:

[0024] A task event is generated through the task driving mechanism, and the task event is received through the data application service. The task event includes processing the data result of the current time period.

[0025] Optionally, generating a task event through the task drive mechanism includes:

[0026] Defining a task identifier and a task triggering condition, wherein the task triggering condition includes that the data warehouse service completes the data task of the current time period;

[0027] When the task triggering condition is met, the task driving mechanism automatically generates the task event, and the task event includes the task identifier and the task type, for the data application service to monitor, receive and process.

[0028] Optionally, after building the task-driven mechanism, also include:

[0029] The task-driven mechanism records a detailed execution log of each data task. When an abnormal state occurs during the execution of a data task, the task-driven mechanism generates an error log and handles the abnormal state according to a predetermined strategy. The error log is available for review by a system administrator.

[0030] Furthermore, in order to achieve the above-mentioned object, the present invention also provides a data warehouse service optimization device, the data warehouse service optimization device comprising:

[0031] The task coordination and decoupling module splits the data warehouse service into data warehouse service and data application service, and builds a task-driven mechanism to achieve decoupling between the data warehouse service and the data application service.

[0032] The data processing and storage module processes data tasks of a preset time period through a data warehouse service. The preset time period includes multiple time periods. After processing the data tasks of a certain time period, the data results of the current time period obtained after processing are stored in the database, and the completion information of the data tasks of the current time period is sent to the task driving mechanism;

[0033] The data application and execution module monitors the task driving mechanism through the data application service. When it is detected that the data warehouse service has completed the data task of the current time period, the data application service processes the data result of the current time period to obtain the data warehouse service processing result.

[0034] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a data warehouse service optimization device, which includes a memory, a processor, and a data warehouse service optimization program stored in the memory and executable on the processor, and the data warehouse service optimization program, when executed by the processor, implements the steps of the data warehouse service optimization method described above.

[0035] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer storage medium, on which a data warehouse service optimization program is stored, and when the data warehouse service optimization program is executed by a processor, the steps of the data warehouse service optimization method as described above are implemented.

[0036] The present invention relates to a data warehouse service optimization method, which divides the data warehouse service into two independent modules, a data warehouse service (DW) and a data application service (DA), and introduces a task-driven mechanism to achieve decoupling between the two. When the data warehouse service processes data tasks within a preset time period, it will store the processing results in a database after completion, and transmit task completion information through a task-driven mechanism. The data application service monitors the task-driven mechanism, and after receiving the task completion information, it processes the stored data results to generate the final output of the data warehouse service. The present invention significantly improves the flexibility and efficiency of the system in processing large-scale data tasks, especially in the scenarios of data adjustment and historical data backtracking, and can effectively optimize resource utilization and task scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the structure of the device hardware operating environment involved in the data warehouse service optimization device embodiment of the present invention;

[0038] Figure 2 This is a flow chart of the first embodiment of the data warehouse service optimization method of the present invention;

[0039] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the data warehouse service optimization device of the present invention.

[0040] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0041] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0042] It should be noted that data warehouse systems play an important role in modern enterprise information management. According to the process of data inflow and outflow, the typical data warehouse architecture is usually divided into three layers: source data layer (ODS), data warehouse layer (DW) and data application layer (DA).

[0043] The source data layer (ODS) is the input end of the data warehouse, responsible for collecting and integrating data from multiple data sources. The main task of ODS is to provide temporary storage and preliminary cleaning of raw data. This layer is designed to ensure that the raw data received by the data warehouse is up-to-date and structured, which is convenient for subsequent processing and analysis.

[0044] The data warehouse layer (DW) is responsible for in-depth cleaning, processing and conversion of the data in the ODS layer. The data processed by the DW layer is high-quality data that has been verified and in a unified format. These data are the basis for enterprises to make important business decisions. The key function of the DW layer is to ensure the consistency and accuracy of the data and provide reliable basic data for the data application layer.

[0045] The data application layer (DA) is the output end of the data warehouse, which extracts, filters and applies data for specific business needs. The output of the DA layer is usually used to generate reports, data analysis or other business applications. The design goal of this layer is to refine the data according to different business needs to support a variety of application scenarios.

[0046] In offline data warehouse systems, although immediate response is not required, data processing tasks must be completed within a specified time to support subsequent task chains. For example, in the regulatory reporting scenario in an investment management system, the system must complete the processing of upstream data before a specific time point and generate corresponding reports for the regulatory system to obtain and submit.

[0047] When upstream data changes, especially when processing data for a quarter or longer, the system needs to backtrack and reprocess the data for the entire time period. In this case, the performance of the data warehouse system is often severely challenged, especially when processing large amounts of historical data. Traditional data warehouse systems usually use a batch processing model to extract data from the ODS, store it in the database after processing at the DW layer, and finally extract and process the data according to demand at the DA layer. When encountering data adjustments, the system needs to backtrack and process data day by day, repeating the same data processing process. This approach often has the problem of inefficiency when dealing with large-scale data processing.

[0048] Based on the above defects, the present invention provides a data warehouse service optimization device, referring to Figure 1 , Figure 1 This is a structural diagram of the device hardware operating environment involved in the data warehouse service optimization device implementation scheme of the present invention.

[0049] like Figure 1 As shown, the data warehouse service optimization device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0050] Those skilled in the art will understand that Figure 1 The hardware structure of the data warehouse service optimization device shown in the figure does not constitute a limitation of the data warehouse service optimization device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0051] like Figure 1As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a data warehouse service optimization program. The operating system is a program for managing and controlling data warehouse service optimization equipment and software resources, and supports the operation of the network communication module, the user interface module, the data warehouse service optimization program, and other programs or software; the network communication module is used to manage and control the network interface 1004; the user interface module is used to manage and control the user interface 1003.

[0052] exist Figure 1 In the hardware structure of the data warehouse service optimization device shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; the processor 1001 can call the data warehouse service optimization program stored in the memory 1005 and perform the following operations:

[0053] Split the data warehouse service into data warehouse service and data application service, and build a task-driven mechanism to achieve decoupling between the data warehouse service and the data application service;

[0054] Processing data tasks of a preset time period through a data warehouse service, wherein the preset time period includes multiple time periods. After processing the data tasks of a certain time period, the data results of the current time period obtained after processing are stored in the database, and the completion information of the data tasks of the current time period is sent to the task driving mechanism;

[0055] The task driving mechanism is monitored by the data application service. When it is detected that the data warehouse service has completed the data task of the current time period, the data application service processes the data result of the current time period to obtain the data warehouse service processing result.

[0056] Furthermore, after sending the data task completion information of the current time period to the task driving mechanism, it also includes:

[0057] Determine whether the data task of the current time period is the data task of the last time period in the preset time period;

[0058] If so, terminate the processing task of the data warehouse service;

[0059] If not, the data tasks of the next time period continue to be processed through the data warehouse service until all the data tasks of the preset time period are processed.

[0060] Furthermore, the data warehouse service processes data tasks of a preset time period, including:

[0061] The data warehouse service includes a plurality of data warehouse sub-services, and the load condition of each data warehouse sub-service is determined;

[0062] Dividing the data tasks of the preset time period according to the load of each data warehouse sub-service to obtain sub-data tasks with the same number as the data warehouse sub-services;

[0063] Each sub-data task is assigned to a data warehouse sub-service corresponding to the load condition, and each data warehouse sub-service processes the assigned sub-data task independently and in parallel.

[0064] Furthermore, the processed data results for the current time period are stored in a database, including:

[0065] The data results of the current time period are processed in blocks, each obtained data block is compressed by a compression algorithm to obtain a compressed data block, and the compressed data block is stored in the database.

[0066] Furthermore, before the data application service processes the data result of the current time period, the method further includes:

[0067] A task event is generated through the task driving mechanism, and the task event is received through the data application service. The task event includes processing the data result of the current time period.

[0068] Furthermore, generating a task event through the task drive mechanism includes:

[0069] Defining a task identifier and a task triggering condition, wherein the task triggering condition includes that the data warehouse service completes the data task of the current time period;

[0070] When the task triggering condition is met, the task driving mechanism automatically generates the task event, and the task event includes the task identifier and the task type, for the data application service to monitor, receive and process.

[0071] Furthermore, after building the task-driven mechanism, it also includes:

[0072] The task-driven mechanism records a detailed execution log of each data task. When an abnormal state occurs during the execution of a data task, the task-driven mechanism generates an error log and handles the abnormal state according to a predetermined strategy. The error log is available for review by a system administrator.

[0073] The specific implementation of the data warehouse service optimization device of the present invention is basically the same as the various embodiments of the data warehouse service optimization method described below, and will not be repeated here.

[0074] The present invention also provides a data warehouse service optimization method based on the above-mentioned data warehouse service optimization device.

[0075] Reference Figure 2 , Figure 2 This is a flow chart of the first embodiment of the data warehouse service optimization method of the present invention.

[0076] An embodiment of the present invention provides an embodiment of a data warehouse service optimization method. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown here.

[0077] In various embodiments of the data warehouse service optimization method, the execution entity is a data warehouse service optimization device.

[0078] The data warehouse service optimization method is applied to a data warehouse service optimization device, and the data warehouse service optimization method includes:

[0079] Step S10, splitting the data warehouse service into a data warehouse service and a data application service, and constructing a task-driven mechanism, wherein the task-driven mechanism is used to achieve decoupling between the data warehouse service and the data application service;

[0080] Data warehouse systems play an important role in modern enterprise information management. According to the process of data inflow and outflow, the typical data warehouse architecture is usually divided into three layers: source data layer (ODS), data warehouse layer (DW) and data application layer (DA).

[0081] The source data layer (ODS) is the input end of the data warehouse, responsible for collecting and integrating data from multiple data sources. The main task of ODS is to provide temporary storage and preliminary cleaning of raw data. This layer is designed to ensure that the raw data received by the data warehouse is up-to-date and structured, which is convenient for subsequent processing and analysis.

[0082] The data warehouse layer (DW) is responsible for in-depth cleaning, processing and conversion of the data in the ODS layer. The data processed by the DW layer is high-quality data that has been verified and in a unified format. These data are the basis for enterprises to make important business decisions. The key function of the DW layer is to ensure the consistency and accuracy of the data and provide reliable basic data for the data application layer.

[0083] The data application layer (DA) is the output end of the data warehouse, which extracts, filters and applies data for specific business needs. The output of the DA layer is usually used to generate reports, data analysis or other business applications. The design goal of this layer is to refine the data according to different business needs to support a variety of application scenarios.

[0084] In offline data warehouse systems, although immediate response is not required, data processing tasks must be completed within a specified time to support subsequent task chains. For example, in the regulatory reporting scenario in an investment management system, the system must complete the processing of upstream data before a specific time point and generate corresponding reports for the regulatory system to obtain and submit.

[0085] When upstream data changes, especially when processing data for a quarter or longer, the system needs to backtrack and reprocess the data for the entire time period. In this case, the performance of the data warehouse system is often severely challenged, especially when processing large amounts of historical data. Traditional data warehouse systems usually use a batch processing model to extract data from the ODS, store it in the database after processing at the DW layer, and finally extract and process the data according to demand at the DA layer. When encountering data adjustments, the system needs to backtrack and process data day by day, repeating the same data processing process. This approach often has the problem of inefficiency when dealing with large-scale data processing.

[0086] In this embodiment, the data warehouse service is divided into two independent modules, namely the data warehouse service (DW) and the data application service (DA). In order to achieve the decoupling between the two modules, a task-driven mechanism is constructed. The main function of this mechanism is to coordinate the task transfer between DW and DA so that they can work independently without directly relying on each other's status. The task-driven mechanism monitors and manages the status of the task and passes the processing results of the data warehouse service to the data application service, thereby making the entire system more flexible and scalable.

[0087] In the specific implementation, when the data warehouse service completes the data task within a certain time period, it will store the processing results in the database and generate task completion information through the task-driven mechanism. This information is passed to the data application service to notify it that it can start processing the data. The data application service listens to the task-driven mechanism, and after receiving the task completion information, it starts to further process the stored data, such as analyzing, summarizing, or generating reports.

[0088] This implementation ensures the independent operation of the data warehouse service and the data application service. Even if the data application service is temporarily unable to process data for some reason, the data warehouse service can still continue its data processing task and will not be affected by the status of the data application service. This decoupling method significantly improves the efficiency and stability of the system when processing large-scale data tasks.

[0089] Example description:

[0090] In a financial system, the data warehouse service is responsible for processing daily transaction data. Whenever the data warehouse service completes processing of a day's transaction data, it stores the results in the database and generates a task completion message. The task-driven mechanism captures this information and notifies the data application service. After receiving the notification, the data application service can start further analysis of the transaction data, such as generating a day-end report or conducting a risk assessment. If the transaction volume on a certain day is particularly large, the data application service may need more time to process the data, while the data warehouse service has already started processing the data tasks for the next day. This decoupling mechanism ensures the efficient operation of the system and does not affect the overall progress due to delays in a certain link.

[0091] Through the above steps, the system can more flexibly respond to large-scale data processing needs and effectively improve task scheduling and resource utilization efficiency.

[0092] Step S20, processing the data task of the preset time period through the data warehouse service, the preset time period includes multiple time periods, when the data task of a certain time period is processed, the data result of the current time period obtained after processing is stored in the database, and the completion information of the data task of the current time period is sent to the task driving mechanism;

[0093] In this embodiment, the data warehouse service is responsible for processing data tasks within a preset time period. The "preset time period" here refers to the time range associated with the data task itself. The time period is composed of multiple time periods, each of which can be a day, a few hours, or any predefined time range. Data tasks are processed based on these time periods, thereby ensuring that data processing can be carried out in an orderly manner according to the specified time nodes.

[0094] After completing the data task within a specific time period, the data warehouse service will store the data results processed within that time period in the database. This step ensures that the data of each time period is saved after processing for subsequent access or application.

[0095] After completing the data task for a certain period of time and storing the results, the data warehouse service generates task completion information for the current period and sends the information to the task driving mechanism. The task driving mechanism then starts the subsequent task processing process based on the information, such as notifying the data application service to start processing the stored data.

[0096] The specific implementation method may be: the data warehouse service will process the data tasks of each time period according to the preset time cycle. These tasks may involve operations such as data extraction, cleaning, conversion and loading. Whenever the data task processing of a time period is completed, the result data will be stored in the database. When the data task processing of a time period is completed, the data warehouse service generates a task completion signal and passes it to other services or modules through the task driving mechanism. The task driving mechanism notifies the data application service to process the data results of the time period based on the received signal to ensure the continuity and timeliness of the data processing process.

[0097] Example description:

[0098] Suppose there is a system for real-time monitoring and analysis of financial market data. The system needs to process a large amount of transaction data and generate analysis reports. In order to meet business needs, the system uses two parts: data warehouse service (DW) and data application service (DA), and decouples them through a task-driven mechanism.

[0099] In the financial market, transaction data changes dynamically, and millions of transaction records may be generated every hour. In order to effectively process and analyze this data, the system divides a day into 24 time periods, each of which is 1 hour. In each time period, the data warehouse service needs to process all transaction data in that time period.

[0100] The preset time period here is a whole day, which includes a 24-hour period, and each hour is an independent period.

[0101] At the beginning of each hour, the data warehouse service automatically extracts all transaction records for that time period from transaction data sources (such as exchange APIs, internal system logs, etc.). The data warehouse service then cleans the data, removes duplicate records, corrects erroneous data, and converts the data into a standard format for subsequent processing.

[0102] After cleaning and converting the data for a certain hour, the data warehouse service stores the processed data in the database. The database uses a columnar storage format to optimize query performance and storage efficiency.

[0103] When the data warehouse service completes the processing and storage of a certain hour's data task, it will generate a task completion message. This message includes the time stamp, processing status, data volume, etc. of that hour, indicating that the data of that time period is ready for subsequent operations.

[0104] After generating task completion information, the data warehouse service sends it to the task driving mechanism. After receiving the information, the task driving mechanism immediately passes it to the data application service and starts the corresponding task processing flow.

[0105] Through the above steps, it is possible to flexibly and efficiently process large amounts of data every day, while maintaining the independence and real-time response capabilities of data tasks in each time period. This not only improves the system's processing capabilities, but also greatly enhances the system's adaptability and stability in the face of sudden data growth or abnormal situations.

[0106] Step S30, monitoring the task driving mechanism through the data application service, when monitoring that the data warehouse service has completed the data task of the current time period, processing the data result of the current time period through the data application service to obtain the data warehouse service processing result.

[0107] In this embodiment, the data application service continuously monitors the task status of the data warehouse service by monitoring the task driving mechanism. When the task driving mechanism detects that the data warehouse service has completed the data processing task in the current time period, the data application service will receive a task completion notification or signal.

[0108] Once the data application service receives the task completion information, it will immediately perform subsequent processing on the processing results of the current time period. These processes may include data analysis, aggregation, report generation, etc., depending on business needs.

[0109] By processing the data results of the current time period, the data application service generates the final data warehouse service processing results. This result may be used to generate business reports, data analysis reports, or as input data for other systems to further support the enterprise's decision-making process.

[0110] A specific implementation method may be: the data application service is configured with a monitoring mechanism that can monitor the signals of the task-driven mechanism in real time. Whenever the data warehouse service completes a data task for a period of time and updates the task-driven mechanism, the data application service will receive a notification. After receiving the task completion information, the data application service starts the corresponding data processing flow. The processing content may include data aggregation analysis, anomaly detection, trend analysis, etc., and generate the required processing results according to the business logic. After the processing is completed, the data application service generates corresponding results, such as daily, weekly, monthly reports, or other forms of analysis reports. These results may be automatically pushed to decision makers, archived in the system, or used to drive other business processes.

[0111] Example description:

[0112] Assume that there is a data warehouse system for monitoring and analyzing sales data of an e-commerce platform. The system needs to process hourly transaction data in real time and generate reports and analysis results. The system includes a data warehouse service (DW) and a data application service (DA), and is coordinated through a task-driven mechanism.

[0113] Data processing tasks: At the end of each hour, the data warehouse service extracts all order data for that hour from the transaction database of the e-commerce platform. This data includes detailed information such as order number, product ID, purchase quantity, price, user ID, transaction time, etc.

[0114] Data cleaning and conversion: The extracted raw data may contain errors, duplications, or incomplete information. The data warehouse service will clean the data (such as deleting duplicate records, correcting erroneous data) and convert the data into a standardized format for subsequent processing and analysis.

[0115] Data storage: The cleaned and converted data is stored in the database to ensure that all transaction data is properly preserved and available for subsequent query and processing.

[0116] Generate task completion information: Whenever the data warehouse service completes a data processing task for a certain hour, it will generate a task completion information. This information includes the processing time period (such as "August 7, 2024 14:00-15:00"), the processing status (success or failure), the amount of data processed (such as the total number of transaction records), and other related metadata.

[0117] Task-driven mechanism transmission: Task completion information is transmitted to the data application service through the task-driven mechanism. The task-driven mechanism records the task completion information of each time period and ensures that the data application service can receive this information in time to start the next data processing operation.

[0118] Monitoring task-driven mechanism: The data application service is configured as a real-time monitoring task-driven mechanism. When the data processing task of a certain time period is completed, the data application service will immediately receive a notification and start processing the data results of that time period stored in the database.

[0119] Data analysis and report generation: Data application services conduct in-depth analysis of the received transaction data. For example, the system can calculate key performance indicators (KPIs) such as total sales, number of orders, and average order value during the time period. In addition, data application services may also conduct user behavior analysis to identify which products are most popular and which promotions are most effective.

[0120] Real-time decision support: The analysis results are used to generate real-time sales reports, which can be automatically pushed to management to help them make business decisions. For example, if a certain product is selling very well within that hour, management can decide whether to increase inventory or extend the promotion.

[0121] Dealing with peak periods: During shopping festivals or promotions, transaction volumes may increase significantly. The data warehouse service can dynamically allocate more computing resources to process this data, ensuring that data tasks in each time period can be completed on time. The task-driven mechanism ensures that the data application service can start analysis immediately after the data warehouse service completes the processing, keeping the entire system running efficiently.

[0122] Exception handling and recovery: If an exception occurs in data processing during a certain period of time (such as data loss or processing failure caused by system failure), the task-driven mechanism will record the exception information and start the error handling process. The system may automatically retry the processing task or notify the administrator for manual intervention. This can minimize the risk of data loss and ensure the continuity and reliability of data processing.

[0123] Real-time marketing and inventory adjustments: The analysis reports generated by data application services are not only provided to management, but can also directly drive other business systems. For example, the inventory management system can automatically adjust the inventory level based on the analysis results, while the marketing system can launch new promotional activities to respond to current market demand.

[0124] Long-term analysis and trend forecasting: Hourly data results can be summarized and used for long-term trend analysis to help companies predict future sales trends and develop more effective strategic plans.

[0125] The decoupling of data warehouse services and data application services is achieved through a task-driven mechanism, ensuring that the system can process and utilize large amounts of data efficiently and flexibly.

[0126] The present invention relates to a data warehouse service optimization method, which divides the data warehouse service into two independent modules, a data warehouse service (DW) and a data application service (DA), and introduces a task-driven mechanism to achieve decoupling between the two. When the data warehouse service processes data tasks within a preset time period, it will store the processing results in a database after completion, and transmit task completion information through a task-driven mechanism. The data application service monitors the task-driven mechanism, and after receiving the task completion information, it processes the stored data results to generate the final output of the data warehouse service. The present invention significantly improves the flexibility and efficiency of the system in processing large-scale data tasks, especially in the scenarios of data adjustment and historical data backtracking, and can effectively optimize resource utilization and task scheduling.

[0127] Furthermore, a second embodiment of the data warehouse service optimization method of the present invention is proposed. In the above step S20, after sending the data task completion information of the current time period to the task driving mechanism, it also includes:

[0128] Step a1, determining whether the data task of the current time period is the data task of the last time period in the preset time period;

[0129] Step a2: if yes, end the processing task of the data warehouse service;

[0130] Step a3: if not, continue to process the data tasks of the next time period through the data warehouse service until all the data tasks of the preset time period are processed.

[0131] In this embodiment, after the data warehouse service completes processing of a data task in a certain time period and sends task completion information, the system will check whether the currently processed time period is the last time period in the preset time period. For example, if the preset time period is one day (24 hours), the system will determine whether the currently processed time period is the last hour of the day. If yes, it means that all data tasks in the entire preset time period have been completed.

[0132] If the result of the judgment is that the current time period is the last time period, the data warehouse service will end all data processing tasks for the day or cycle. This means that the processing flow of the data warehouse service is completed, all data tasks for the time period have been processed, and no new data tasks need to be processed. The system enters the stage of waiting for the task processing of the next cycle.

[0133] If the result of the judgment is that the current time period is not the last time period, the data warehouse service will continue to process the data tasks of the next time period. This process will be repeated until all data tasks within the preset time period are processed. For example, the system will process from 9 am to 9 pm (assuming the cycle is 12 hours) until all data tasks of all time periods are processed.

[0134] Example description:

[0135] Suppose there is a financial analysis system that needs to process the previous day's transaction data every day to generate reports and perform trend analysis. The system's preset time period is 24 hours a day, but it actually processes all the data from the previous day, which includes data tasks for 24 time periods.

[0136] System initialization: At 0:00 every day, the system starts a new day's task processing cycle. The data warehouse service (DW) starts to process the transaction data of the previous day. The system starts processing the transaction data from 0:00 to 24:00 of the previous day.

[0137] Data extraction and storage: The data warehouse service extracts all transaction data from the previous day from the historical database or transaction record storage system. This data includes detailed information such as transaction time, transaction type, amount, user ID, etc. The system then processes the data according to time periods, such as cleaning and conversion, and stores the processed data in the database.

[0138] The role of the task-driven mechanism: The task-driven mechanism receives the task completion information of the data warehouse service and confirms that the data processing of the first time period has been completed (for example, 0:00 to 1:00 the previous day). Then, the system continues to process the data tasks of the next time period, such as 1:00 to 2:00, until the data tasks of all time periods are processed.

[0139] Continuous data processing: When the data task of a time period is processed, the system immediately processes the data task of the next time period. This ensures the consistency of data processing, even if the current data being processed is the data of the previous day.

[0140] Processing the data task of the last time period: When the system processes the data from 23:00 to 24:00 of the previous day, it processes the data according to the normal process and sends the task completion information to the task driving mechanism.

[0141] End task processing: The task-driven mechanism detects that this is the last time period of the previous day's data task. The system ends the previous day's data processing task and prepares for the new task of the day.

[0142] Through the above steps, this embodiment can process all data tasks within the preset time period in an orderly and efficient manner, avoid missing any time period, and reduce unnecessary waste of system resources.

[0143] Furthermore, a third embodiment of the data warehouse service optimization method of the present invention is proposed. In the above step S20, the data task of the preset time period is processed by the data warehouse service, including:

[0144] Step b1, the data warehouse service includes multiple data warehouse sub-services, and the load of each data warehouse sub-service is determined;

[0145] Step b2, dividing the data tasks of the preset time period according to the load of each data warehouse sub-service to obtain sub-data tasks with the same number as the data warehouse sub-services;

[0146] Step b3: assign each sub-data task to a data warehouse sub-service corresponding to the load condition, and each data warehouse sub-service processes the assigned sub-data task independently and in parallel.

[0147] In this embodiment, the data warehouse service is decomposed into multiple data warehouse sub-services (such as different computing nodes or processing modules), and each sub-service can independently process data tasks. The system monitors the current load of each sub-service, including their processing capacity, resource usage, etc., in order to reasonably allocate data tasks.

[0148] The system divides the overall data task within the entire preset time period into multiple subtasks based on the load of each data warehouse subservice. The number of these subtasks is equal to the number of data warehouse subservices to ensure that each subservice can handle a subtask. When dividing, the system will consider the load of each subservice, so that the task division can balance the workload of the subservices and optimize the processing efficiency.

[0149] The system assigns each sub-data task to a sub-service that matches its load. Each sub-service processes independently and in parallel according to the assigned task, thereby improving the efficiency of the entire data processing process. The advantage of parallel processing is that it can make full use of system resources, speed up task processing, and reduce the overall processing time.

[0150] A specific implementation method may be: the system first monitors the current load of each data warehouse sub-service and evaluates the processing capacity, memory usage, CPU occupancy rate, etc. of each sub-service. This information will be used to decide how to divide the data tasks.

[0151] Based on the load analysis results, the system divides the data tasks within the preset time period into multiple subtasks. For example, if there is 24 hours of transaction data in a time period and the system has 4 data warehouse subservices, the system may divide the 24 hours of data into 4 subtasks, each task containing 6 hours of data. When dividing, the system will consider the load of each subservice to ensure a balanced distribution of workload.

[0152] After the division is completed, the system assigns each subtask to the corresponding subservice. For example, a subservice with a lighter load may be assigned a larger data task, while a subservice with a heavier load may be assigned a smaller task. After receiving the task, each subservice starts processing the assigned data task independently and in parallel. These subservices are independent of each other and can process different data subtasks at the same time.

[0153] After processing the assigned subtasks, each subservice stores the result data in the central database or its own storage system. The system periodically checks the processing status of each subservice to ensure that all subtasks are processed and stored correctly.

[0154] Example description:

[0155] In the data warehouse system of a large e-commerce platform:

[0156] The platform's data warehouse system has five data warehouse sub-services. The system monitors the load of each sub-service and finds that three of the sub-services are currently heavily loaded, while the other two are lightly loaded.

[0157] Based on these load conditions, the system divides the previous day's transaction data task into 5 subtasks. Larger subtasks (such as time periods containing more transaction records) are assigned to subservices with lighter loads, while smaller subtasks are assigned to subservices with heavier loads.

[0158] Each sub-service processes its assigned data tasks independently and in parallel. One sub-service may be processing transaction data from 0:00 to 6:00, while another is processing data from 6:00 to 12:00. All processing tasks are performed simultaneously in the background, and after processing, each sub-service stores the results in a unified database.

[0159] Through the above steps, this embodiment not only optimizes the use of system resources, but also ensures the rapid processing of data tasks, and is particularly suitable for complex systems that need to process large-scale data.

[0160] Furthermore, a fourth embodiment of the data warehouse service optimization method of the present invention is proposed. In the above step S20, the data result of the current time period obtained after processing is stored in the database, including:

[0161] Step c1, dividing the data results of the current time period into blocks, compressing each obtained data block by a compression algorithm to obtain a compressed data block, and storing the compressed data block in the database.

[0162] In this embodiment, after the data task processing of the current time period is completed, the system processes the data results in blocks. The purpose of block processing is to split a large data set into smaller data blocks to facilitate subsequent compression, storage and access operations. This block processing can be performed based on the size, structure or other specific rules of the data.

[0163] After block processing, the system compresses each data block using a specific compression algorithm. The compression algorithm can be common such as ZIP, GZIP or other special compression algorithms. The purpose of compression is to reduce storage space usage while improving the efficiency of data storage and transmission. The compressed data block will be more compact, which helps to optimize the storage performance of the database.

[0164] After the data blocks are compressed, the system will store them in the database. When storing, the database system will perform corresponding storage management according to the size and format of the data blocks to ensure that the data can be accessed and managed effectively.

[0165] The specific implementation method may be: the system divides the data results into blocks according to the size of the data volume, the complexity of the data structure, or specific business logic. For example, the processed data may be divided into blocks by time, geographic location, or transaction type to ensure that each data block has a reasonable size to facilitate subsequent operations. If a data block is found to be too large during processing, the system can further subdivide it into smaller blocks to ensure that the size of each data block is within a reasonable range to facilitate subsequent compression processing.

[0166] According to the data type and storage requirements, the system selects an appropriate compression algorithm to compress the data block. For example, GZIP may be used for text data, and a special binary compression algorithm may be used for binary data. If the amount of data is large, the system can compress multiple data blocks at the same time, which can speed up the compression process and improve the overall processing efficiency of the system.

[0167] The compressed data blocks are stored in the database. The system will index each data block to facilitate subsequent retrieval and access. The database system may also perform additional management operations on these data blocks, such as data block backup and redundant storage. During the storage process, the system will also perform integrity checks on each compressed data block to ensure that the data is not damaged or lost during the compression and storage process.

[0168] Example description:

[0169] In a large e-commerce platform, thousands of transaction records need to be processed and stored every hour:

[0170] The system divides the processed transaction data into blocks according to the hash value of the order number, and each data block contains a certain number of transaction records. Assume that the system processes the transaction data from 8:00 to 9:00 in the morning, which contains millions of transaction records in total.

[0171] After block division, the system uses the GZIP algorithm to compress each data block, and the size of the compressed data block is greatly reduced. The original data that originally occupied 100MB may be compressed to 20MB, which significantly saves storage space.

[0172] The system stores the compressed data blocks in a distributed database and creates an index for each data block for quick query and access. During the storage process, the system also performs integrity checks on the data blocks to ensure that the data is not damaged during the storage process.

[0173] Through the above steps, this embodiment not only improves the efficiency of data storage, but also ensures the reliability and stability of large-scale data processing, and is particularly suitable for systems that need to process a large amount of historical data.

[0174] Furthermore, a fifth embodiment of the data warehouse service optimization method of the present invention is proposed. In the above step S30, before the data result of the current time period is processed by the data application service, the method further includes:

[0175] Step d1, generating a task event through the task drive mechanism, receiving the task event through the data application service, wherein the task event includes processing the data result of the current time period.

[0176] In this embodiment, after the data warehouse service completes the data processing task in the current time period, the task driving mechanism will automatically generate a task event. This task event contains the necessary information to instruct the data application service to perform subsequent data processing. The generation of task events is a key step in the communication between the task driving mechanism and the data application service to ensure the continuity of the data processing process.

[0177] The data application service is configured to listen to the task-driven mechanism and receive the task event immediately after it is generated. After receiving the task event, the data application service will start processing the data results of the current time period.

[0178] The data application service performs the specified processing tasks on the data results of the current time period according to the instructions in the task event. For example, this may involve analyzing data, generating reports, or further transmitting data to other systems or modules for more in-depth analysis.

[0179] Example description:

[0180] In a financial reporting system:

[0181] The data warehouse service completes the processing: Assume that the data warehouse service completes the processing of the transaction data from 9:00 to 10:00 the previous day and stores the results in the database. After the task drive mechanism detects that the task is completed, a task event is automatically generated. The task event contains a requirement to instruct the data application service to analyze and generate reports for the data results during this period.

[0182] Data application service monitoring and startup: The data application service continuously monitors the task-driven mechanism, and immediately starts processing the data results from 9:00 to 10:00 after receiving the task event. The system may need to analyze transaction data, calculate the total transaction amount, generate risk analysis reports, etc.

[0183] Task completion: The data application service completes the processing of transaction data according to the instructions in the task event and generates corresponding reports. These reports may be pushed to the financial management or used for further decision support systems.

[0184] Through the above steps, this embodiment ensures smooth interaction between the data warehouse service and the data application service. The task-driven mechanism ensures that each step of the data processing flow can be carried out in an orderly manner by generating and transmitting task events, thereby improving the automation and efficiency of the system.

[0185] Furthermore, a sixth embodiment of the data warehouse service optimization method of the present invention is proposed. In the above step d1, generating a task event through the task driving mechanism includes:

[0186] Step e1, defining a task identifier and a task triggering condition, wherein the task triggering condition includes that the data warehouse service completes the data task in the current time period;

[0187] Step e2, when the task triggering condition is met, the task driving mechanism automatically generates the task event, and the task event includes the task identifier and task type, for the data application service to monitor, receive and process.

[0188] In this embodiment, in the task-driven mechanism, it is necessary to first define the identifier of each task (such as task ID) and the condition for triggering the task. The task triggering condition usually includes that the data warehouse service completes the data processing task in the current time period, which means that the system will generate a task event according to the pre-set conditions after the data warehouse service completes the processing.

[0189] Once the trigger condition (such as the data warehouse service completing the task) is met, the task-driven mechanism will automatically generate a task event. This task event contains the task identifier (such as the task ID) and the task type (such as data analysis, report generation, etc.), which will guide the data application service on how to process the data results of the current time period.

[0190] After a task event is generated, the data application service receives the task event by monitoring the task drive mechanism, and performs corresponding processing operations on the data according to the instructions in the task event.

[0191] A specific implementation method may be: the system assigns a unique task identifier to each data processing task to ensure that the task can be accurately tracked and managed. For example, the task identifier can be defined based on factors such as timestamp, data type, and task priority. The setting of the task trigger condition is based on the status of the data warehouse service. For example, the task-driven mechanism monitors the status of the data warehouse service, and when it is detected that the data task of a certain time period has been completed and the data has been successfully stored in the database, the trigger condition is met.

[0192] When the task triggering conditions are met, the task-driven mechanism will automatically generate task events. These events contain the task identifier and task type. The task type can be a specific data processing operation, such as data aggregation, trend analysis, report generation, etc. Task events not only contain task identifiers and types, but also other necessary information, such as data storage location, processing parameters, or related dependent tasks.

[0193] The data application service is designed to continuously listen for task events from the task-driven mechanism. Once a new task event is detected, it immediately receives and parses the event and starts the corresponding data processing flow. According to the instructions in the task event, the data application service processes the data in the current time period and performs the required task type, such as generating reports, performing analysis, or transmitting the results to downstream systems.

[0194] Example description:

[0195] In an online advertising analysis system:

[0196] Define task identifiers and trigger conditions: Assume that the system needs to analyze the previous day's ad click data every day. The system defines a task identifier for each ad click data analysis task, such as "AD_ANALYSIS_20240807". The task trigger condition is set to "the processing task of the previous day's ad click data is completed."

[0197] Task event generation: When the data warehouse service completes the processing of the previous day's ad click data, the task-driven mechanism automatically detects that the trigger condition is met. At this time, the task-driven mechanism generates a task event, and the event content includes the task identifier "AD_ANALYSIS_20240807" and the task type "Click Data Analysis".

[0198] Processing of data application service: After the data application service monitors this task event, it immediately starts the data processing process. The system may analyze the click-through rate, conversion rate, user behavior pattern, etc. of the advertisement, and generate a detailed advertising effect report. These reports will be used to optimize advertising delivery strategies or provided to customers for marketing decisions.

[0199] Through the above steps, this embodiment can automatically process a large number of complex data tasks, ensuring that each data processing step can be triggered and executed in a timely manner, thereby improving the efficiency and responsiveness of the system.

[0200] Furthermore, a seventh embodiment of the data warehouse service optimization method of the present invention is proposed. In the above step S10, after the task driving mechanism is constructed, the following is further included:

[0201] Step f1, the task-driven mechanism records the detailed execution log of each data task. When an abnormal state occurs during the execution of a data task, the task-driven mechanism generates an error log and handles the abnormal state according to a predetermined strategy. The error log is for the system administrator to view.

[0202] In this embodiment, the task-driven mechanism will record the execution log of each data task in detail during its execution. These logs include the task start time, task execution steps, resource usage, execution status (such as success, failure), etc. These logs help track the execution process of the task, provide auditability of the system, and provide a basis for future troubleshooting.

[0203] If an error or exception occurs during the execution of a data task, the task-driven mechanism will generate an error log. These exceptions may include task failure, insufficient resources, inconsistent data, etc. The task-driven mechanism will handle these abnormal conditions according to pre-defined strategies, such as retrying the task, skipping the task, or notifying the administrator. The error log records in detail the time when the exception occurred, the type of exception, and the handling process.

[0204] After the error logs are generated, system administrators can view these logs to analyze and understand the reasons for the task failure. Administrators can take further measures based on the error logs, such as manual intervention or adjusting system configuration to ensure that the task is more stable and efficient in future executions.

[0205] Specific implementations may be:

[0206] Detailed execution log records:

[0207] Task execution log content: Whenever a data task starts to execute, the task-driven mechanism will start the logging function. The log content includes the unique identifier of the task, the execution time of the task, the execution steps, the system resources used (such as CPU, memory, etc.), and the execution result of the task (success or failure).

[0208] Log storage and management: These logs are usually stored in a dedicated log management system, which can classify and index the data for subsequent query and analysis.

[0209] Generation and processing of error logs:

[0210] Anomaly detection: During the task execution process, if any abnormal state is detected (such as task timeout, insufficient resources, etc.), the task-driven mechanism will immediately generate an error log.

[0211] Execution of preset policies: After an error occurs, the task-driven mechanism will handle these exceptions according to the system's preset policies. For example, if the policy is set to "retry 3 times", the system will try to re-execute the task up to three times; if the problem persists, the system may terminate the task and notify the administrator.

[0212] Error log content: The error log records in detail the type of exception, the time of occurrence, the data tasks affected, the attempted processing methods, and the final processing results.

[0213] Administrators view and analyze:

[0214] Visualization of error logs: Error logs are stored in the system and made available to system administrators. System administrators can view detailed information about errors and analyze the root causes of anomalies through specialized log analysis tools or dashboards.

[0215] Subsequent processing measures: Based on the information in the log, the administrator can decide whether to adjust the system configuration, optimize the task execution strategy, or take other measures to prevent similar problems from happening again in the future.

[0216] Example description:

[0217] In a financial data processing system:

[0218] Logging and management: In the system, each data task (such as daily transaction data processing) generates a detailed execution log. These logs record each step of the task from start to finish, resource usage, and task status. For example, the system may record that the execution time of a transaction data processing task is 08:00 on August 7, 2024, using 50% of the CPU and 70% of the memory, and the task is successfully completed.

[0219] Error log generation: If the system finds that the data file is damaged when processing a batch of transaction data, the task-driven mechanism will generate an error log to record this exception. Assuming that the system's predetermined policy is "retry once and notify the administrator", the system will retry to process the task. If the retry fails, the system will generate a second error log and send a notification to the administrator.

[0220] Administrators view logs: System administrators can view these error logs through the log management system to understand the specific reasons for task failure. By analyzing the error logs, administrators may find that there is a problem with the data source, so they contact the relevant department to fix the data source problem and adjust the system configuration to avoid similar failures in the future.

[0221] Through the above steps, this embodiment not only improves the robustness of the system, but also provides detailed diagnostic information for administrators so that they can respond quickly and take appropriate measures when problems occur.

[0222] The present invention also provides a data warehouse service optimization device.

[0223] Reference Figure 3 , Figure 3 This is a functional module diagram of the first embodiment of the data warehouse service optimization device of the present invention. The data warehouse service optimization device includes:

[0224] The task coordination and decoupling module splits the data warehouse service into data warehouse service and data application service, and builds a task-driven mechanism to achieve decoupling between the data warehouse service and the data application service.

[0225] The data processing and storage module processes data tasks of a preset time period through a data warehouse service. The preset time period includes multiple time periods. After processing the data tasks of a certain time period, the data results of the current time period obtained after processing are stored in the database, and the completion information of the data tasks of the current time period is sent to the task driving mechanism;

[0226] The data application and execution module monitors the task driving mechanism through the data application service. When it is detected that the data warehouse service has completed the data task of the current time period, the data application service processes the data result of the current time period to obtain the data warehouse service processing result.

[0227] In addition, an embodiment of the present invention further provides a computer storage medium.

[0228] The storage medium stores a data warehouse service optimization program, which implements the steps of the data warehouse service optimization method described above when executed by the processor.

[0229] The specific implementation method of the storage medium of the present invention is basically the same as the embodiments of the above-mentioned data warehouse service optimization method, and will not be repeated here.

[0230] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims. All equivalent structures or equivalent process changes made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are protected by the present invention.

[0231] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0232] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a disk or an optical disk.

[0233] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0234] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data warehouse service optimization method, It is characterized in that The following steps are involved: Split the data warehouse service into data warehouse service and data application service, and build a task-driven mechanism to achieve decoupling between the data warehouse service and the data application service; Processing data tasks of a preset time period through a data warehouse service, wherein the preset time period includes multiple time periods. After processing the data tasks of a certain time period, the data results of the current time period obtained after processing are stored in the database, and the completion information of the data tasks of the current time period is sent to the task driving mechanism; The task driving mechanism is monitored by the data application service. When it is detected that the data warehouse service has completed the data task of the current time period, the data application service processes the data result of the current time period to obtain the data warehouse service processing result.

2. The data warehouse service optimization method according to claim 1, It is characterized in that After sending the data task completion information of the current time period to the task driving mechanism, the method further includes: Determine whether the data task of the current time period is the data task of the last time period in the preset time period; If so, terminate the processing task of the data warehouse service; If not, the data tasks of the next time period continue to be processed through the data warehouse service until all the data tasks of the preset time period are processed.

3. The data warehouse service optimization method according to claim 1, It is characterized in that The data warehouse service processes data tasks in a preset time period, including: The data warehouse service includes a plurality of data warehouse sub-services, and the load condition of each data warehouse sub-service is determined; Dividing the data tasks of the preset time period according to the load of each data warehouse sub-service to obtain sub-data tasks with the same number as the data warehouse sub-services; Each sub-data task is assigned to a data warehouse sub-service corresponding to the load condition, and each data warehouse sub-service processes the assigned sub-data task independently and in parallel.

4. The data warehouse service optimization method according to claim 1, It is characterized in that The processed data results for the current time period are stored in the database, including: The data results of the current time period are processed in blocks, each obtained data block is compressed by a compression algorithm to obtain a compressed data block, and the compressed data block is stored in the database.

5. The data warehouse service optimization method according to claim 1, It is characterized in that Before processing the data result of the current time period by the data application service, the method further includes: A task event is generated through the task driving mechanism, and the task event is received through the data application service. The task event includes processing the data result of the current time period.

6. The data warehouse service optimization method according to claim 5, It is characterized in that Generating a task event through the task driving mechanism includes: Defining a task identifier and a task triggering condition, wherein the task triggering condition includes that the data warehouse service completes the data task of the current time period; When the task triggering condition is met, the task driving mechanism automatically generates the task event, and the task event includes the task identifier and the task type, for the data application service to monitor, receive and process.

7. The data warehouse service optimization method as claimed in claim 1, It is characterized in that After building the task-driven mechanism, it also includes: The task-driven mechanism records a detailed execution log of each data task. When an abnormal state occurs during the execution of a data task, the task-driven mechanism generates an error log and handles the abnormal state according to a predetermined strategy. The error log is available for review by a system administrator.

8. A data warehouse service optimization device, It is characterized in that The data warehouse service optimization device comprises: The task coordination and decoupling module splits the data warehouse service into data warehouse service and data application service, and builds a task-driven mechanism to achieve decoupling between the data warehouse service and the data application service. The data processing and storage module processes data tasks of a preset time period through a data warehouse service. The preset time period includes multiple time periods. After processing the data tasks of a certain time period, the data results of the current time period obtained after processing are stored in the database, and the completion information of the data tasks of the current time period is sent to the task driving mechanism; The data application and execution module monitors the task driving mechanism through the data application service. When it is detected that the data warehouse service has completed the data task of the current time period, the data application service processes the data result of the current time period to obtain the data warehouse service processing result.

9. A data warehouse service optimization device, It is characterized in that The data warehouse service optimization device includes a memory, a processor, and a data warehouse service optimization program stored in the memory and executable on the processor. When the data warehouse service optimization program is executed by the processor, the steps of the data warehouse service optimization method according to any one of claims 1 to 7 are implemented.

10. A computer storage medium, It is characterized in that The storage medium stores a data warehouse service optimization program, which, when executed by a processor, implements the steps of the data warehouse service optimization method according to any one of claims 1 to 7.

Images