Data warehouse-based data processing method and device, equipment and storage medium

Abstract

The application provides a data processing method and device based on a data warehouse, electronic equipment and a storage medium; the data warehouse comprises a source data layer, a data detail layer, a data service layer and a data application layer; the method comprises: obtaining to-be-processed data through the source data layer; determining at least two data dimensions corresponding to the to-be-processed data and at least two generation time periods corresponding to the generation time of the to-be-processed data through the data detail layer; storing the to-be-processed data into at least two storage shards in combination with each data dimension and the generation time of the to-be-processed data through the data service layer; wherein the storage shards and the generation time periods are in one-to-one correspondence; when a query request is received, data is queried in the at least two storage shards through the data application layer to obtain a query result. Through the application, the accuracy and efficiency of data storage and query in the data warehouse can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device and storage medium based on a data warehouse. Background Technology

[0002] A data warehouse is a subject-oriented, integrated, stable, and time-varying collection of data used to support decision-making in operations management.

[0003] In related technologies, data processing in data warehouses typically involves directly storing data in a storage space and retrieving the corresponding data from that storage space. This results in low efficiency and accuracy in data storage and retrieval. Summary of the Invention

[0004] This application provides a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product based on a data warehouse, which can effectively improve the accuracy and efficiency of data storage and retrieval in the data warehouse.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a data processing method based on a data warehouse, including:

[0007] The data warehouse includes a source data layer, a data detail layer, a data service layer, and a data application layer;

[0008] The data to be processed is obtained through the source data layer;

[0009] Through the data detail layer, at least two data dimensions corresponding to the data to be processed, and at least two generation time periods corresponding to the generation time of the data to be processed are determined;

[0010] Through the data service layer, the data to be processed is stored in at least two storage shards, taking into account each data dimension and the generation time of the data to be processed.

[0011] The storage shards and the generation time periods are in a one-to-one correspondence.

[0012] When a query request is received, the data application layer performs a data query on at least two storage shards to obtain the query result.

[0013] This application provides a data processing apparatus based on a data warehouse, including:

[0014] The acquisition module is used to acquire the data to be processed through the source data layer;

[0015] The determination module is used to determine, through the data detail layer, at least two data dimensions corresponding to the data to be processed, and at least two generation time periods corresponding to the generation time of the data to be processed;

[0016] The storage module is used to store the data to be processed into at least two storage shards by combining the data dimensions and the generation time of the data to be processed through the data service layer; wherein the storage shards correspond one-to-one with the generation time period;

[0017] The query module is used to perform data queries on at least two storage shards through the data application layer when a query request is received, and obtain the query results.

[0018] In some embodiments, the determining module is further configured to perform dimensional aggregation on the data to be processed through the data detail layer to obtain at least two data dimensions corresponding to the data to be processed, and dimensional aggregation data corresponding to each data dimension; the data processing device based on the data warehouse further includes: a partitioning module, configured to perform the following processing on the dimensional aggregation data of each data dimension through the data detail layer: obtaining the generation time of the dimensional aggregation data, and partitioning the dimensional aggregation data of the data dimension according to the at least two generation time periods based on the generation time of the dimensional aggregation data, to obtain time period data corresponding to each generation time period; the storage module is further configured to find at least two storage shards corresponding to the data dimension through the data service layer, and store each time period data into the corresponding storage shard.

[0019] In some embodiments, the dimensional aggregation data includes multiple data units; the partitioning module is further configured to obtain the timestamp of each data unit in the dimensional aggregation data; compare the time point indicated by the timestamp of each data unit with each generation time period to obtain the generation time period to which each data unit belongs; for each generation time period, the data units belonging to the same generation time period are determined as the time period data corresponding to the generation time period.

[0020] In some embodiments, the query module is further configured to, upon receiving the query request, parse the query request through the data application layer to obtain at least one request dimension carried by the query request and at least one request time period corresponding to each request dimension; determine multiple target storage shards corresponding to each request dimension; for each request dimension, query target time period data in the corresponding multiple target storage shards, and use the target time period data as the query result; wherein, the target time period data is the time period data that matches each request time period corresponding to the request dimension.

[0021] In some embodiments, the above-described data warehouse-based data processing apparatus further includes: an anomaly detection module, configured to perform the following processing on each of the time period data through the data detail layer: obtaining a data fluctuation threshold for the generation time period corresponding to the time period data; determining the data fluctuation ratio of the time period data within the generation time period; comparing the data fluctuation threshold and the data fluctuation ratio to obtain a comparison result, wherein the comparison result indicates whether the data fluctuation ratio is greater than the data fluctuation threshold; and determining the type of the time period data based on the comparison result, wherein the type is used to indicate whether the time period data is abnormal fluctuation data.

[0022] In some embodiments, the above-mentioned anomaly judgment module is further configured to determine the type of the time period data as a data type with fluctuation anomaly when the comparison result indicates that the data fluctuation ratio is greater than the data fluctuation threshold; the above-mentioned data processing device based on data warehouse further includes: an output module, configured to output alarm prompt information, the alarm prompt information being used to indicate that the type of the time period data is the data type with fluctuation anomaly.

[0023] In some embodiments, the above-described data warehouse-based data processing apparatus further includes: a partitioning module, configured to obtain the generation time of the data to be processed through the data detail layer, and partition the data to be processed according to the at least two generation time periods based on the generation time of the data to be processed, to obtain time period data corresponding to each generation time period; the determining module is further configured to perform the following processing on the time period data of each generation time period through the data detail layer: perform dimension aggregation on the time period data to obtain at least two data dimensions corresponding to the time period data, and dimension aggregation data corresponding to each data dimension; the storage module is further configured to find the storage shard corresponding to each generation time period through the data service layer, wherein the storage shard corresponds one-to-one with the generation time period; and store the dimension aggregation data of the at least two data dimensions corresponding to each generation time period to the storage location corresponding to each data dimension in the corresponding storage shard.

[0024] In some embodiments, the above-mentioned segmentation module is further configured to obtain the timestamp of each data unit in the data to be processed; compare the time point indicated by the timestamp of each data unit with each generation time period to obtain the generation time period to which each data unit belongs; and perform the following processing for each generation time period: determine each data unit belonging to the same generation time period as the time period data corresponding to the generation time period.

[0025] In some embodiments, the query module is further configured to, upon receiving the query request, parse the query request through the data application layer to obtain at least one request time period carried by the query request and at least one request dimension corresponding to each request time period; determine the target storage shard corresponding to each request time period; query the target dimension aggregate data in the target storage shard corresponding to each request time period, and use the target dimension aggregate data as the query result; wherein the target dimension aggregate data is the dimension aggregate data stored at the storage location corresponding to each request dimension.

[0026] In some embodiments, the above-described data warehouse-based data processing apparatus further includes: a cleaning module, configured to obtain the data to be processed from the source data layer and configuration data from the dimension table layer through the data detail layer, wherein the configuration data is used to configure parameters of the data warehouse; to merge the data to be processed and the configuration data through the data detail layer to obtain merged data; and to identify abnormal data in the merged data through the data detail layer and delete the abnormal data in the merged data to obtain data to be processed after deleting the abnormal data.

[0027] In some embodiments, the determining module is further configured to determine, through the data detail layer, at least two data dimensions corresponding to the data to be processed after the deletion of the abnormal data, and at least two generation time periods corresponding to the generation time of the data to be processed after the deletion of the abnormal data.

[0028] This application provides an electronic device, including:

[0029] Memory, used to store executable instructions;

[0030] The processor, when executing executable instructions stored in the memory, implements the data processing method based on a data warehouse provided in the embodiments of this application.

[0031] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the data processing method based on a data warehouse provided in this application.

[0032] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the data processing method based on a data warehouse as described in this application.

[0033] The embodiments of this application have the following beneficial effects:

[0034] Data to be processed is obtained from the source data layer. Then, through the data service layer, combined with data dimensions and generation time, the data to be processed is stored in at least two storage shards. When a query request is received, the data application layer performs a data query on at least two storage shards to obtain the query results. In this way, since there is a one-to-one correspondence between storage shards and generation time periods, storing data to be processed in at least two storage shards allows data from different generation time periods to be stored in different storage shards. During data queries, the required data can be retrieved from different storage shards. Therefore, by utilizing storage shards, the accuracy of data storage and querying in the data warehouse can be effectively improved. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the data processing system architecture based on a data warehouse provided in an embodiment of this application;

[0036] Figure 2 This is a schematic diagram of the structure of a data processing device based on a data warehouse provided in an embodiment of this application;

[0037] Figures 3A to 3CThis is a flowchart illustrating the data processing method based on a data warehouse provided in an embodiment of this application;

[0038] Figures 4A to 4I This is a schematic diagram illustrating the principle of the data processing method based on a data warehouse provided in the embodiments of this application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0041] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0043] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0044] 1) Data Warehouse: A structured data environment that serves as a data source for decision support systems and online analytical applications. Data warehouses study and solve problems related to retrieving information from databases. The characteristics of a data warehouse include subject-oriented nature, integration, stability, and time-varying characteristics.

[0045] 2) Source Data Layer: The data in the source data layer remains unchanged. It directly uses the data structure and data of the external system and is not exposed to the outside world. It is a temporary storage layer and a temporary storage area for interface data, preparing for the next step of data processing.

[0046] 3) Data Warehouse Detail (DWD) Layer: This layer stores detailed data, which is the finest-grained fact data. It generally maintains the same data granularity as the source data layer and provides a certain level of data quality assurance. Furthermore, to improve usability, this layer employs dimension degradation techniques, reducing dimensions to the fact table and minimizing the association between the fact table and dimension tables.

[0047] 4) Data Warehouse Middle (DWM): This layer stores intermediate data, including intermediate tables created for data statistics. This data typically aggregates data across multiple dimensions and usually originates from the data detail layer. Wide tables store aggregated data specific to a particular business domain. Business layer data often originates from this layer. Wide tables are used to centrally aggregate and store all business-related data for easy access by the business layer. This layer typically draws from both the data service layer and the data detail layer.

[0048] 5) Data application layer: The data source that the front-end application reads directly; the data calculated and generated according to the needs of reports and special analysis.

[0049] During the implementation of the embodiments of this application, the applicant discovered the following problems with the related technology:

[0050] In related technologies, data processing in data warehouses typically involves directly storing data in a storage space and retrieving the corresponding data from that storage space. This results in low efficiency and accuracy in data storage and retrieval.

[0051] This application provides a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product based on a data warehouse, which can effectively improve the accuracy and efficiency of data storage and retrieval in the data warehouse. The following describes exemplary applications of the electronic device provided in this application. The device provided in this application can be implemented as various types of user terminals such as laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or as a server. The following describes exemplary applications when the device is implemented as a server.

[0052] See Figure 1 , Figure 1This is a schematic diagram of the architecture of a data processing system 100 based on a data warehouse provided in this application embodiment. In order to realize the application scenario of data processing based on a data warehouse, the terminal (terminals 400-1 to 400-N are shown as examples) connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0053] Terminals 400-1 to 400-N are used by users to access client 410, which is displayed on graphical interface 410-1 (graphical interface 410-1 is shown as an example). Terminals 400-1 to 400-N and server 200 are interconnected via wired or wireless network.

[0054] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal 400 may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, in-vehicle terminal, etc., but is not limited to these. Terminals and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0055] In some embodiments, server 200 obtains data to be processed from terminals 400-1 to 400-N through the source data layer, and stores the data to be processed in at least two storage shards by combining at least two data dimensions corresponding to the data to be processed and at least two generation time periods corresponding to the generation time of the data to be processed. When server 200 receives a query request sent by at least one of terminals 400-1 to 400-N, it performs data query in at least two storage shards through the data application layer, and sends the query result obtained to the terminal that sent the query request from terminals 400 to 400-N.

[0056] In other embodiments, terminals 400-1 to 400-N obtain the data to be processed through the source data layer, and combine it with at least two data dimensions corresponding to the data to be processed and at least two generation time periods corresponding to the generation time of the data to be processed, and store the data to be processed in at least two storage shards. At least one of terminals 400-1 to 400-N sends a query request to the other terminals 400-1 to 400-N. The terminal that receives the query request performs a data query in at least two storage shards through the data application layer, and sends the query result obtained to the terminal that initiated the query request.

[0057] In other embodiments, the embodiments of this application can be implemented with the aid of cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0058] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, allowing for on-demand use with flexibility and convenience. Cloud computing technology will become a crucial support. The backend services of cloud computing systems require substantial computing and storage resources.

[0059] See Figure 2 , Figure 2 This is a schematic diagram of the structure of a server 200 for a data processing method based on a data warehouse provided in an embodiment of this application. Figure 2 The server 200 shown includes at least one processor 210, memory 250, and at least one network interface 220. The various components of server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 240.

[0060] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0061] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices physically located away from the processor 210.

[0062] The memory 250 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 250 described in this application embodiment is intended to include any suitable type of memory.

[0063] In some embodiments, memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0064] Operating system 251 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks.

[0065] The network communication module 252 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 220, such as Bluetooth, WiFi, and Universal Serial Bus (USB).

[0066] In some embodiments, the data processing apparatus based on a data warehouse provided in this application can be implemented in software. Figure 2 A data warehouse-based data processing apparatus 255, stored in memory 250, is shown. This apparatus can be software in the form of programs and plug-ins, and includes the following software modules: an acquisition module 2551, a determination module 2552, a storage module 2553, and a query module 2554. These modules are logically linked and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0067] In other embodiments, the data processing apparatus based on a data warehouse provided in this application can be implemented in hardware. As an example, the data processing apparatus based on a data warehouse provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method based on a data warehouse provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0068] The data processing method based on a data warehouse provided in this application will be described in conjunction with exemplary applications and implementations of the server or terminal provided in the embodiments of this application.

[0069] See Figure 3A , Figure 3A This is a flowchart illustrating the data processing method based on a data warehouse provided in an embodiment of this application, which will be combined with... Figure 3A Steps 101 to 104 are described below. The execution subject of steps 101 to 104 can be a server or a terminal. The following description will take the execution subject as a server as an example.

[0070] In some embodiments, a data warehouse includes a source data layer, a data detail layer, a data service layer, and a data application layer. The source data layer is used to acquire data to be processed; the data detail layer is used to determine at least two data dimensions corresponding to the data to be processed, and at least two generation time periods corresponding to the generation time of the data to be processed; the data service layer is used to combine each data dimension and the generation time of the data to be processed to store the data to be processed in at least two storage shards; and the data application layer is used to perform data queries in at least two storage shards to obtain query results.

[0071] In step 101, the data to be processed is obtained through the source data layer.

[0072] In some embodiments, the data warehouse further includes dimension tables, which are used to store configuration data for the data warehouse. This configuration data includes Supply Side Platform (SSP) configuration data (SSP is a platform serving the media, i.e., a platform that provides traffic); Demand Side Platform (DSP) configuration data; and Data Management Platform (DMP), etc.

[0073] In some embodiments, the data to be processed may be data streams from different time periods obtained by the source data layer from the Software Development Kit (SDK).

[0074] In some embodiments, after step 101 above, abnormal data can also be determined in the following ways: obtain the data to be processed from the source data layer and obtain the configuration data from the dimension table layer through the data detail layer, wherein the configuration data is used to configure the parameters of the data warehouse; merge the data to be processed and the configuration data through the data detail layer to obtain merged data; determine the abnormal data in the merged data through the data detail layer and delete the abnormal data in the merged data to obtain the data to be processed after the abnormal data is deleted.

[0075] In some embodiments, configuration data is used to configure parameters of the data warehouse. For example, configuration data can configure parameters of the supplier platform, parameters of the demand side platform, and parameters of the data management platform.

[0076] In some embodiments, the fused data includes data to be processed and configuration data. The data to be processed and configuration data are fused through a data detail layer to obtain fused data. Then, the data from the source data layer and the dimension table layer are summarized through the data detail layer to obtain fused data.

[0077] In some embodiments, abnormal data in the fused data can be determined based on specific business configurations. For example, the field type of normal data in the fused data is integer, meaning that abnormal data in the fused data can be string data. The field value range of normal data in the fused data is 0 to 100, while the field value range of abnormal data is greater than 100. Thus, by defining relevant parameters for the fused data under different business scenarios, abnormal data in the fused data can be identified and deleted, resulting in data to be processed after abnormal data removal. This ensures that there is no abnormal data in the data processed by the subsequent data detail layer, effectively improving the accuracy of data processing.

[0078] In step 102, at least two data dimensions corresponding to the data to be processed and at least two generation time periods corresponding to the generation time of the data to be processed are determined through the data detail layer.

[0079] In some embodiments, the at least two data dimensions corresponding to the data to be processed may be data media dimension, exposure dimension, click volume dimension, etc., and the at least two data dimensions corresponding to the data to be processed may be specifically determined based on different business scenarios.

[0080] In some embodiments, step 102 above can be implemented in the following way: through the data detail layer, determine at least two data dimensions corresponding to the data to be processed after the abnormal data is deleted, and at least two generation time periods corresponding to the generation time of the data to be processed after the abnormal data is deleted.

[0081] In some embodiments, the data detail layer can first perform dimensional aggregation on the data to be processed to obtain dimensional aggregated data, and then divide the dimensional aggregated data into time periods. Alternatively, the data detail layer can first divide the data to be processed into time periods to obtain time period data, and then perform dimensional aggregation on the time period data. The following will explain these two processing methods respectively.

[0082] In some embodiments, step 102 above can be implemented as follows: through the data detail layer, the data to be processed is aggregated in dimensions to obtain at least two data dimensions corresponding to the data to be processed, and the dimension aggregated data corresponding to each data dimension.

[0083] As an example, through the data detail layer, the data to be processed is aggregated in dimensions to obtain the exposure dimension, click dimension, and the aggregated data of the exposure dimension and click dimension respectively.

[0084] In some embodiments, before step 103, the dimensional aggregated data can be divided according to the generation time period in the following way: through the data detail layer, the following processing is performed on the dimensional aggregated data of each data dimension respectively: obtain the generation time of the dimensional aggregated data, and based on the generation time of the dimensional aggregated data, divide the dimensional aggregated data of the data dimension according to at least two generation time periods to obtain the time period data corresponding to each generation time period.

[0085] As an example, through the data detail layer, the following processing is performed on the dimension aggregation data of the exposure dimension: obtain the generation time of the dimension aggregation data of the exposure dimension, and based on the generation time of the dimension aggregation data of the exposure dimension, divide the dimension aggregation data of the exposure dimension into generation time periods (10:00 to 12:00 and 12:00 to 15:00) to obtain the time period data corresponding to the generation time period (10:00 to 12:00) and generation time period (12:00 to 15:00).

[0086] In some embodiments, the dimensional aggregated data includes multiple data units; the above-mentioned division of the dimensional aggregated data of the data dimension into at least two generation time periods to obtain the time period data corresponding to each generation time period can be achieved in the following way: obtaining the timestamp of each data unit in the dimensional aggregated data; comparing the time point indicated by the timestamp of each data unit with each generation time period to obtain the generation time period to which each data unit belongs; for each generation time period, the data units belonging to the same generation time period are determined as the time period data corresponding to the generation time period.

[0087] In some embodiments, the dimensional aggregated data includes multiple data units, and the timestamps of different data units indicate different points in time.

[0088] In some embodiments, by comparing the timestamps of each data unit in the acquired dimensional aggregation data with each generation time period, the generation time period to which each data unit belongs is obtained, and then the data units belonging to the same generation time period are determined as the time period data corresponding to that generation time period.

[0089] As an example, the timestamps of each data unit in the dimensional aggregation data are obtained. The time points indicated by the timestamps of each data unit (e.g., the time point indicated by the timestamp of data unit 1 is 12:31, the time point indicated by the timestamp of data unit 2 is 14:32, the time point indicated by the timestamp of data unit 3 is 16:08, and the time point indicated by the timestamp of data unit 4 is 12:51) are compared with each generation time period (e.g., each hour in 24-hour format is a generation time period) to obtain the generation time period to which each data unit belongs. For example, the generation time period to which data unit 1 belongs is the time period from 12:00 to 13:00, the generation time period to which data unit 2 belongs is the time period from 14:00 to 15:00, the generation time period to which data unit 3 belongs is the time period from 16:00 to 17:00, and the generation time period to which data unit 4 belongs is the time period from 12:00 to 13:00. For each generation time period, the data units belonging to the same generation time period are determined as the time period data corresponding to the generation time period. For example, the time period data corresponding to the generation time period from 12:00 to 13:00 are data unit 1 and data unit 4.

[0090] In this way, the data detail layer can first perform dimensional aggregation on the data to be processed to obtain dimensional aggregated data. Then, the dimensional aggregated data can be divided into generation time periods to obtain the time period data corresponding to the generation time periods. Thus, the data to be processed can be divided and aggregated according to dimensions and generation time periods to obtain time period data for storage in storage shards. This ensures a one-to-one correspondence between storage shards and generation time periods, and enables the data to be processed in different generation time periods to be stored in different storage shards, thereby effectively improving the accuracy and efficiency of data storage in the data warehouse.

[0091] In some embodiments, after the data dimension aggregator data is divided according to the generation time period corresponding to the generation time of the dimension aggregator data to obtain the time period data corresponding to each generation time period, it can be determined whether the time period data is abnormal fluctuation data in the following way: For each time period data, the following processing is performed through the data detail layer: obtain the data fluctuation threshold of the generation time period corresponding to the time period data; determine the data fluctuation ratio of the time period data within the generation time period; compare the data fluctuation threshold and the data fluctuation ratio to obtain the comparison result, wherein the comparison result indicates whether the data fluctuation ratio is greater than the data fluctuation threshold; based on the comparison result, determine the type of the time period data, wherein the type is used to indicate whether the time period data is abnormal fluctuation data.

[0092] In some embodiments, the aforementioned data fluctuation threshold is used to characterize the critical value of data fluctuation amplitude. When the data fluctuation ratio is less than the data fluctuation threshold, it indicates that the data fluctuation is relatively small; when the data fluctuation ratio is greater than the data fluctuation threshold, it indicates that the data fluctuation is relatively large. The data fluctuation threshold can be specifically set according to different business scenarios.

[0093] In some embodiments, the data fluctuation ratio of the time period data within the generated time period can be determined as follows: determine the value of the data unit corresponding to the start time point and the value of the data unit corresponding to the end time point in the time period data; determine the difference between the value of the data unit corresponding to the start time point and the value of the data unit corresponding to the end time point as the first difference; determine the difference between the start time point and the end time point as the second difference; and determine the ratio of the first difference and the second difference as the data fluctuation ratio of the time period data within the generated time period.

[0094] In some embodiments, the above-mentioned determination of the type of time period data based on the comparison results can be achieved in the following way: when the comparison results indicate that the proportion of data fluctuation is greater than the data fluctuation threshold, the type of time period data is determined to be a data type of fluctuation anomaly.

[0095] In this way, by comparing the data fluctuation threshold with the data fluctuation ratio within the generated time period, a comparison result is obtained. Based on the comparison result, it is determined whether the time period data has abnormal fluctuations, and thus outputs abnormal fluctuation data, effectively indicating relevant abnormal fluctuation data, providing data support for subsequent data processing, and effectively improving the accuracy of data processing.

[0096] In some embodiments, when the data type of the time period is a data type with abnormal fluctuations, an alarm message can be output to indicate that the data type of the time period is a data type with abnormal fluctuations.

[0097] As an example, alarm messages can be output via telephone, email, or other means to inform operators that the data for a certain period of time is of the type of fluctuating abnormal data.

[0098] In some other embodiments, before step 103, the data to be processed can be divided according to the generation time period in the following way: the generation time of the data to be processed is obtained through the data detail layer, and based on the generation time of the data to be processed, the data to be processed is divided into at least two generation time periods to obtain the time period data corresponding to each generation time period.

[0099] As an example, the generation time of the data to be processed is obtained through the data detail layer. Based on the generation time of the data to be processed, the data to be processed is divided into generation time periods (10:00 to 12:00 and 12:00 to 15:00) to obtain the time period data corresponding to the generation time period (10:00 to 12:00) and the generation time period (12:00 to 15:00).

[0100] In other embodiments, the data to be processed includes multiple data units. The above-mentioned division of the data to be processed into at least two generation time periods to obtain the time period data corresponding to each generation time period can be achieved as follows: obtain the timestamp of each data unit in the data to be processed; compare the time point indicated by the timestamp of each data unit with each generation time period to obtain the generation time period to which each data unit belongs; perform the following processing for each generation time period: determine the data units belonging to the same generation time period as the time period data corresponding to that generation time period.

[0101] In other embodiments, the data to be processed includes multiple data units, and the timestamps of the different data units indicate different time points.

[0102] In other embodiments, the generation time period to which each data unit belongs is obtained by comparing the timestamp of each data unit in the acquired data to be processed with each generation time period, and then the data units belonging to the same generation time period are determined as the time period data corresponding to that generation time period.

[0103] As an example, the timestamps of each data unit in the data to be processed are obtained. The time point indicated by the timestamp of each data unit (for example, the time point indicated by the timestamp of data unit 1 is 12:31, the time point indicated by the timestamp of data unit 2 is 14:32, the time point indicated by the timestamp of data unit 3 is 16:08, and the time point indicated by the timestamp of data unit 4 is 12:51) is compared with each generation time period (for example, each hour in 24-hour format is a generation time period) to obtain the generation time period to which each data unit belongs. For example, the generation time period to which data unit 1 belongs is the time period from 12:00 to 13:00, the generation time period to which data unit 2 belongs is the time period from 14:00 to 15:00, the generation time period to which data unit 3 belongs is the time period from 16:00 to 17:00, and the generation time period to which data unit 4 belongs is the time period from 12:00 to 13:00. For each generation time period, the data units belonging to the same generation time period are determined as the time period data corresponding to the generation time period. For example, the time period data corresponding to the generation time period from 12:00 to 13:00 are data unit 1 and data unit 4.

[0104] In other embodiments, after dividing the data to be processed according to the generation time corresponding to the generation time of the data to be processed, and obtaining the time period data corresponding to each generation time period, it can be determined whether the time period data is abnormally fluctuating data in the following way: For each time period data, the following processing is performed through the data detail layer: obtain the data fluctuation threshold of the generation time period corresponding to the time period data; determine the data fluctuation ratio of the time period data within the generation time period; compare the data fluctuation threshold and the data fluctuation ratio to obtain a comparison result, wherein the comparison result indicates whether the data fluctuation ratio is greater than the data fluctuation threshold; based on the comparison result, determine the type of the time period data, wherein the type is used to indicate whether the time period data is abnormally fluctuating data.

[0105] In other embodiments, step 102 above can be implemented as follows: through the data detail layer, the following processing is performed on the time period data of each generated time period: the time period data is aggregated in dimensions to obtain at least two data dimensions corresponding to the time period data, and the dimension aggregated data corresponding to each data dimension.

[0106] As an example, through the data detail layer, the following processing is performed on the time period data for each generated time period: dimensional aggregation is performed on the time period data to obtain the exposure volume dimension, click volume dimension, and the dimensional aggregation data corresponding to the exposure volume dimension and click volume dimension, respectively.

[0107] In this way, the data detail layer can first divide the data to be processed into time periods to obtain time period data, and then perform dimensional aggregation on the time period data to obtain dimensional aggregated data. Thus, the data to be processed is divided and aggregated according to dimensions and time periods to obtain time period data for storage in storage shards. This ensures a one-to-one correspondence between storage shards and time periods, and enables the data to be processed in different time periods to be stored in different storage shards, thereby effectively improving the accuracy and efficiency of data storage in the data warehouse.

[0108] In step 103, the data service layer, in conjunction with the generation time of each data dimension and the data to be processed, stores the data to be processed in at least two storage shards.

[0109] In some embodiments, there is a one-to-one correspondence between storage fragments and generation time periods.

[0110] In some embodiments, when the data to be processed is first aggregated by dimension and then divided by the generation time period, the above step 103 can be implemented in the following way: through the data service layer, find at least two storage shards corresponding to the data dimension, and store the data of each time period into the corresponding storage shards respectively.

[0111] As an example, the data service layer finds at least two storage shards corresponding to the click volume data dimension, and stores the click volume data for each time period into the storage shard corresponding to each generated time period. For example, the click volume data for each time period (generated time period is 11:00 to 12:00) is stored into the storage shard corresponding to the generated time period of 11:00 to 12:00.

[0112] In other embodiments, when the data to be processed is first divided according to the generation time period and then aggregated by dimension, the above step 103 can be implemented in the following way: through the data service layer, find at least two storage shards corresponding to each generation time period, and the storage shards correspond one-to-one with the generation time period; store the dimension aggregated data of at least two data dimensions corresponding to each generation time period in the corresponding storage shards at the storage locations corresponding to each data dimension.

[0113] As an example, through the data service layer, the two storage shards corresponding to each generation time period (generation time period 11:00-12:00, generation time period 13:00-14:00) are located, and the storage shards correspond one-to-one with the generation time periods. The data of at least two data dimensions corresponding to the generation time period 11:00-12:00 is aggregated and stored in the storage location corresponding to each data dimension in the storage shard corresponding to the generation time period 11:00-12:00. The data of at least two data dimensions corresponding to the generation time period 13:00-14:00 is aggregated and stored in the storage location corresponding to each data dimension in the storage shard corresponding to the generation time period 13:00-14:00.

[0114] In step 104, when a query request is received, the data application layer performs a data query in at least two storage shards to obtain the query result.

[0115] In some embodiments, the query request carries at least one request dimension and at least one request time period corresponding to each request dimension.

[0116] As an example, a query request may carry a click request dimension and an impression request dimension, as well as at least one request time period corresponding to the click request dimension (e.g., 11:00 to 12:00) and at least one request time period corresponding to the impression request dimension (e.g., 13:00 to 14:00).

[0117] In some embodiments, see Figure 3B , Figure 3B This is a flowchart illustrating the data processing method based on a data warehouse provided in an embodiment of this application. Figure 3B Step 104 shown can be achieved by performing steps 1041 to 1043.

[0118] In step 1041, when a query request is received, the query request is parsed by the data application layer to obtain at least one request dimension carried by the query request and at least one request time period corresponding to each request dimension.

[0119] As an example, when a query request is received, the data application layer parses the query request to obtain the click volume request dimension and the exposure volume request dimension carried by the query request, as well as at least one request time period (e.g., 11:00 to 12:00) corresponding to the click volume request dimension and at least one request time period (e.g., 13:00 to 14:00) corresponding to the exposure volume request dimension.

[0120] In step 1042, multiple target storage shards corresponding to each request dimension are determined.

[0121] As an example, the target storage shards corresponding to the click request dimension and the impression request dimension are determined respectively.

[0122] As an example, the target storage shard is the storage shard that stores data for the corresponding request dimension. It's understandable that, since there's a one-to-one correspondence between the generation time period and the storage shard, different storage shards store data for different generation time periods, while the same storage shard stores data for the same generation time period. The same storage shard may store data for different data dimensions. If the first data dimension only generates data during the generation time period of 11:00 AM to 12:00 PM, and no data is generated during other generation time periods, then the data for this data dimension will only be stored in the storage shard corresponding to the generation time period of 11:00 AM to 12:00 PM. Therefore, when the request dimension is the first data dimension, the storage shard corresponding to the data for the first data dimension can be the storage shard corresponding to the generation time period of 11:00 AM to 12:00 PM. That is, the target storage shard in this case is the storage shard corresponding to the generation time period of 11:00 AM to 12:00 PM.

[0123] In step 1043, for each request dimension, the target time period data is queried in the corresponding multiple target storage shards, and the target time period data is used as the query result.

[0124] In some embodiments, the target time period data is time period data that matches each request time period corresponding to the request dimension.

[0125] In other embodiments, see Figure 3C , Figure 3C This is a flowchart illustrating the data processing method based on a data warehouse provided in an embodiment of this application. Figure 3C Step 104 shown can be achieved by performing steps 1044 to 1046.

[0126] In step 1044, when a query request is received, the query request is parsed by the data application layer to obtain at least one request time period carried by the query request, and at least one request dimension corresponding to each request time period.

[0127] As an example, when a query request is received, the data application layer parses the query request to obtain at least one request time period (request time period 11:00 to 12:00, request time period 13:00 to 14:00) carried by the query request, as well as the request dimension corresponding to the request time period 11:00 to 12:00 (e.g., click volume request dimension and impression volume request dimension), and at least one request dimension corresponding to the request time period 13:00 to 14:00 (e.g., 11:00 to 12:00).

[0128] In step 1045, the target storage shards corresponding to each requested time period are determined.

[0129] As an example, the target storage shards corresponding to the request time periods of 11:00 to 12:00 and 13:00 to 14:00 are determined respectively.

[0130] In step 1046, the target dimension aggregated data is queried in the target storage shards corresponding to each request time period, and the target dimension aggregated data is used as the query result.

[0131] In some embodiments, the target dimension aggregation data is the dimension aggregation data stored at the storage location corresponding to each requested dimension.

[0132] In this way, the data to be processed is obtained through the source data layer, and then stored in at least two storage shards through the data service layer, taking into account the data dimensions and generation time. When a query request is received, the data application layer performs a data query on at least two storage shards to obtain the query results. Since there is a one-to-one correspondence between storage shards and generation time periods, storing data to be processed in at least two storage shards allows data from different generation time periods to be stored in different storage shards. During data queries, the required data can be retrieved from different storage shards. Therefore, by utilizing storage shards, the accuracy of data storage and querying in the data warehouse can be effectively improved.

[0133] The following will describe an exemplary application of the embodiments of this application in a real data warehouse application scenario.

[0134] A data warehouse is a structured data environment that serves as a data source for decision support systems and online analytical applications. Data warehouses study and solve the problem of retrieving information from databases. The characteristics of a data warehouse include subject-oriented nature, integration, stability, and time-varying characteristics.

[0135] This application embodiment divides the data warehouse into layers. At the structural level, the data warehouse is divided into a source data layer, a dimension table layer, a detail layer, a light aggregation layer, and an application layer. At the data link level, the data warehouse is divided into five layers: an access layer, a computation layer, an intermediate result layer, a storage layer, and a presentation layer. The data processing method based on the data warehouse provided in this application embodiment can decouple data modules, reduce redundant development costs, improve report usage efficiency, increase data utilization, and improve data development efficiency.

[0136] In some embodiments, see Figure 4A , Figure 4AThis is a schematic diagram illustrating the principle of the data processing method based on a data warehouse provided in this application embodiment. Structurally, the data warehouse is divided into a source data layer, a dimension table layer, a detail layer, a light aggregation layer, and an application layer. The source data layer is used to obtain source data based on different data sources. The dimension table layer stores the original system configuration, such as SSP platform configuration and DSP platform configuration. The data detail layer processes the source data through data cleaning, field expansion, and field translation to generate detailed data, such as request detail tables, behavior detail tables, process detail tables, and effect detail tables. The data service layer aggregates the data from the data detail layer according to certain dimensions to obtain lightly aggregated data, such as request information summary tables, process summary tables, promotion behavior summary tables, and promotion effect summary tables. Applications in the data application layer include real-time dashboards, business monitoring, and online learning. The real-time dashboard is generated by highly aggregating the lightly aggregated data from the data service layer according to dashboard application themes, facilitating efficient querying. Business monitoring is used for real-time monitoring of business data, such as detecting fluctuations in exposure and consumption every minute. If any anomalies are detected, notifications will be sent via real-time phone calls or other means. Online learning connects the detailed data layer to the algorithm side, providing reliable and real-time data to support real-time algorithm online learning and recommendation services.

[0137] The data processing method based on a data warehouse provided in this application can be divided into two parts: real-time computing and real-time storage. The following is a detailed description of these two parts.

[0138] The real-time computing component includes data cleaning and transformation, light data aggregation, real-time data storage, and data monitoring. The following section provides a detailed explanation of these aspects.

[0139] For the data cleaning and transformation section, the source data undergoes operations such as anomaly filtering, dimension expansion and translation, and formatting cleaning. The cleaned data is then output to the downstream normal data layer to generate detailed data with complete dimensions. Simultaneously, anomaly data is also output to a message queue for troubleshooting and reconciliation. The data volume from the source data layer to the detailed data layer remains almost unchanged; only the amount of anomaly data is reduced.

[0140] See Figure 4B , Figure 4BThis is a schematic diagram illustrating the principle of the data processing method based on a data warehouse provided in this application embodiment. The source data in the message queue of the source data layer and the dimension table data of the dimension table layer are cleaned and transformed to obtain the input data for the data detail layer. The access layer receives the source data from the source data layer and uses a message queue to receive the reported real-time data. The calculation layer and intermediate result layer primarily process and calculate the data, which can be further subdivided into: Cleaning and Transformation: After reading the source data from the message queue of the source data layer, operations such as filtering, translation, formatting, field expansion, field filling, and merging are performed. Field translation and transformation are performed on the dimension table configuration of the dimension table layer. After processing, the data is output to the downstream message queue as data detail layer data. Light Aggregation: Detail data is read from the message queue of the data detail layer, lightly aggregated according to the business subject, and the aggregated data is output to the downstream message queue as input data for the data service layer. Real-time Ingestion: This is used to consume the result data from the data service layer and ingest the data in real-time into a columnar database management system for storage. Storage Layer: This layer primarily uses a columnar database management system as its storage component, storing data from the data service layer, as well as highly aggregated data from the data source service layer after materialized views have aggregated the detailed data layer data. Application Layer: This layer mainly applies the data to scenarios such as real-time dashboards, business monitoring, and online algorithm learning. The real-time dashboard primarily interfaces with data queries from the data service layer and the detailed data layer. Business monitoring primarily interfaces with lightly aggregated data from the detailed data layer, and online learning primarily interfaces with the detailed data from the data service layer.

[0141] See Figure 4C , Figure 4C This is a schematic diagram illustrating the principle of the data processing method based on a data warehouse provided in this application embodiment. Interface A is used to receive data, for example, exposure data. Interface B is used to receive data, for example, click data. The data received by Interface A and Interface B are merged to obtain dual-stream data, and abnormal data in the dual-stream data is identified. For example, if data of integer type is identified as string type, the data is determined to be abnormal data and backed up. The abnormal data in the dual-stream data is then filtered and cleaned. Interface C is used to receive data from the dimension layer. The data from the dimension layer is read in a manner that is first fully read and then incrementally read thereafter. Each read stores the source and location of the read data in the form of a partition key. Even if the task is interrupted, it can still read the configuration normally after restarting and recovery, and the read configuration is backed up to a remote dictionary. If the platform fails and data is lost, the remote dictionary configuration is updated to the local machine. The data read from interface C is merged with the data read from interfaces A and B into a dimension table to obtain merged dimension table data. The merged dimension table data is then transformed to obtain output data, which is then sent to the data detail layer as input data for the data detail layer.

[0142] See Figure 4D , Figure 4D This is a schematic diagram illustrating the principle of the data processing method based on a data warehouse provided in this application embodiment. Data is read from the data detail layer and aggregated according to specified dimensions. Aggregation calculations are performed on metrics within a 1-minute window. The window size can be adjusted according to business needs, and it supports calculations for delayed data windows. For example, to calculate the exposure metrics of media within a 1-minute time window, the data within this 1-minute time window needs to be grouped by media, and the exposure of each media needs to be calculated separately. The aggregated results are output to the downstream data service layer. The purpose of light aggregation can be to decouple data before it is stored in the database, avoiding data loss, and it can also be used for business detection.

[0143] See Figure 4E , Figure 4E This is a schematic diagram illustrating the principle of the data processing method based on a data warehouse provided in this application embodiment. Data from a lightly aggregated data service layer is read, and the read data is output, thus completing the data ingestion. The ingested data can be integrated with a real-time dashboard.

[0144] See Figure 4F , Figure 4F This is a schematic diagram illustrating the principle of the data processing method based on a data warehouse provided in this application embodiment. It primarily involves real-time monitoring of business data, such as issuing alarms for exposure exceeding a certain value and consumption exceeding a certain value. This facilitates timely detection of fluctuations in business metrics, prompt identification of online anomalies in the system, timely notification of relevant communication processes to resolve issues, timely mitigation of business losses when consumption exceeds limits, and reasonable budget adjustments when consumption is low. Figure 4F The diagram illustrates the detailed alarm detection process. It reads data from the lightly aggregated data service layer. Based on the configured alarm rules, it performs alarm detection on real-time data. It periodically reads the configured alarm rules via an HTTP interface, for example: an alarm for a 100% increase in market exposure compared to the previous hour on Monday, an alarm for an 80% fluctuation on Tuesday, with the alarm receiving method being telephone and the alarm recipient corresponding to the daily duty roster. It also includes alarm convergence every half hour, etc. This is used for detailed configuration of alarm rules, convergence rules, etc. When a data metric reaches an alarm threshold, it first retrieves the alarm rule identifier from the remote dictionary to determine if the current alarm rule identifier is still within the convergence time (current time minus alarm time is greater than the convergence time). If convergence is achieved, no alarm is triggered, and no data is output downstream. Otherwise, the data is output downstream, and the alarm rule identifier and alarm time are recorded in the remote dictionary for future checks based on the rule identifier. Finally, it sends a telephone alarm based on the alarm receiving method and recipient configured in the alarm rules. At the same time, the alarm results will be output and stored for subsequent investigation of alarm records and troubleshooting.

[0145] See Figure 4G , Figure 4G This is a schematic diagram illustrating the principle of the data processing method based on a data warehouse provided in this application embodiment. The real-time data warehouse in this application embodiment employs a real-time storage component of a columnar database management system (Click Stream Data WareHouse, Click House), whose millisecond-level single-table query efficiency is widely recognized in the industry. A distributed processing engine (Flink) is used to write data to the columnar database management system, and a dashboard on a data visualization platform is used to query the data. In this application embodiment, an optimized write and query method is adopted: data writing involves using the distributed processing engine to write data consumed from the message queue to the columnar database management system. (Return status refers to the return code, such as 0 for successful write and -1 for failed write). Data is read from the message queue, such as lightly aggregated data from the data service layer. Based on the data corresponding to the partition key field specified when creating the table in the columnar database management system, hashing is performed, and the data is written to the corresponding shard in the backend of the columnar database management system, directly writing to the local table to improve write efficiency and avoid writing to the distributed table (to prevent write amplification caused by data distribution, which could overload a single node's disk).

[0146] See Figure 4H , Figure 4H This is a schematic diagram illustrating the principle of the data processing method based on a data warehouse provided in this application embodiment. Data querying involves retrieving data from the columnar database management system and displaying it on a dashboard for data analysis. Figure 4H This describes the detailed data query process. The real-time dashboard sends a query request to the distributed table of the columnar database management system. This request is forwarded to all backend shards. Each shard sends its query results to one of the shards (shard 1 - 1), merges the results (e.g., when querying today's exposure data in the columnar database management system, the "result" here is the sum of today's exposures), and returns it to the distributed table. The final result is then returned to the dashboard. Using a distributed table query ensures that the query results are complete and accurate, especially useful for deduplicated metrics applications.

[0147] See Figure 4I , Figure 4I This is a schematic diagram illustrating the principle of the data processing method based on a data warehouse provided in this application embodiment. After designing efficient write and query methods for the columnar database management system, a detailed overall design scheme for the columnar database management system used in this application embodiment is given, including write operations, local tables, materialized views, distributed tables, and queries. Figure 4IAs shown, after consuming data, the data is hashed according to the partition key and then routed to the corresponding shard, directly stored in the local table. This means the data service layer has entered the database. At this point, materialized views are used to highly aggregate data according to the dimensions required by the business. Data from the data service layer is used to create distributed tables to query data from the local table or materialized views, and then the distributed tables are used to display real-time dashboard data.

[0148] Thus, the data processing method based on a data warehouse provided in this application can effectively improve data timeliness, accuracy, and integrity. It utilizes a distributed processing engine computing framework with millisecond-level latency, and a columnar database management system for storage with millisecond-level latency. By optimizing internal structural components, a real-time data warehouse is built to minimize overall computation and query latency. The real-time data warehouse framework cleverly employs local and backup storage configurations, improving concurrent data computation performance and accuracy, while also effectively avoiding the problem of abnormal result data due to the inability to merge dimensions. A highly reliable message queue is used as the intermediate message queue layer, decoupling intermediate result data and data service layer data from the columnar database management system, thus preventing data loss due to cluster failures of the columnar database management system. Data integrity is further improved.

[0149] It is understood that in the embodiments of this application, data such as data to be processed is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0150] The following continues to describe the exemplary structure of the data processing apparatus 255 based on a data warehouse provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the data warehouse-based data processing device 255 in the memory 250 may include: an acquisition module 2551, used to acquire data to be processed through the source data layer; a determination module 2552, used to determine at least two data dimensions corresponding to the data to be processed, and at least two generation time periods corresponding to the generation time of the data to be processed, through the data detail layer; a storage module 2553, used to store the data to be processed in at least two storage shards through the data service layer, combining each data dimension and the generation time of the data to be processed; wherein, the storage shards and the generation time periods have a one-to-one correspondence; and a query module 2554, used to perform data query in at least two storage shards through the data application layer when a query request is received, and obtain query results.

[0151] In some embodiments, the determining module 2552 is further configured to perform dimensional aggregation on the data to be processed through the data detail layer to obtain at least two data dimensions corresponding to the data to be processed, and dimensional aggregation data corresponding to each data dimension; the data processing device based on the data warehouse further includes: a partitioning module, configured to perform the following processing on the dimensional aggregation data of each data dimension through the data detail layer: obtain the generation time of the dimensional aggregation data, and based on the generation time of the dimensional aggregation data, partition the dimensional aggregation data of the data dimension according to at least two generation time periods to obtain time period data corresponding to each generation time period; the storage module 2553 is further configured to find at least two storage shards corresponding to the data dimension through the data service layer, and store the time period data into the corresponding storage shards respectively.

[0152] In some embodiments, the dimensional aggregation data includes multiple data units; the above-mentioned partitioning module is further used to obtain the timestamp of each data unit in the dimensional aggregation data; compare the time point indicated by the timestamp of each data unit with each generation time period to obtain the generation time period to which each data unit belongs; for each generation time period, the data units belonging to the same generation time period are determined as the time period data corresponding to the generation time period.

[0153] In some embodiments, the query module 2554 is further configured to, when a query request is received, parse the query request through the data application layer to obtain at least one request dimension carried by the query request and at least one request time period corresponding to each request dimension; determine multiple target storage shards corresponding to each request dimension; for each request dimension, query target time period data in the corresponding multiple target storage shards, and use the target time period data as the query result; wherein, the target time period data is the time period data that matches each request time period corresponding to the request dimension.

[0154] In some embodiments, the above-described data warehouse-based data processing apparatus further includes: an anomaly detection module, configured to perform the following processing on data for each time period through a data detail layer: obtaining the data fluctuation threshold corresponding to the generation time period of the time period data; determining the data fluctuation ratio of the time period data within the generation time period; comparing the data fluctuation threshold and the data fluctuation ratio to obtain a comparison result, wherein the comparison result indicates whether the data fluctuation ratio is greater than the data fluctuation threshold; and determining the type of the time period data based on the comparison result, wherein the type is used to indicate whether the time period data is abnormal fluctuation data.

[0155] In some embodiments, the above-mentioned anomaly judgment module is further configured to determine the type of time period data as a data type with fluctuation anomaly when the comparison result indicates that the data fluctuation ratio is greater than the data fluctuation threshold; the above-mentioned data processing device based on data warehouse further includes: an output module, configured to output alarm prompt information, the alarm prompt information being used to indicate that the type of time period data is a data type with fluctuation anomaly.

[0156] In some embodiments, the above-described data warehouse-based data processing apparatus further includes: a partitioning module, configured to obtain the generation time of the data to be processed through a data detail layer, and partition the data to be processed according to at least two generation time periods based on the generation time of the data to be processed, to obtain time period data corresponding to each generation time period; the determination module 2552 is further configured to perform the following processing on the time period data of each generation time period through the data detail layer: perform dimensional aggregation on the time period data to obtain at least two data dimensions corresponding to the time period data, and dimensional aggregation data corresponding to each data dimension; the storage module 2553 is further configured to find the storage shard corresponding to each generation time period through a data service layer, wherein the storage shard corresponds one-to-one with the generation time period; and store the dimensional aggregation data of the at least two data dimensions corresponding to each generation time period to the storage location corresponding to each data dimension in the corresponding storage shard.

[0157] In some embodiments, the above-mentioned segmentation module is further configured to obtain the timestamp of each data unit in the data to be processed; compare the time point indicated by the timestamp of each data unit with each generation time period to obtain the generation time period to which each data unit belongs; and perform the following processing for each generation time period: determine each data unit belonging to the same generation time period as the time period data corresponding to the generation time period.

[0158] In some embodiments, the query module 2554 is further configured to, when a query request is received, parse the query request through the data application layer to obtain at least one request time period carried by the query request and at least one request dimension corresponding to each request time period; determine the target storage shard corresponding to each request time period; query the target dimension aggregate data in the target storage shard corresponding to each request time period, and use the target dimension aggregate data as the query result; wherein, the target dimension aggregate data is the dimension aggregate data stored at the storage location corresponding to each request dimension.

[0159] In some embodiments, the above-described data warehouse-based data processing apparatus further includes: a cleaning module, configured to obtain data to be processed from the source data layer and configuration data from the dimension table layer through the data detail layer, wherein the configuration data is used to configure parameters of the data warehouse; to merge the data to be processed and the configuration data through the data detail layer to obtain merged data; and to identify abnormal data in the merged data through the data detail layer and delete the abnormal data in the merged data to obtain data to be processed after the abnormal data is deleted.

[0160] In some embodiments, the determining module 2552 is further configured to determine, through the data detail layer, at least two data dimensions corresponding to the data to be processed after the deletion of abnormal data, and at least two generation time periods corresponding to the generation time of the data to be processed after the deletion of abnormal data.

[0161] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the computer device to perform the data processing method based on a data warehouse as described in this application.

[0162] This application provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by a processor, they cause the processor to execute the data processing method based on a data warehouse provided in this application. For example, ... Figure 3A The data processing method based on a data warehouse is shown.

[0163] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0164] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0165] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0166] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0167] In summary, the embodiments of this application have the following beneficial effects:

[0168] (1) Data to be processed is obtained through the source data layer. Then, through the data service layer, combined with data dimensions and generation time, the data to be processed is stored in at least two storage shards. When a query request is received, the data application layer performs a data query on at least two storage shards to obtain the query results. Thus, since there is a one-to-one correspondence between storage shards and generation time periods, storing data to be processed in at least two storage shards allows data from different generation time periods to be stored in different storage shards. During data queries, the required data can be retrieved from different storage shards. Therefore, by utilizing storage shards, the accuracy and efficiency of data storage and querying in the data warehouse can be effectively improved.

[0169] (2) By defining the relevant parameters of the fused data under different business scenarios, the abnormal data in the fused data can be determined through the relevant parameters of the fused data, and then the abnormal data in the fused data can be deleted to obtain the data to be processed after the abnormal data is deleted, thereby ensuring that there is no abnormal data in the data processed by the subsequent data detail layer, which effectively improves the accuracy of data processing.

[0170] (3) The data detail layer can first perform dimensional aggregation on the data to be processed to obtain dimensional aggregated data, and then divide the dimensional aggregated data into generation time periods to obtain the time period data corresponding to the generation time periods. Thus, the data to be processed is divided and aggregated according to the dimensions and generation time periods to obtain the time period data used to store in the storage shards, thereby ensuring a one-to-one correspondence between the storage shards and the generation time periods. It can store the data to be processed in different generation time periods in different storage shards, thereby effectively improving the accuracy and efficiency of data storage in the data warehouse.

[0171] (4) By comparing the data fluctuation threshold and the data fluctuation ratio of the time period data within the generated time period, the comparison result is obtained. Based on the comparison result, it is determined whether the time period data is abnormally fluctuating, and thus the abnormal fluctuation data is output, effectively prompting the relevant abnormal fluctuation data, providing data support for subsequent data processing, and effectively improving the accuracy of data processing.

[0172] (5) The data detail layer can first divide the data to be processed into time periods to obtain time period data, and then perform dimensional aggregation on the time period data to obtain dimensional aggregated data. Thus, the data to be processed is divided and aggregated according to the dimensions and the time period to obtain the time period data used to store in the storage shards, thereby ensuring a one-to-one correspondence between the storage shards and the time period. It can store the data to be processed in different time periods into different storage shards, thereby effectively improving the accuracy and efficiency of data storage in the data warehouse.

[0173] (6) The data processing method based on a data warehouse provided in this application can effectively improve data timeliness, accuracy, and integrity. It utilizes a distributed processing engine computing framework with millisecond-level latency, and a columnar database management system for storage with millisecond-level latency. A real-time data warehouse is built after optimizing internal structural components to ensure that overall computation and query latency are minimized. The real-time data warehouse framework cleverly employs local and backup storage configurations, improving concurrent data computation performance and accuracy, while also effectively avoiding the problem of abnormal result data due to the inability to merge dimensions. A highly reliable message queue is used as the intermediate message queue layer, decoupling intermediate result data and data service layer data from the columnar database management system, thus avoiding data loss due to cluster failures of the columnar database management system. Data integrity is further improved.

[0174] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method based on a data warehouse, characterized in that, The data warehouse includes a source data layer, a data detail layer, a data service layer, and a data application layer; the method includes: The data to be processed is obtained through the source data layer; Through the data detail layer, the data to be processed is aggregated in dimensions to obtain at least two data dimensions corresponding to the data to be processed, as well as the aggregated data of each data dimension, and to determine at least two generation time periods corresponding to the generation time of the data to be processed. Through the data detail layer, the following processing is performed on the dimensional aggregated data for each of the data dimensions: Based on the generation time of the dimensional aggregated data, the dimensional aggregated data of the data dimension is divided according to the at least two generation time periods to obtain the time period data corresponding to each generation time period. The data service layer hashes the time period data according to the partition key and routes it to the corresponding shard, which is then directly stored in the local table. The storage shard and the generated time period are in a one-to-one correspondence. When a query request is received, the data application layer uses materialized views to highly aggregate the data in the data service layer according to the dimensions required by the business; a distributed table is established to query the data in the local table or materialized view; the query request is forwarded to all backend shards so that the backend shards send the query results to one of the backend shards for merging and return them to the distributed table.

2. The method according to claim 1, characterized in that, The dimensional aggregation data includes multiple data units; the step of dividing the dimensional aggregation data according to the at least two generation time periods to obtain time period data corresponding to each generation time period includes: Obtain the timestamp of each data unit in the aggregated data of the dimension; The time point indicated by the timestamp of each data unit is compared with the generation time period to obtain the generation time period to which each data unit belongs; For each of the aforementioned generation time periods, the data units belonging to the same generation time period are determined as the time period data corresponding to that generation time period.

3. The method according to claim 1, characterized in that, When a query request is received, the data application layer uses materialized views to highly aggregate the data in the data service layer according to the dimensions required by the business; a distributed table is established to query the data in the local table or materialized view; the query request is forwarded to all backend shards so that each backend shard sends the queried results to one of the backend shards for merging and returns them to the distributed table, including: When the query request is received, the data application layer parses the query request to obtain at least one request dimension carried by the query request and at least one request time period corresponding to each request dimension; Each of the requested dimensions is then identified as a multiple target storage shard; For each of the requested dimensions, the target time period data is queried in the corresponding multiple target storage shards, and the target time period data is used as the query result; The target time period data is the time period data that matches each of the request time periods corresponding to the request dimension.

4. The method according to claim 1, characterized in that, After aggregating the data according to the dimensions of the data dimensions and dividing it according to the at least two generation time periods to obtain the time period data corresponding to each generation time period, the method further includes: For each of the aforementioned time period data, the following processing is performed through the data detail layer: Obtain the data fluctuation threshold of the generation time period corresponding to the time period data; Determine the data fluctuation ratio within the generated time period; The data fluctuation threshold and the data fluctuation ratio are compared to obtain a comparison result, wherein the comparison result indicates whether the data fluctuation ratio is greater than the data fluctuation threshold; Based on the comparison results, the type of the time period data is determined, wherein the type is used to characterize whether the time period data is abnormal fluctuation data.

5. The method according to claim 4, characterized in that, Determining the type of the time period data based on the comparison result includes: When the comparison result indicates that the data fluctuation ratio is greater than the data fluctuation threshold, the data type of the time period is determined to be a data type with abnormal fluctuation. The method further includes: Output alarm information, which indicates that the data type for the time period is the fluctuating abnormal data type.

6. The method according to claim 1, characterized in that, The data warehouse further includes a dimension table layer; after obtaining the data to be processed through the source data layer, the method further includes: The data to be processed is obtained from the source data layer through the data detail layer, and configuration data is obtained from the dimension table layer, wherein the configuration data is used to configure the parameters of the data warehouse; The data detail layer is used to merge the data to be processed and the configuration data to obtain merged data. Abnormal data in the fused data is identified through the data detail layer and deleted, resulting in data to be processed after the abnormal data is deleted.

7. The method according to claim 6, characterized in that, The step of performing dimensional aggregation on the data to be processed through the data detail layer to obtain at least two data dimensions corresponding to the data to be processed, and dimensional aggregation data corresponding to each data dimension, and determining at least two generation time periods corresponding to the generation time of the data to be processed, includes: Through the data detail layer, the data to be processed after the deletion of the abnormal data is aggregated in dimensions to obtain at least two data dimensions corresponding to the data to be processed after the deletion of the abnormal data, as well as the dimension aggregated data corresponding to each data dimension, and to determine at least two generation time periods corresponding to the generation time of the data to be processed after the deletion of the abnormal data.

8. A data processing device based on a data warehouse, characterized in that, The data warehouse includes a source data layer, a data detail layer, a data service layer, and a data application layer; the device includes: The acquisition module is used to acquire the data to be processed through the source data layer; The determination module is used to perform dimensional aggregation on the data to be processed through the data detail layer to obtain at least two data dimensions corresponding to the data to be processed, and dimensional aggregated data corresponding to each data dimension, and to determine at least two generation time periods corresponding to the generation time of the data to be processed. The partitioning module is used to perform the following processing on the dimension aggregated data of each data dimension through the data detail layer: based on the generation time of the dimension aggregated data, the dimension aggregated data of the data dimension is partitioned according to the at least two generation time periods to obtain the time period data corresponding to each generation time period; The storage module is used to hash the time period data according to the partition key through the data service layer, and then route it to the corresponding shard, directly storing it in the local table; wherein, the storage shard and the generated time period have a one-to-one correspondence. The query module is used to, when a query request is received, use the data application layer and materialized views to highly aggregate the data of the data service layer according to the dimensions required by the business; establish a distributed table to query the data of the local table or materialized view; the query request is used to forward to all backend shards, so that the backend shards send the query results to one of the backend shards for merging, and return them to the distributed table.

9. The apparatus according to claim 8, characterized in that, The dimensional aggregation data includes multiple data units; the partitioning module is further used for: Obtain the timestamp of each data unit in the aggregated data of the dimension; The time point indicated by the timestamp of each data unit is compared with the generation time period to obtain the generation time period to which each data unit belongs; For each of the aforementioned generation time periods, the data units belonging to the same generation time period are determined as the time period data corresponding to that generation time period.

10. The apparatus according to claim 8, characterized in that, The query module is also used for: When the query request is received, the data application layer parses the query request to obtain at least one request dimension carried by the query request and at least one request time period corresponding to each request dimension; Each of the requested dimensions is then identified as a multiple target storage shard; For each of the requested dimensions, the target time period data is queried in the corresponding multiple target storage shards, and the target time period data is used as the query result; The target time period data is the time period data that matches each of the request time periods corresponding to the request dimension.

11. The apparatus according to claim 8, characterized in that, The device further includes: The anomaly detection module is used to perform the following processing on the data for each time period through the data detail layer: Obtain the data fluctuation threshold of the generation time period corresponding to the time period data; Determine the data fluctuation ratio within the generated time period; The data fluctuation threshold and the data fluctuation ratio are compared to obtain a comparison result, wherein the comparison result indicates whether the data fluctuation ratio is greater than the data fluctuation threshold; Based on the comparison results, the type of the time period data is determined, wherein the type is used to characterize whether the time period data is abnormal fluctuation data.

12. The apparatus according to claim 11, characterized in that, The anomaly detection module is also used for: When the comparison result indicates that the data fluctuation ratio is greater than the data fluctuation threshold, the data type of the time period is determined to be a data type with abnormal fluctuation. The device further includes: The output module is used to output alarm information, which indicates that the data type for the time period is the fluctuating abnormal data type.

13. The apparatus according to claim 8, characterized in that, The data warehouse further includes a dimension layer; the device further includes: The cleaning module is used to obtain the data to be processed from the source data layer and the configuration data from the dimension table layer through the data detail layer, wherein the configuration data is used to configure the parameters of the data warehouse; to merge the data to be processed and the configuration data through the data detail layer to obtain merged data; and to identify and delete abnormal data in the merged data through the data detail layer to obtain data to be processed after deleting abnormal data.

14. The apparatus according to claim 13, characterized in that, The determining module is further configured to: Through the data detail layer, the data to be processed after the deletion of the abnormal data is aggregated in dimensions to obtain at least two data dimensions corresponding to the data to be processed after the deletion of the abnormal data, as well as the dimension aggregated data corresponding to each data dimension, and to determine at least two generation time periods corresponding to the generation time of the data to be processed after the deletion of the abnormal data.

15. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions or computer programs stored in the memory, implements the data processing method based on a data warehouse as described in any one of claims 1 to 7.

16. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed by the processor, they implement the data processing method based on a data warehouse as described in any one of claims 1 to 7.

17. A computer program product comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by a processor, the data processing method based on a data warehouse as described in any one of claims 1 to 7 is implemented.

Images