Data processing method, data query method, server, device and storage medium

Abstract

The present invention discloses a data processing method, a data query method, a server, a device and a storage medium. The data processing method of one embodiment includes: regularly pulling full data according to a preset time, partitioning the distributed file system according to the date, and storing the full data in the partition corresponding to the distributed file system; filtering the full data to obtain incremental data, dividing the incremental data and storing them in the dimension table and event table of the data detail layer respectively, and the dimension table and the event table are associated by a surrogate key; associating the dimension table and the event table to generate a report, generating an aggregation table according to the preset aggregation requirements and storing it in the report layer. The embodiment provided by the present application can solve the problem of numerous modules and complex relationships in data processing in the prior art by using associated dimension tables and event tables, while improving the efficiency of data processing and supporting data analysis, and has broad application prospects.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data processing method, a server, a data query method applied to the server, a data query method applied to a user end, a computer device and a storage medium. Background Art

[0002] With the development of the Internet and the diversification of IoT devices and their users, the data generated has become more complex and diverse, and the amount of data that needs to be transmitted is increasing, but at the same time, the timeliness requirements for processing data are also getting higher and higher.

[0003] Faced with a large amount of complex data, the existing technology for processing, analyzing, querying, and maintaining the data often involves multiple data modules with complex relationships, while consuming a lot of time and storage space. Summary of the invention

[0004] In order to solve at least one of the above problems, a first embodiment of the present invention provides a data processing method, comprising:

[0005] Pulling full data regularly according to a preset time, partitioning the distributed file system according to the date, and storing the full data in the partition corresponding to the distributed file system;

[0006] The full data is screened to obtain incremental data, the incremental data is divided and stored in a dimension table and an event table of a data detail layer respectively, and the dimension table and the event table are associated with each other through a surrogate key;

[0007] The dimension table and the event table are associated to generate a report, and an aggregation table is generated according to preset aggregation requirements and stored in the report layer.

[0008] Furthermore, after generating a report by associating the dimension table and the event table, generating an aggregation table according to a preset aggregation requirement and storing it in a report layer, the data processing method further includes:

[0009] The reports and aggregate tables are incrementally synchronized to a columnar storage database.

[0010] Further, the dimension table includes multiple dimension data and changes in each dimension data, the event table includes multiple event data and changes in each event data, the full data is filtered to obtain incremental data, the incremental data is divided and stored in the dimension table and event table of the data detail layer respectively, and the dimension table and event table are associated through a surrogate key and further include:

[0011] Filtering the full amount of data stored in the partition corresponding to the distributed file system to obtain incremental data according to the date;

[0012] Identify the incremental data according to a preset data type;

[0013] The dimension data is stored in the dimension table in the order of the life cycle and change time of the data;

[0014] generating the surrogate key according to the life cycle of the dimension data;

[0015] The event data is stored in the event table in the order of the life cycle and change time of the data, and the corresponding relationship between the life cycle of the event data and the surrogate key.

[0016] Furthermore, the generating the surrogate key according to the life cycle of the dimension data further includes:

[0017] The life cycle of the dimension data is sorted according to the change time and the sorting sequence number is used as the surrogate key.

[0018] Furthermore, the change frequency of the dimension data is less than a preset frequency threshold, and the change frequency of the event data is greater than the preset frequency threshold.

[0019] Furthermore, the filtering of the full data to obtain incremental data, data segmentation of the incremental data and storage of the incremental data in a dimension table and an event table of a data detail layer, wherein the dimension table and the event table are associated through a surrogate key, further comprises:

[0020] The historical data in the incremental data is divided and stored in a dimension table and an event table of a data detail layer respectively;

[0021] or

[0022] The incremental data is partitioned and stored in a dimension table and an event table at a data detail layer respectively.

[0023] A second embodiment of the present invention provides a server for implementing the above data processing method, comprising:

[0024] A data pulling unit is configured to pull full data regularly according to a preset time, partition the distributed file system according to the date, and store the full data in the partition corresponding to the distributed file system;

[0025] a data partitioning unit configured to filter the full data to obtain incremental data, partition the incremental data and store them in a dimension table and an event table of a data detail layer respectively, wherein the dimension table and the event table are associated with each other through a surrogate key;

[0026] The report generation unit is configured to associate the dimension table and the event table to generate a report, generate an aggregation table according to a preset aggregation requirement, and store the aggregation table in a report layer.

[0027] Furthermore, it also includes a report query unit configured to synchronize the report and aggregate table increments to a column storage database.

[0028] A third embodiment of the present invention provides a data query method, which is applied to the above server and includes:

[0029] Display reports and / or aggregate tables at the reporting level;

[0030] In response to a query request from a user, query the report layer and / or aggregate table and output the query result;

[0031] The query result is transmitted to the user terminal.

[0032] Further, the server further includes a report query unit for synchronizing the report and aggregate table increments to a column storage database, and the querying of the report layer report and / or aggregate table in response to a query request from the user terminal and outputting the query result further includes:

[0033] In response to a query request from a user, the report and / or aggregate table of the column storage database is queried and the query result is output.

[0034] A fourth embodiment of the present invention provides a data query method, applied to a user terminal, comprising:

[0035] generating a query request in response to a user operation and sending the query request to the server according to claim 7, so that the server performs a query based on a report and / or an aggregate table at a report layer, the report and / or the aggregate table being obtained from a dimension table and an event table at an associated data detail layer, the dimension table and the event table being divided into incremental data filtered from the full amount of data stored in a partition corresponding to the distributed file system according to a date;

[0036] Receive the query result of the server.

[0037] Furthermore, the server further comprises a report query unit for synchronizing the report and the aggregate table increment to the column storage database, and the query method further comprises:

[0038] In response to the user's operation, a query request is generated and sent to the server, so that the server performs a query based on the report and / or aggregate table of the column storage database.

[0039] A fifth embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned data processing method when executed by a processor;

[0040] or

[0041] When the program is executed by a processor, the data query method as described above is implemented;

[0042] or

[0043] When the program is executed by a processor, the above-mentioned data query method is implemented.

[0044] A sixth embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0045] When the processor executes the program, the data processing method as described above is implemented;

[0046] or

[0047] When the processor executes the program, the data query method as described above is implemented;

[0048] or

[0049] When the processor executes the program, the data query method as described above is implemented.

[0050] The beneficial effects of the present invention are as follows:

[0051] In view of the existing problems, the present invention develops a data processing method, which partitions the full data to achieve partition processing of the full data, classifies and stores the incremental data in dimension tables and event tables associated by proxy keys, and forms effective associations while achieving classified storage of incremental data, generates reports by associating dimension tables and event tables, and generates aggregation tables according to preset aggregation requirements and stores them in the report layer, which effectively improves the efficiency of data processing and can support data analysis, can solve the problem of numerous modules and complex relationships in data processing in the prior art, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 A flow chart showing a data processing method according to an embodiment of the present invention;

[0054] Figure 2 A specific example diagram showing the dimensional data described in the embodiment of the present invention;

[0055] Figure 3 A specific example diagram showing event data according to an embodiment of the present invention;

[0056] Figure 4a and 4b Specific example diagrams of the dimension table and the event table according to the embodiment of the present invention are respectively shown;

[0057] Figure 5 A block diagram of a server according to an embodiment of the present invention is shown;

[0058] Figure 6 A flow chart showing a data query method applied to a user terminal according to another embodiment of the present invention;

[0059] Figure 7 An architectural diagram showing a data query method applied to a user terminal according to an embodiment of the present invention;

[0060] Figure 8 A flow chart showing a data query method applied to a server according to another embodiment of the present invention;

[0061] Fig. 9 A schematic structural diagram of a computer device provided by another embodiment of the present invention is shown. DETAILED DESCRIPTION

[0062] In order to more clearly illustrate the present invention, the present invention is further described below in conjunction with preferred embodiments and accompanying drawings. Similar components in the accompanying drawings are represented by the same reference numerals. It should be understood by those skilled in the art that the content specifically described below is illustrative rather than restrictive, and should not be used to limit the scope of protection of the present invention.

[0063] It should also be noted that in the description of the present invention, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0064] The existing technology pulls the business data of the business to be analyzed in full by regularly synchronizing the database. However, the existing data processing includes multiple complex business modules, and the relationships between the modules are relatively complex. Therefore, the storage of full data and the screening, maintenance, processing and storage of incremental data are extremely difficult.

[0065] In order to solve the above problems, the present invention provides a data processing method, a server, a data query method applied to the server, and a data query method applied to a user end, a computer device and a storage medium.

[0066] like Figure 1 As shown, an embodiment of the present invention provides a data processing method, comprising:

[0067] Pulling full data regularly according to a preset time, partitioning the distributed file system according to the date, and storing the full data in the partition corresponding to the distributed file system;

[0068] The full data is screened to obtain incremental data, the incremental data is divided and stored in a dimension table and an event table of a data detail layer respectively, and the dimension table and the event table are associated with each other through a surrogate key;

[0069] The dimension table and the event table are associated to generate a report, and an aggregation table is generated according to preset aggregation requirements and stored in the report layer.

[0070] This embodiment partitions the full data to achieve partition processing of the full data, classifies and stores the incremental data in dimension tables and event tables associated by proxy keys, achieves classified storage of incremental data and forms effective associations at the same time, generates reports by associating dimension tables and event tables, and generates aggregation tables according to preset aggregation requirements and stores them in the report layer, which effectively improves the efficiency of data processing and can support data analysis, can solve the problem of numerous modules and complex relationships in data processing in the prior art, and has broad application prospects.

[0071] In a specific example, the data processing method is applied to a server, which may be a local server or a cloud server. The data processing method applied to the server is now described:

[0072] S1. Pull the full amount of data regularly according to a preset time, partition the distributed file system according to the date, and store the full amount of data in the partition corresponding to the distributed file system.

[0073] In this step, still taking the large amount of data generated by the above business processing as an example, the embodiment of the present invention processes the business data by building a data warehouse. Specifically, the embodiment of the present invention uses the Hive tool to build a data warehouse, and uses the Hive tool to regularly pull the full amount of data according to the preset time, for example, pulling the full amount of data at a fixed time point every day and storing the pulled full amount of data in a distributed file system.

[0074] Considering the large amount of full data, an embodiment of the present invention partitions the distributed file system by date and stores the pulled full data in corresponding partitions to achieve orderly storage of the full data in the distributed file system for subsequent rapid calling and processing.

[0075] In a specific example, for example, the business to be analyzed has been in operation since March 1, and the data processing method of this embodiment has been used to process the business data since March 8, wherein the preset time for pulling is 5:00 a.m. every day, that is, the full amount of data is pulled at 5:00 a.m. every day starting from March 8. Specifically, the full amount of data pulled on March 8 is the data from March 1 to March 8, which is historical data and all are incremental data. The distributed file system is partitioned according to the date, and the full amount of data pulled on March 8 is stored in the partition with the partition name "2021 / 03 / 08"; similarly, the full amount of data pulled on March 9 is the data from March 1 to March 9, and at the same time, the full amount of data is stored in the partition with the partition name "2021 / 03 / 09" of the distributed file system according to the date.

[0076] It is worth noting that the present application does not impose any specific restrictions on the time for periodic pulling. For example, the full amount of data can be pulled in a time period of one month and partitioned according to the time period. Technical personnel in this field should select the specific data pulling time according to actual needs, and no restrictions are imposed here.

[0077] Through the above process, the orderly storage of all data in the distributed file system is achieved, which facilitates subsequent calls and speeds up storage.

[0078] It is worth noting that the full data in this embodiment can be all the data of the business to be analyzed, or it can be part of the data of the business to be analyzed. This embodiment does not limit this. Technical personnel in this field should set it according to actual application requirements to meet the design criteria of including incremental data and ensuring data integrity. It will not be repeated here.

[0079] S2. Filter the full data to obtain incremental data, divide the incremental data and store them in a dimension table and an event table of a data detail layer respectively, wherein the dimension table and the event table are associated with each other through a surrogate key.

[0080] In this step, this embodiment implements classified storage of incremental data by dividing and associating the filtered incremental data, and associates the dimension table with the event table to form a connection for easy retrieval.

[0081] In this embodiment, incremental data is filtered out from the corresponding full data according to the partition name of the distributed file system. For example, based on the "2021 / 03 / 08" partition and the "2021 / 03 / 09" partition, the incremental data of the "2021 / 03 / 09" partition can be determined, and the data from 5:00 a.m. on March 8 to 5:00 a.m. on March 9 stored in the "2021 / 03 / 09" partition from March 1 to March 9 can be filtered out as incremental data.

[0082] In an optional embodiment, the dimension table includes multiple dimension data and changes in each dimension data, and the event table includes multiple event data and changes in each event data.

[0083] The dimension table of the embodiment of the present invention can record the information of all changes of the dimension data of a thing from the beginning to the current state. The dimension table can record the start time and end time of the dimension data in a tabular form. When the dimension data is initialized, the creation time is the initial time, and the end time is set to infinity. When the content field of the dimension data changes, the end time is set to the current modification time, which means the end of the dimension data, and the modified dimension data is added as a new record, so that the table has the historical full data and the latest data of the dimension data. In other words, the dimension table records the life cycle of each dimension data, and a record is a life cycle of the dimension data. For example, the dimension data is set to A at the initial time, changes to B at time 1, and changes to C at time 2. Then the life cycle of the dimension data A is from the initial time to time 1, the life cycle of the dimension data B is from time 1 to time 2, and the life cycle of the dimension data C starts from time 2 until it changes again.

[0084] It is worth noting that the dimension data of this embodiment is data with a low frequency of change, such as attribute data, or data with a frequency of change less than a preset frequency threshold, such as a frequency threshold of once a day. When the frequency of change of the data is less than the frequency threshold, it is set as dimension data, otherwise it is set as event data. The main application scenario of the dimension table is to record dimension data with a low frequency of change, for example, the historical snapshot information of the dimension data at a certain point in time or time period can be viewed through the dimension table.

[0085] The event table of this embodiment can record changes in event data, for example, every event that occurs, and records the life cycle and change time of the event data in a table.

[0086] It is worth noting that those skilled in the art can set dimension data and event data according to actual application requirements, taking the realization of the functions of the dimension table and the event table as the design criteria, which will not be elaborated here.

[0087] In another optional embodiment, the step S2 further comprises:

[0088] The incremental data is partitioned and stored in a dimension table and an event table at a data detail layer respectively.

[0089] In this embodiment, data identification and data division are performed on the incremental data according to preset data types, for example, the data type is determined according to the change frequency of each data in the incremental data.

[0090] In an optional embodiment, step S2 includes:

[0091] S21. Filter and obtain incremental data from the full amount of data stored in the partition corresponding to the distributed file system according to the date.

[0092] In an optional embodiment, the step S2 further comprises:

[0093] The historical data in the incremental data is divided and stored in a dimension table and an event table at a data detail layer respectively.

[0094] In a specific example, still taking the above-mentioned business to be analyzed as an example, since data processing started on March 8, the full data from March 1 to March 8 pulled on March 8 is historical data and is all incremental data, that is, all data since the start of the business operation is processed.

[0095] In another specific example, still taking the above-mentioned business to be analyzed as an example, the full data pulled on March 9 is the full data from March 1 to March 9. According to the "2021 / 03 / 08" partition and the "2021 / 03 / 09" partition in the distributed file system, the incremental data from March 8 to March 9 is filtered out from the full data stored in the "2021 / 03 / 09" partition.

[0096] S22: Identify the incremental data according to a preset data type.

[0097] Furthermore, taking the incremental data from March 8 to March 9 as an example, the specific data types therein are identified. In a specific example, the data types may include two types: event data that changes frequently and dimension data that changes infrequently. Therefore, the dimension data and event data are identified and divided from the incremental data. Specifically, the change frequency of the dimension data is less than the change frequency of the event data.

[0098] Take the business to be analyzed as obtaining the store order records in the Taobao backend as an example. For example, the order data is: user ID - "Li Si", delivery address - "Company A", order details - "Purchase Bookshelf". Among them, the dimension data that does not change frequently includes: user ID and delivery address, and the event data that changes frequently includes: order details. Therefore, the embodiment of the present invention identifies each order data and divides it into dimension data and time data, and stores it according to different data categories.

[0099] S23. Store the dimension data in the dimension table in the order of the life cycle and change time of the data.

[0100] Since dimensional data does not change frequently, dimension tables are used to store the life cycle and change time of dimensional data. For example, in the incremental data, the delivery address of user "Li Si" changes from "Company A" to "Family B" and then to "Parents' Address C". At the same time, there are order data at different delivery addresses. Among them, the delivery address is a dimensional data. There is a change from Company A to Family B. In the dimension table, it is represented by a new record added to the end of the current record of this dimensional data, that is, the end time of Company A is the time when the delivery address is changed to Family B. At the same time, a record is added to the dimensional data delivery address. This record is a record with the address of Family B until the address is modified again, for example, when it is changed to Parents' Address C, the record with the address of Family B ends and the address starts with Parents' Address C.

[0101] Dimension tables can quickly store the life cycle and change time of dimension data. Figure 2 As shown, the delivery address is used as the dimension data to record the change time and life cycle of the dimension data.

[0102] S24. Generate the surrogate key according to the life cycle of the dimension data.

[0103] In this embodiment, in order to associate the dimension table and the event table, a surrogate key is generated based on the life cycle of the dimension data with low change frequency, that is, the surrogate key is a variable that can identify the change of the dimension data. Figure 3 The following is the order data of user "Li Si" in the incremental data. User "Li Si" has changed the dimension data delivery address and placed an order at a different delivery address, causing the change in the event data order details. Figure 3 It can be seen that the incremental data has a large amount of data and contains redundant information, making it difficult to process the data.

[0104] In this embodiment, dimension data and event data are stored in the dimension table and event table respectively according to data type, and a surrogate key is established to associate dimension data and event data, so as to record each event data and avoid duplicate storage of dimension data. Figure 4a and 4b As shown, the changes in the dimension data "delivery address" are associated with the event data "order details", and the proxy key of the "order details" is generated based on the life cycle of the "delivery address". Multiple changed "delivery addresses" and proxy keys are recorded in the dimension table, and the event table clearly records the proxy keys, as well as the "order details" and "update time" of each "delivery address" corresponding to each proxy key, thereby using the proxy key to associate the two.

[0105] The embodiment of the present invention utilizes a surrogate key. When a dimension attribute is involved in an event table, there is no need to repeatedly store the dimension information. Instead, corresponding storage can be achieved simply by associating the surrogate keys of the two, thereby fully saving storage space.

[0106] In an optional embodiment, generating the surrogate key according to the life cycle of the dimension data further includes:

[0107] The life cycle of the dimension data is sorted according to the change time and the sorting sequence number is used as the surrogate key.

[0108] As shown in Figure 4a, the dimension table of this embodiment stores not only dimension data and the change time of dimension data, but also the proxy key identifying the life cycle of dimension data. The life cycle of dimension data is sorted according to the update time of dimension data to obtain the proxy key, for example, the self-incrementing change number is used as the proxy key, so as to further enhance the relevance of dimension data and event data on the basis of improving the utilization rate of storage space. This embodiment uses the sorting sequence number as the proxy key for unified naming, for example, the dimension data "Company A", "Family B" and "Parents' Address C" are respectively set to the corresponding proxy key 1, proxy key 2 and proxy key 3 according to the sorting order, that is, according to the life cycle of the dimension data "delivery address", the corresponding proxy key is established in a self-incrementing form for easy identification and processing.

[0109] S25: Store the event data in the event table in order of the life cycle and change time of the data, and the corresponding relationship between the life cycle of the event data and the surrogate key.

[0110] In this embodiment, according to the dimension data stored in the dimension table and the proxy key representing the association, the event table stores the event data corresponding to the proxy key. Since the event data has the attribute of frequent changes, this embodiment stores the event data in order according to the life cycle, change time, and the corresponding relationship between the life cycle of the event data and the proxy key.

[0111] In a specific example, if Figure 4b As shown, the event data stored in the event table is relatively Figure 3Dimension data is omitted, and the storage of event data is simplified by using proxy keys. When calling data, the association between dimension data and event data is achieved through proxy keys, which effectively improves the processing efficiency of data processing, enhances the relevance of data, and effectively reduces the occupied storage space.

[0112] It is also worth noting that this embodiment is only used to illustrate the specific implementation of the present application. In actual applications, the amount of data obtained is more complex. For example, the incremental data of computer equipment may include changes in the number of equipment, distribution of equipment types, equipment activity trends and resource usage, etc.; the incremental data of various online education systems may include: plans, programs, materials, templates, etc.; the incremental data of distributed institutional terminals may include user attributes, user needs, institutional attributes, institutional distribution, etc. More complex, the association relationship formed by the above-mentioned dimensional data and event data may include: equipment and institutions, plans and labels, plans and programs, plans and planned broadcast start and end times, plans and institutions, programs and labels, programs and materials, programs and institutions, materials and labels, materials and institutions, templates and institutions, users and institutions, etc. Therefore, after identifying the incremental data, the embodiment of the present invention associates the dimensional data and event data according to the data type, and generates a proxy key using the changes in the dimensional data. On the basis of being able to maintain comprehensive association, it can also effectively control the storage space and effectively improve the data processing speed.

[0113] S3. Associate the dimension table and the event table to generate a report, generate an aggregation table according to preset aggregation requirements, and store it in the report layer.

[0114] On the basis of the aforementioned division and association of dimension data and event data, quick calling and direct display for user needs can be achieved. For example, the report layer generates reports of incremental data by calling dimension data in the dimension table and event data in the event table for easy display. Furthermore, the report layer can also form an aggregation table according to preset aggregation requirements to facilitate data analysis and data display, wherein the aggregation requirements are to set analysis items according to the analysis requirements of the business data of the business to be analyzed, for example, according to the category of purchased items in the "Order Details", the purchase of daily necessities is aggregated into one category, and the purchase of food is aggregated into another category, thereby achieving more intuitive data analysis and data display.

[0115] In an optional embodiment, after generating a report by associating the dimension table and the event table, generating an aggregation table according to a preset aggregation requirement and storing it in a report layer, the data processing method further includes:

[0116] S4. Incrementally synchronize the report and aggregate table to a column storage database.

[0117] Considering that the report layer built based on the Hive tool component cannot respond quickly to user query instructions, this embodiment synchronizes the reports and aggregate tables stored in the report layer to the column storage database clickhouse, and utilizes the characteristics of the column storage database that supports fast queries. In response to the query instructions of the user end, the report or aggregate table can be output to the user end within ms, thereby further improving the user experience.

[0118] Corresponding to the data processing method provided in the above embodiment, as Figure 5 As shown, an embodiment of the present invention further provides a server for implementing the above data processing method, the server comprising:

[0119] A data pulling unit is configured to pull full data regularly according to a preset time, partition the distributed file system according to the date, and store the full data in the partition corresponding to the distributed file system;

[0120] a data partitioning unit configured to filter the full data to obtain incremental data, partition the incremental data and store them in a dimension table and an event table of a data detail layer respectively, wherein the dimension table and the event table are associated with each other through a surrogate key;

[0121] The report generation unit is configured to associate the dimension table and the event table to generate a report, generate an aggregation table according to a preset aggregation requirement, and store the aggregation table in a report layer.

[0122] In this embodiment, the data pulling unit partitions the full amount of data to achieve partition processing of the full amount of data. The data partitioning unit classifies and stores the incremental data in the dimension table and event table associated by the proxy key, thereby achieving classified storage of the incremental data and forming effective associations. The report generation unit associates the dimension table and the event table to generate a report, and generates an aggregation table according to preset aggregation requirements and stores it in the report layer, which effectively improves the efficiency of data processing and can support data analysis. It can solve the problem of numerous modules and complex relationships in data processing in the prior art, and has broad application prospects.

[0123] In an optional embodiment, the server further includes a report query unit configured to incrementally synchronize the report and aggregate table to the column storage database. This embodiment synchronizes the report and aggregate table stored in the report layer to the report query unit, and utilizes the column storage database to support the function of storing and querying a large number of reports, thereby realizing fast query of reports and aggregate tables.

[0124] Since the server provided in the embodiment of the present application corresponds to the data processing methods provided in the above-mentioned embodiments, the previous implementation is also applicable to the data processing method provided in this embodiment and will not be described in detail in this embodiment.

[0125] like Figure 6 As shown, an embodiment of the present invention further provides a data query method, which is applied to a user end and includes:

[0126] In response to the user's operation, a query request is generated and sent to the server, so that the server performs a query based on a report and / or an aggregate table at the report layer, where the report and / or the aggregate table is obtained from a dimension table and an event table at the associated data detail layer, where the dimension table and the event table are divided into incremental data filtered from the full data stored in the partition corresponding to the distributed file system according to the date;

[0127] Receive the query result of the server.

[0128] In a specific example, the user may be a seller in a Taobao store, and the server stores the order data of the user's Taobao store every year. The user needs to query the product category with the highest monthly sales in the store. Based on this demand, the user can preset the aggregation demand, and the server executes the above data processing method, and selects the incremental data of each month from the full amount of data stored in the partition corresponding to the distributed file system according to the date, and identifies and divides the incremental data to obtain dimension tables and event tables with associated relationships, and generates an aggregation table based on the dimension tables and event tables for query.

[0129] In another specific example, the server further generates reports and stores them in the report layer.

[0130] like Figure 7 As shown, when the user conducts a query in early April, the store computer 100 is used to conduct the query, thereby outputting a query instruction, which is uploaded to the above-mentioned server 101. The server 101 queries the stored aggregation table according to the query instruction and outputs the queried aggregation table.

[0131] In another specific example, when the user needs to count all orders in the store, the server 101 can also query the stored report according to the query instruction and output the query report. The user uses the store computer as the user end to receive the query result of the server.

[0132] By using this embodiment to perform data query, data analysis results can be obtained in a timely manner, and the problem of performing data analysis through a database in the prior art involving multiple data modules with complex relationships can be solved. The data processing process and the data processing architecture of the server can be effectively simplified, and it has broad application prospects.

[0133] In an optional embodiment, the server further includes a report query unit for synchronizing the report and the aggregate table increment to the column storage database, and the query method further includes:

[0134] In response to the user's operation, a query request is generated and sent to the server, so that the server performs a query based on the report and / or aggregate table of the column storage database.

[0135] In this embodiment, the server synchronizes the reports and aggregate tables generated by the report layer to the report query unit, and uses the column storage database to support the functions of storing and querying a large number of reports. When responding to user query requests, it achieves fast processing and improves the query speed.

[0136] In a specific example, the server can obtain data from multiple information publishing systems, such as monitoring data, publishing information, plans, and other data. On the user side, the user can input multiple query instructions, such as retrieving the playback content, playback date, and playback time of the monitoring data, to fully meet the user's playback needs. In another specific example, the server supports publishing to a single user side and publishing to multiple groups of user sides.

[0137] like Figure 8 As shown, an embodiment of the present invention further provides a data query method, which is applied to the server of the above embodiment, comprising:

[0138] Display reports and / or aggregate tables at the reporting level;

[0139] In response to a query request from a user, query the report layer and / or aggregate table and output the query result;

[0140] The query result is transmitted to the user terminal.

[0141] Still taking the user as a Taobao seller as an example, the server executes the above data processing method, filters out the incremental data of each month from the full data stored in the partition corresponding to the distributed file system according to the date, and identifies and divides the incremental data to obtain dimension tables and event tables with associated relationships, and generates an aggregation table based on the dimension tables and event tables for query.

[0142] In one specific example, the server can display an aggregate table at the report level. In another specific example, the server also generates a report and displays the report.

[0143] When a user needs to make a query, the server queries the reports and aggregate tables stored in the report layer in response to the query request, outputs the query results, and transmits the query results to the user end.

[0144] By using this embodiment to perform data query, data analysis results can be obtained in a timely manner, and the problem of performing data analysis through a database in the prior art involving multiple data modules with complex relationships can be solved. The data processing process and the data processing architecture of the server can be effectively simplified, and it has broad application prospects.

[0145] In an optional embodiment, the server further includes a report query unit for synchronizing the report and aggregate table increments to a column storage database, and the querying of the report layer report and / or aggregate table in response to a query request from the user terminal and outputting the query result further includes:

[0146] In response to a query request from a user, the report and / or aggregate table of the column storage database is queried and the query result is output.

[0147] In this embodiment, the server synchronizes the reports and aggregate tables generated by the report layer to the report query unit, and uses the column storage database to support the functions of storing and querying a large number of reports. When responding to user query requests, it achieves fast processing and improves the query speed.

[0148] Since the data query method applied to the server provided in the embodiment of the present application corresponds to the data query method applied to the client provided in the above-mentioned embodiments, the previous implementation method is also applicable to the data query method provided in this embodiment and will not be described in detail in this embodiment.

[0149] Another embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements: periodically pulling full data at a preset time, partitioning a distributed file system according to date, and storing the full data in partitions corresponding to the distributed file system; filtering the full data to obtain incremental data, partitioning the incremental data and storing them respectively in dimension tables and event tables at a data detail layer, wherein the dimension tables and event tables are associated via surrogate keys; associating the dimension tables and event tables to generate reports, generating aggregation tables according to preset aggregation requirements and storing them at a report layer.

[0150] In another embodiment, when the program is executed by the processor, it can also achieve: displaying reports and / or aggregate tables at the report layer; querying reports and / or aggregate tables at the report layer in response to query requests from the user end and outputting query results; transmitting the query results to the user end.

[0151] In another embodiment, when the program is executed by the processor, it can also achieve: generating a query request in response to the user's operation and sending it to the server, so that the server performs a query based on the report and / or aggregation table of the report layer, the report and / or aggregation table is obtained by the dimension table and event table of the associated data detail layer, and the dimension table and event table are divided into incremental data filtered from the full data stored in the partition corresponding to the distributed file system according to the date.

[0152] Those skilled in the art select a computer program stored in a storage medium according to actual applications, so that the above-mentioned different methods are implemented when the program is executed by a processor.

[0153] In practical applications, the computer-readable storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0154] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0155] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0156] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0157] like Fig. 9 As shown, a structural schematic diagram of a computer device provided by another embodiment of the present invention. Fig. 9 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0158] like Fig. 9 As shown, the computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0159] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0160] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0161] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Fig. 9 not shown, usually called a "hard drive"). Although Fig. 9 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0162] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0163] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed through an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 20. Fig. 9 As shown, the network adapter 20 communicates with other modules of the computer device 12 via the bus 18. It should be understood that although Fig. 9 Not shown, other hardware and / or software modules may be used in conjunction with computer device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0164] The processor unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the data processing method provided in the embodiment of the present invention, the data query method applied to the server, and the data query method applied to the user end.

[0165] It should be noted that the sequence of the steps of the above method provided in the embodiment of the present invention can be appropriately adjusted, and the steps can be increased or decreased accordingly according to the circumstances. Any method that can be easily changed by any technician familiar with the technical field within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention, so it will not be repeated.

[0166] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not limitations on the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: Pulling full data regularly according to a preset time, partitioning the distributed file system according to the date, and storing the full data in the partition corresponding to the distributed file system; The full data is screened to obtain incremental data, the incremental data is divided and stored in a dimension table and an event table of a data detail layer respectively, and the dimension table and the event table are associated with each other through a surrogate key; The dimension table includes multiple dimension data and changes of each dimension data, and the event table includes multiple event data and changes of each event data, further including: Filtering the full amount of data stored in the partition corresponding to the distributed file system to obtain incremental data according to the date; Identify the incremental data according to a preset data type; The dimension data is stored in the dimension table in the order of the life cycle and change time of the data; generating the surrogate key according to the life cycle of the dimension data; The event data is stored in the event table in the order of the life cycle and change time of the data and the correspondence between the life cycle of the event data and the surrogate key; The dimension table and the event table are associated to generate a report, and an aggregation table is generated according to preset aggregation requirements and stored in the report layer.

2. The data processing method according to claim 1, characterized in that: After generating a report by associating the dimension table with the event table, generating an aggregation table according to a preset aggregation requirement and storing it in a report layer, the data processing method further includes: The reports and aggregate tables are incrementally synchronized to a columnar storage database.

3. The data processing method according to claim 1, characterized in that: Generating the surrogate key according to the life cycle of the dimension data further includes: The life cycle of the dimension data is sorted according to the change time and the sorting sequence number is used as the surrogate key.

4. The data processing method according to claim 1, characterized in that: The change frequency of the dimension data is less than a preset frequency threshold, and the change frequency of the event data is greater than the preset frequency threshold.

5. The data processing method according to claim 1, characterized in that: The step of filtering the full data to obtain incremental data, dividing the incremental data and storing them in a dimension table and an event table of a data detail layer respectively, wherein the dimension table and the event table are associated through a surrogate key, further comprises: The historical data in the incremental data is divided and stored in a dimension table and an event table of a data detail layer respectively; or The incremental data is partitioned and stored in a dimension table and an event table at a data detail layer respectively.

6. A server for implementing the data processing method according to any one of claims 1 to 5, characterized in that: include: A data pulling unit is configured to pull full data regularly according to a preset time, partition the distributed file system according to the date, and store the full data in the partition corresponding to the distributed file system; a data partitioning unit configured to filter the full data to obtain incremental data, partition the incremental data and store them in a dimension table and an event table of a data detail layer respectively, wherein the dimension table and the event table are associated with each other through a surrogate key; The report generation unit is configured to associate the dimension table and the event table to generate a report, generate an aggregation table according to a preset aggregation requirement, and store the aggregation table in a report layer.

7. The server according to claim 6, characterized in that: It also includes a report query unit configured to synchronize the report and the aggregate table increments to a column storage database.

8. A data query method, applied to the server according to claim 6, characterized in that: include: Display reports and / or aggregate tables at the reporting level; In response to a query request from a user, query the report layer and / or aggregate table and output the query result; The query result is transmitted to the user terminal.

9. The data query method according to claim 8, characterized in that: The server further includes a report query unit for synchronizing the report and aggregate table increments to a column storage database, and the querying of the report layer report and / or aggregate table in response to a query request from the user terminal and outputting the query result further includes: In response to a query request from a user, the report and / or aggregate table of the column storage database is queried and the query result is output.

10. A data query method, applied to a user end, characterized in that: include: generating a query request in response to a user operation and sending the query request to the server according to claim 6, so that the server performs a query based on a report and / or an aggregate table at a report layer, wherein the report and / or the aggregate table is obtained by associating a dimension table and an event table at a data detail layer, wherein the dimension table and the event table are divided by incremental data filtered from the full amount of data stored in the partition corresponding to the distributed file system according to a date; Receive the query result of the server.

11. The data query method according to claim 10, characterized in that: The server further includes a report query unit for synchronizing the report and the aggregate table increment to a column storage database, and the query method further includes: In response to the user's operation, a query request is generated and sent to the server, so that the server performs a query based on the report and / or aggregate table of the column storage database.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data processing method according to any one of claims 1 to 5 is implemented; or When the program is executed by a processor, the data query method according to claim 8 or 9 is implemented; or When the program is executed by a processor, the data query method according to claim 10 or 11 is implemented.

13. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the data processing method according to any one of claims 1 to 5 is implemented; or When the processor executes the program, the data query method according to claim 8 or 9 is implemented; or When the processor executes the program, the data query method according to claim 10 or 11 is implemented.

Images