Data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining original business data and configuration information, carrying out the shunting of the original business data based on the configuration information, obtaining dimension data and fact data, and storing the dimension data and the fact data in different databases, after the data summarization demand is obtained, the fact data under the specified dimension can be calculated based on the data summarization demand to obtain the summarized data. The dynamic distribution of the original business data is realized based on the configuration information, so that the dimension data is efficiently stored, and the query delay is avoided.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] With the continuous accumulation of data in smart healthcare, smart cities, and various IoTs, query applications based on stream computing are becoming more and more common. Therefore, how to avoid query delays is a technical problem that needs to be solved urgently. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a data processing method, device, electronic device and storage medium to avoid query delay. The specific technical solution is as follows:

[0004] The present application provides a data processing method, the method comprising:

[0005] Obtain original business data and configuration information;

[0006] Divide the original business data based on the configuration information to obtain dimension data and fact data;

[0007] storing the dimension data in a first database, and storing the fact data in a first message system;

[0008] Acquire data aggregation requirements; the data aggregation requirements represent the specified dimension that needs to be aggregated and the metric value corresponding to the specified dimension; the specified dimension belongs to the dimension in the dimensional data; the metric value corresponding to the specified dimension is calculated based on the fact data;

[0009] According to the data aggregation requirement, the metric value corresponding to the specified dimension is obtained from the fact data, and is calculated to obtain the aggregated data, and the aggregated data is stored in the first database.

[0010] In a possible implementation manner, the dividing the original business data based on the configuration information to obtain dimension data and fact data includes:

[0011] Filtering the original business data to obtain mainstream data;

[0012] Parsing the configuration information to obtain a configuration information broadcast stream, and setting the configuration information broadcast stream to a broadcast state;

[0013] Connecting the mainstream data and the configuration information broadcast stream to obtain connected stream data;

[0014] The connection stream data is split and processed based on a streaming computing program to obtain dimension data and fact data.

[0015] In a possible implementation manner, filtering the original business data to obtain mainstream data includes:

[0016] For any original business data, determine whether the original business data is in a preset mainstream data format. If so, treat the original business data as mainstream data; if not, treat the original business data as dirty data.

[0017] In a possible implementation, the first database is a database supporting random reading and writing and real-time analysis, and the first message system is a message system supporting a distributed subscription function.

[0018] In a possible implementation manner, the first database includes a cache database and a KUDU database;

[0019] The step of obtaining the metric value corresponding to the specified dimension from the fact data according to the data aggregation requirement and performing calculation to obtain the aggregated data includes:

[0020] According to the data aggregation requirement, when the fact data and the dimension data are associated, it is determined whether the preset cache database has the specified dimension corresponding to the data aggregation requirement. If so, an association request for the specified dimension is sent to the preset cache database, and the specified dimension is obtained from the preset cache database to aggregate the data to obtain the aggregated data;

[0021] If not, a correlation request of the specified dimension is sent to the first database based on the fact data in the first message system, and the metric value of the specified dimension is calculated according to the data aggregation requirement to obtain the aggregated data.

[0022] In a possible implementation, the method further includes:

[0023] According to the data aggregation requirement, the metric value corresponding to the specified dimension is obtained from the fact data, and after calculation is performed to obtain the aggregated data, the specified dimension corresponding to the data aggregation requirement is loaded from the first database to the preset cache database.

[0024] In a possible implementation, storing the dimension data in a first database and storing the fact data in a first message system includes:

[0025] Convert the dimension data into a dimension logic table, and convert the counted fact data into a fact logic table;

[0026] The dimension logic table is stored in a first database, and the fact logic table is stored in a first message system.

[0027] In a possible implementation, after obtaining the metric value corresponding to the specified dimension from the fact data according to the data aggregation requirement, performing calculation to obtain the aggregated data, and storing the aggregated data in the first database, the method further includes:

[0028] Obtaining a data cube model pre-built by a user through a preset BI analysis platform; the pre-built data cube model includes a combination of dimensions that need to be aggregated;

[0029] For each dimension combination, aggregate the dimension combination to obtain the metric value corresponding to the dimension combination;

[0030] For each dimension combination, the dimension combination and the metric value corresponding to the dimension combination are stored in the folder corresponding to the dimension combination in the first database in the form of a data cube model.

[0031] In a possible implementation, the method further includes:

[0032] For each dimension combination, determine the first folder corresponding to the dimension combination in the first database, and determine the first storage block to which the first folder belongs; obtain the local preset deduplication mode corresponding to the first storage block, and use the local preset deduplication mode corresponding to the first storage block to deduplicate the dimension logic table under the dimension combination to obtain the deduplication logic table under the dimension combination; store the deduplication logic table under the dimension combination in the first folder.

[0033] In a possible implementation, aggregating each dimension combination to obtain a metric value corresponding to the dimension combination includes:

[0034] Obtain each maximum subset of the dimension set including each dimension, and obtain multiple dimension combinations of the current level;

[0035] For each dimension combination at the current level, the dimension logic table of the dimension combination is aggregated to obtain the dimension logic table under the dimension combination;

[0036] For each dimension combination of the current level, obtain each maximum subset of the dimension combination of the current level, and obtain the dimension combination of the next level of the current level;

[0037] Take the next level dimension combination as the current level dimension combination, and return to the execution step: for each current level dimension combination, aggregate the dimension logic table of the dimension combination to obtain the dimension logic table under the dimension combination, until there are no subsets for each current level dimension combination, and obtain the dimension logic table under each dimension combination.

[0038] In a possible implementation, for each dimension combination at the current level, aggregating the dimension logic table of the dimension combination to obtain the dimension logic table under the dimension combination includes:

[0039] For each dimension combination at the current level, based on the deduplication logic table at the previous level of the dimension combination at the current level, the dimension logic table for the dimension combination at the current level is aggregated to obtain the dimension logic table under the dimension combination.

[0040] The present application also provides a data processing device, the device comprising:

[0041] Acquisition module, used to obtain original business data and configuration information;

[0042] A distribution module, used to distribute the original business data based on the configuration information to obtain dimension data and fact data;

[0043] A storage module, used for storing the dimension data in a first database and storing the fact data in a first message system;

[0044] A calculation module, used for obtaining data aggregation requirements; the data aggregation requirements represent the specified dimension to be aggregated and the metric value corresponding to the specified dimension; the specified dimension belongs to the dimension in the dimensional data; the metric value corresponding to the specified dimension is calculated based on the fact data;

[0045] The analysis module is used to obtain the measurement value corresponding to the specified dimension from the fact data, perform calculations to obtain summary data, and store the summary data in the first database.

[0046] In a possible implementation manner, the flow splitting module is specifically used for:

[0047] Filtering the original business data to obtain mainstream data;

[0048] Parsing the configuration information to obtain a configuration information broadcast stream, and setting the configuration information broadcast stream to a broadcast state;

[0049] Connecting the mainstream data and the configuration information broadcast stream to obtain connected stream data;

[0050] The connection stream data is split and processed based on a streaming computing program to obtain dimension data and fact data.

[0051] In a possible implementation manner, the flow splitting module is specifically used for:

[0052] For any original business data, determine whether the original business data is in a preset mainstream data format. If so, treat the original business data as mainstream data; if not, treat the original business data as dirty data.

[0053] In a possible implementation, the first database is a database supporting random reading and writing and real-time analysis, and the first message system is a message system supporting a distributed subscription function.

[0054] In a possible implementation manner, the first database includes a cache database and a KUDU database;

[0055] The calculation module is specifically used for:

[0056] According to the data aggregation requirement, when the fact data and the dimension data are associated, it is determined whether the preset cache database has the specified dimension corresponding to the data aggregation requirement. If so, an association request for the specified dimension is sent to the preset cache database, and the specified dimension is obtained from the preset cache database to aggregate the data to obtain the aggregated data;

[0057] If not, a correlation request of the specified dimension is sent to the first database based on the fact data in the first message system, and the metric value of the specified dimension is calculated according to the data aggregation requirement to obtain the aggregated data.

[0058] In a possible implementation, the device further includes:

[0059] The cache module is used to obtain the measurement value corresponding to the specified dimension from the fact data according to the data aggregation requirement, and calculate and obtain the aggregated data, and then load the specified dimension corresponding to the data aggregation requirement from the first database to the preset cache database.

[0060] In a possible implementation manner, the storage module is specifically used for:

[0061] Convert the dimension data into a dimension logic table, and convert the counted fact data into a fact logic table;

[0062] The dimension logic table is stored in a first database, and the fact logic table is stored in a first message system.

[0063] In a possible implementation, the device further includes:

[0064] A preprocessing module is used to obtain a data cube model pre-built by a user through a preset BI analysis platform; the pre-built data cube model includes a dimension combination that needs to be aggregated; for each dimension combination, the dimension combination is aggregated to obtain a measurement value corresponding to the dimension combination; for each dimension combination, the dimension combination and the measurement value corresponding to the dimension combination are stored in the form of a data cube model in a folder corresponding to the dimension combination in the first database.

[0065] In a possible implementation manner, the storage module is specifically used for:

[0066] For each dimension combination, determine the first folder corresponding to the dimension combination in the first database, and determine the first storage block to which the first folder belongs; obtain the local preset deduplication mode corresponding to the first storage block, and use the local preset deduplication mode corresponding to the first storage block to deduplicate the dimension logic table under the dimension combination to obtain the deduplication logic table under the dimension combination; store the deduplication logic table under the dimension combination in the first folder.

[0067] In a possible implementation manner, the storage module is specifically used for:

[0068] Obtain each maximum subset of the dimension set including each dimension, and obtain multiple dimension combinations of the current level;

[0069] For each dimension combination at the current level, the dimension logic table of the dimension combination is aggregated to obtain the dimension logic table under the dimension combination;

[0070] For each dimension combination of the current level, obtain each maximum subset of the dimension combination of the current level, and obtain the dimension combination of the next level of the current level;

[0071] Take the next level dimension combination as the current level dimension combination, and return to the execution step: for each current level dimension combination, aggregate the dimension logic table of the dimension combination to obtain the dimension logic table under the dimension combination, until there are no subsets for each current level dimension combination, and obtain the dimension logic table under each dimension combination.

[0072] In a possible implementation manner, the storage module is specifically used for:

[0073] For each dimension combination at the current level, based on the deduplication logic table at the previous level of the dimension combination at the current level, the dimension logic table for the dimension combination at the current level is aggregated to obtain the dimension logic table under the dimension combination.

[0074] An embodiment of the present application also provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor is used to implement any of the above-mentioned data processing methods when executing the programs stored in the memory.

[0075] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned data processing methods is implemented.

[0076] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned data processing methods.

[0077] Beneficial effects of the embodiments of the present application:

[0078] The data processing method, device, electronic device and storage medium provided in the embodiments of the present application obtain original business data and configuration information, and divert the original business data based on the configuration information to obtain dimension data and fact data, respectively store the dimension data and fact data in different databases, and after obtaining the data aggregation requirements, calculate the fact data under the specified dimension based on the data aggregation requirements to obtain the aggregation data. Diverting the original business data based on the configuration information can make the dimension data efficiently stored and avoid query delays.

[0079] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0081] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0082] Figure 2 A schematic diagram of the implementation process of the dynamic diversion mechanism provided in the embodiment of the present application;

[0083] Figure 3 A schematic diagram of synchronous and asynchronous IO provided for this application;

[0084] Figure 4A schematic diagram of the preload cache mechanism provided by this application;

[0085] Figure 5-1 A schematic diagram of the Cube model provided for this application;

[0086] Figure 5-2 A schematic diagram of the Cube decomposition principle diagram provided for this application;

[0087] Figure 5-3 A schematic diagram of the Flink pre-computation + Kudu storage OLAP engine provided for this application;

[0088] Figure 6 An interactive schematic diagram of the data processing method provided in this application;

[0089] Figure 7 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0090] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0091] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0092] In order to avoid query delays, embodiments of the present application provide a data processing method, apparatus, electronic device, computer-readable storage medium, and a computer program product containing instructions.

[0093] Below, the data processing method provided by the embodiment of the present application is first introduced. The method is applied to any electronic device that can provide data processing services, such as a computer, a mobile phone, a tablet, a game console, etc. The data processing method provided by the embodiment of the present application can be implemented by at least one of software, hardware circuits, and logic circuits set in the electronic device.

[0094] like Figure 1 As shown, Figure 1 A schematic diagram of a data processing method provided in an embodiment of the present application, the method comprising:

[0095] S110, obtaining original service data and configuration information;

[0096] S120, splitting the original business data based on the configuration information to obtain dimension data and fact data;

[0097] S130, storing the dimension data in a first database, and storing the fact data in a first message system;

[0098] S140, obtaining data aggregation requirements; the data aggregation requirements represent the specified dimensions that need to be aggregated and the metric values ​​corresponding to the specified dimensions; wherein the specified dimensions belong to the dimensions in the dimensional data; the metric values ​​corresponding to the specified dimensions are calculated based on the fact data;

[0099] S150, according to the data aggregation requirement, obtaining the metric value corresponding to the specified dimension from the fact data, performing calculations to obtain aggregated data, and storing the aggregated data in the first database.

[0100] The original business data includes structured data Binlog logs, semi-structured server logs and unstructured data under the business scenarios to be analyzed. Specifically, the real-time business data and log data can be obtained from the business database and the log server to obtain the original business data, and the original business data can be written into Kafka in real time, where Kafka is a distributed publish / subscribe message queue system across programming languages. In the publish-subscribe message system, messages are persisted to a topic. Consumers can subscribe to one or more topics, and consumers can consume all the data in the topic. The same data can be consumed by multiple consumers, and the data will not be deleted immediately after being consumed. In the publish-subscribe message system, the producer of the message is called the publisher, and the consumer is called the subscriber. The business scenario can be smart medical care, smart city, or various IoT application scenarios, such as the Internet of Vehicles.

[0101] Configuration information is used to determine the conditions for processing raw business data, and may include data source information, data sink (storage) information, data operation information, valid fields, primary key information, etc. Among them, data source information represents the data source and connection method, for example, specifying that the raw data comes from different topics of different message queues; data sink information represents the data flow, target source, and docking method, for example, for each type of data in the raw business data, to which database the data is sent for storage, for example, data is sent to Kafka, or to other databases, such as Hbase (an open source non-relational distributed database) database, KUDU database, etc. Valid fields can be used to filter raw business data, retaining only data containing "valid fields" and filtering out other data. Primary key information is one or more fields in a table, and its value is used to uniquely identify a record in the table.

[0102] Data operations (CURD) are any atomic operations for processing data, including: create, update, read and delete operations. For example, according to the created standard data format, the data format is converted to a unified message standard format; data of a specified type is deleted, or data is aggregated, etc. Configuration information is maintained by the administrator and can be stored in MySQL, which is a relational database management system.

[0103] After obtaining the original business data and configuration information, the original business data can be diverted based on the configuration information to obtain dimensional data and factual data. Exemplarily, the original business data can be desensitized, dimensionally degraded, and processed by ETL (Extraction-Transformation-Loading) based on the configuration information to obtain dimensional data and factual data. Exemplarily, mainstream data is obtained based on the original business data, and then CDC (Change Data Capture) is used to read the configuration information, parse the configuration information, obtain the configuration information broadcast stream, and process the mainstream data and broadcast stream data based on FLINK to obtain dimensional data and factual data.

[0104] Among them, dimension information is information describing dimensions, including: time dimension, space dimension, user dimension, equipment common dimension, theme / topic dimension, etc., among which, time dimension includes: year, month, day, season, hour, minute and second; space dimension includes: poster height, ground height, longitude and latitude, etc.; region dimension includes: country, province, city, district / county, township, street, road, door number, room number, etc.; user dimension can include user name, user age, gender, etc.; equipment common dimension includes: equipment name, category, application occasion, etc.; theme / topic dimension includes topic name, theme name, etc. Among them, the dimension under each application scenario is set based on the current application scenario. The table designed after the integration of dimension information through dimensionality reduction, desensitization, etc. is a dimension table, such as: user dimension table, region dimension table, category dimension table, etc. Fact data is usually a description of the business process, which can be summarized as inseparable behavioral events, such as registration, cancellation of registration, payment, return of orders, etc. in medical transactions, which are all business processes. Declare granularity based on the business process, clarify associated dimensions and metrics, and then construct fact tables, such as user registration transaction tables in the user domain and purchase fact tables in the transaction domain.

[0105] After the dimension data and the fact data are acquired, the dimension data may be stored in the first database, and the fact data may be stored in the first message system.

[0106] Exemplarily, the first database may be a database supporting random reading and writing and real-time analysis, such as an Hbase database, a KUDU database, etc., and the first message system may be a message system supporting a distributed subscription function, such as the first message system is Kafka.

[0107] Data aggregation requirements represent the specified dimensions that need to be aggregated and the metric values ​​corresponding to the specified dimensions. For example, in the application scenario of counting mobile phone sales, the dimensional data includes the regional dimension, the sales time dimension, the mobile phone brand dimension, and the mobile phone model dimension. Data aggregation requirements represent the specified dimensions that need to be aggregated, such as the regional dimension, the sales time dimension, and the mobile phone brand dimension. The metric values ​​corresponding to the specified dimensions are the specific metric values ​​of the regional dimension, the sales time dimension, and the mobile phone brand dimension. Data aggregation requirements can be obtained by users by operating preset access tools. The preset access tools can be management information system tools, online analysis tools, data mining tools, etc. For example, OLAP (Online Analytical Processing), BI (Business Intelligence) tools, etc.

[0108] In one example, after obtaining the data aggregation requirement, based on the data aggregation requirement, the first computing engine can be used to calculate the dimension data in the first database to obtain the calculation result of the dimension data, and the second computing engine can be used to calculate the fact data in the first message system to obtain the calculation result of the fact data. Finally, the calculation result of the dimension data and the calculation result of the fact data are associated to obtain the aggregation result. The first computing engine and the second computing engine can be the same or different. The first computing engine is a computing engine supported by the first database, and the second computing engine is a computing engine supported by the first message system, which is specifically determined based on the actual situation. For example, when the first database is a KUDU database, the first computing engine is an Impala computing engine. Based on the first computing engine and the second computing engine, user interaction and downstream business application use can be realized. Similarly, in the application scenario of counting mobile phone sales, the dimension data includes the regional dimension, the sales time dimension, the mobile phone brand dimension, and the mobile phone model dimension. The data aggregation requirement indicates the specified dimension that needs to be aggregated, for example, the regional dimension, the sales time dimension, and the mobile phone brand dimension. The aggregated data is the mobile phone sales indicators corresponding to the regional dimension, the sales time dimension, and the mobile phone brand dimension, and a summary table of the last 7 days of the transaction domain and province granularity orders is constructed.

[0109] The summary results are stored in the first database to provide a data basis for downstream applications. For example, OLAP business, BI analysis business, etc. can query the data they need to query based on the summary results stored in the first database.

[0110] By obtaining the original business data and configuration information, the original business data is diverted based on the configuration information to obtain dimension data and fact data, which are stored in different databases respectively. After obtaining the data aggregation requirements, the fact data under the specified dimension can be calculated based on the data aggregation requirements to obtain the aggregation data. Diverting the original business data based on the configuration information can make the dimension data stored efficiently and avoid query delays.

[0111] In a possible implementation, the above-mentioned original business data is split based on the above-mentioned configuration information to obtain dimension data and fact data, including:

[0112] Step 1: Filter the above original business data to obtain mainstream data;

[0113] Filter the original business data. For example, you can determine whether each data in the original business data is in Json format. If it is, the data is regarded as mainstream data. If not, the data is regarded as dirty data and saved. Mainstream data refers to data required for data query, and dirty data refers to data irrelevant to data query. The division of mainstream data and dirty data can be customized according to the actual application scenario. For example, data in XML and File format is regarded as dirty data.

[0114] Step 2: parse the configuration information to obtain a configuration information broadcast stream, and set the configuration information broadcast stream to a broadcast state;

[0115] Flink is a framework and distributed processing engine for stateful computing of unbounded and bounded data streams. For example, Flink CDC (Change Data Capture) can be used to obtain submitted changes from the database and send these changes to the downstream for use by the downstream. In an example, the configuration information can be stored in MySQL, and FlinkCDC can be used to dynamically read the configuration information in MySQL, and the configuration information can be obtained by parsing the configuration data and verifying the configuration information write status. Then, the configuration information is generated into a broadcast stream type of configuration information broadcast stream based on the configuration information changes, and the configuration information broadcast stream is set to the broadcast state, and compared with the mainstream data, thereby realizing the separation of dimension and fact data. In an example, when the configuration information is first launched, it can be considered that all configuration information is full information; when the configuration information is subsequently modified, the broadcast stream data also changes in real time.

[0116] Step 3, connecting the above mainstream data and the above configuration information broadcast stream to obtain connection stream data;

[0117] The mainstream data and the configuration information broadcast stream are associated to obtain the connection stream data. Specifically, the MyBroadcastFunction function can be customized to implement the broadcast association.

[0118] Step 4: Based on the streaming computing program, the above connection stream data is split and processed to obtain dimension data and fact data.

[0119] Flink streaming computing programs can be used to divert mainstream data based on the configuration information broadcast stream to obtain dimension data and fact data, thereby realizing functions such as automated data flow, dynamic perception of table structure, and real-time data ETL.

[0120] The connection stream data includes mainstream data and configuration information broadcast streams. The configuration information broadcast stream may include data source information, data sink information, valid fields and other information. Among them, the valid fields can be used to filter the mainstream data fields. When there are "valid fields" in the mainstream data, the fields corresponding to the mainstream data are retained, otherwise the non-valid fields of the mainstream data are filtered out. The data source information represents the source of the data and the connection method, for example, specifying that the original data comes from different topics of different message queues; the data sink information represents the data flow direction, target source and docking method, for example, for each type of data in the original business data, the data is sent to which databases for storage, for example, the data is sent to Kafka, or to other databases, such as Hbase (an open source non-relational distributed database) database, KUDU database, etc. In one example, a data model may include dimensional data and factual data, where dimensional data refers to the associated data of user attribute information, and factual data refers to the actual actions that occur during the user's business operations. For example, in the application scenario of smart medical care, dimensional data may be consultation time, hospital name, consultation department, consultation doctor, equipment used, and drug information; factual data may be actual diagnosis information, registration information, purchase information, evaluation information, etc.; or in the application scenario of mobile phone sales, dimensional data may be mobile phone brand type, time, region, and factual data may be actual sales quantity, sales amount, etc.

[0121] Using FlinkCDC to dynamically monitor configuration information and associate it with Flink mainstream data in the form of a broadcast stream can ensure real-time capture of real-time updates of configuration information. Once the configuration information changes, the computing engine automatically perceives and adapts, improving the accuracy and consistency of configuration information during task processing, realizing automatic diversion of dimensional information and factual information, and improving the stability and responsiveness of the system.

[0122] In a possible implementation manner, filtering the original business data to obtain mainstream data includes:

[0123] For any original business data, determine whether the original business data is in a preset mainstream data format. If so, treat the original business data as mainstream data; if not, treat the original business data as dirty data.

[0124] The preset mainstream data format is set based on the actual situation, for example, the Json format. If the original business data is not in the preset mainstream data format, the data is written as dirty data to the side output stream, and the dirty data is persisted. By filtering the original business data based on the preset mainstream data format, the selected mainstream data can be data that conforms to the preset format, which can avoid calculating data that does not conform to the format, ensure the compliance and standardization of the data, avoid secondary pollution of dirty data caused by format errors, and improve the accuracy of data governance.

[0125] When collecting data produced by data producers, the data collection application will store the specified type of data (mainstream data) in a preset mainstream data format (such as Json format) according to pre-set rules, while other types of data (dirty data) are stored as logs in non-preset mainstream data formats. Mainstream data is the data required for data query. In an example, in a certain query scenario, time dimension data is related to data query, so the data collection application can save time dimension data such as year, month, day, season, hour, minute and second as logs in Json format. Dirty data is data that is not related to data query. For example, in a certain query scenario, it is believed that the operator company that the user accesses the network is irrelevant to the data query, so the data collection application can save the operator company that the user accesses the network as a log in a non-Json format.

[0126] Based on the above embodiments, Figure 2 As shown, Figure 2 A schematic diagram of the implementation process of the dynamic diversion mechanism provided in this application.

[0127] The original business data is stored in Kafka, so it is read from the business data ODS layer of Kafka;

[0128] Determine whether the original business data is in Json format. If so, treat it as mainstream data; if not, write the original business data as dirty data to the side output stream and persist the dirty data.

[0129] The configuration information can be a configuration information table. Flink CDC reads the configuration information table, parses the configuration information table, and verifies the writing status of the configuration information table, thereby obtaining the changes in the configuration information table. Then, the changes in the configuration information table are used to generate a configuration information broadcast stream of the broadcast stream type, and the broadcast stream processing data is obtained. The mainstream data is filtered based on the broadcast stream processing data to obtain the connection stream data. Flink shunts the mainstream data for the connection stream data to obtain dimension data and fact data. The dimension data is written into the dimension data layer (DIM). In an example, the way to write the dimension data into the dimension data layer (DIM) can be implemented based on the Open() and Invoke() programs, wherein Open() initializes the connection, and Invoke() performs a specific data combing process. For example, the dimension data is stored in the KUDU database, and the fact data is written into the fact data layer (DWD). In an example, the fact data is stored in the Kafka database. When querying data, the dimension data can be obtained from the KUDU database, and the dimension data can be queried through Impala (a new query system).

[0130] When Flink interacts with external systems, the default is single-parallelism synchronous IO (Input / Output). Its disadvantages are severe IO blocking, severe data request queuing, and a lot of time spent waiting for results.

[0131] In a possible implementation manner, the first database includes a cache database and a KUDU database;

[0132] The above preset data aggregation requirement obtains the metric value corresponding to the above specified dimension from the above fact data, and performs calculation to obtain the aggregated data, including:

[0133] According to the data aggregation requirement, when the fact data and the dimension data are associated, it is determined whether the preset cache database has the specified dimension corresponding to the above data aggregation requirement. If yes, an association request for the above specified dimension is sent to the above preset cache database, and the above specified dimension is obtained from the above preset cache database to aggregate the data to obtain the aggregated data;

[0134] If not, a correlation request of the specified dimension is sent to the first database based on the fact data in the first message system, and the measurement value of the specified dimension is calculated according to the data aggregation requirement to obtain the aggregated data.

[0135] The KUDU database is used to store summary data in various dimensions, because KUDU not only has high-performance random read and write performance, but also far exceeds Hbase scanning performance, saving space and being more suitable for OLAP efficient query. Its deep integration with Impala provides guarantee for reducing data latency, can reduce code coupling, and solve the problem of dynamically identifying table structure changes and table additions. Introducing KUDU as a multidimensional analysis tool can meet the high throughput requirements of the query platform, that is, query requirements.

[0136] The KUDU database is equivalent to a cold storage, which has network delays, data delays, and data transmission needs to be serialized and deserialized. The preset cache database can store the summary data of each dimension that has been obtained, wherein the preset cache database can be a Redis database. After obtaining the data summary requirements, the preset cache database is read first. If the preset cache database cannot be read, the data is obtained from the KUDU database according to the user number (Custno), thereby ensuring the timeliness and accuracy of the data through the pre-loaded cache method. Specifically, according to the data summary requirements, when the fact data and the dimension data are associated, it can be determined first whether the preset cache database has the specified dimension corresponding to the data summary requirements. If so, a request for associating the specified dimension is sent to the preset cache database, and then the specified dimension is obtained from the preset cache database for data aggregation to obtain the aggregated data. If not, a request for associating the specified dimension is sent to the first database based on the fact data in the first message system. According to the data aggregation requirements, the dimension data is obtained from the first database to calculate the measurement value of the specified dimension to obtain the aggregated data, and the dimension data obtained by the request is synchronized to the cache database. In order to avoid the unlimited expansion of the cache database, it is necessary to set a life cycle for it.

[0137] In one example, after obtaining the data aggregation requirements, based on the data aggregation requirements, the first computing engine can be used to calculate the dimensional data in the first database to obtain the calculation results of the dimensional data, and the second computing engine can be used to calculate the fact data in the first message system to obtain the calculation results of the fact data. Finally, the calculation results of the dimensional data and the calculation results of the fact data are associated to obtain the aggregation results, and a lightly aggregated table is generated and stored in the first database.

[0138] In addition, when using the first computing engine to calculate the dimension data information returned by the above KUDU database, you can call a third-party API interface (such as the AsyncIO interface) to asynchronously access the dimension data in KUDU and host it in a separate thread pool, thereby improving the task parallel throughput. Among them, AsyncDateStream is mainly used for asynchronous association, the RichAsyncFunction function is inherited, and the AysncInvoke method is rewritten for dimension association. For details, see Figure 3 , Figure 3 A schematic diagram of synchronous and asynchronous IO provided for this application, wherein the left part of the figure is a schematic diagram of single-parallelism synchronous IO. When Flink interacts with an external system, in order to avoid IO blocking caused by multi-table associations, the default is single-parallelism synchronous IO. After data requests a and b are sent to the KUDU database, they can only be processed in the order in which the requests are sent. That is to say, the KUDU database responds to data request a and returns the corresponding data a1 before it can respond to data request b and process data request b.

[0139] The asynchronous IO mechanism can obtain data requests a, b, c, d at the same time, call the API interface to asynchronously access the dimension data in KUDU, and host it to a separate thread pool to perform asynchronous processing on data requests a, b, c, d at the same time, and obtain the corresponding data a1, b1, c1, d1 of data requests a, b, c, d, thereby converting the original single-parallelism synchronous request into a single-parallelism asynchronous request, reducing query waiting and avoiding data loss.

[0140] By selecting asynchronous IO, multiple requests are processed in parallel in order, and the requests no longer have to wait in a blocked manner, thus improving real-time processing efficiency.

[0141] In a possible implementation, the above method further includes:

[0142] After obtaining dimension data from the first database to calculate the measurement value of the specified dimension and obtain the summary data, the specified dimension corresponding to the above data summary requirement is loaded from the first database to the preset cache database.

[0143] According to the above data aggregation requirements, the fact data under the above specified dimensions are calculated. After the aggregated data is obtained, the aggregated data corresponding to the above data aggregation requirements is stored in the preset cache database, so that the next time the data is read, it can be read directly from the preset cache database, thereby ensuring the timeliness and accuracy of the data.

[0144] Based on the above embodiments, see Figure 4 , Figure 4 A schematic diagram of the preload cache mechanism provided for this application.

[0145] The dimension data is written to the DIM layer, that is, the dimension data is stored in the KUDU database. After obtaining the data query request, the Redis cache is first queried based on the asynchronous IO mechanism to determine whether the data corresponding to the asynchronous request is hit in the Redis cache. If so, the calculation result is obtained from the Redis cache. If not, the data aggregation requirement is sent to the above-mentioned KUDU database and the above-mentioned first message system. According to the above-mentioned data aggregation requirement, the fact data under the above-mentioned specified dimension is calculated to obtain the aggregated data. And the aggregated data is synchronized to the Redis cache database. The dimension data layer uses KUDU+Impala instead of Hbase+Phoenix storage to achieve idempotent writing.

[0146] In order to achieve high-performance OLAP real-time analysis, a BI analysis platform is designed. Users customize pre-calculation content according to business specifications, mainly reading data from the summary data (DWS layer) of the first database, and pre-processing the dimension data and fact data. For example, after obtaining the dimension data and fact data, the dimension data can be aggregated to obtain dimension data under different dimension combinations, and then each dimension combination is stored in the first database respectively.

[0147] For example, there are currently three dimensions, X, Y, and Z. Aggregating the dimensional data can obtain dimensional combinations XY, XZ, YZ, X, Y, and Z. Then, XY, XZ, YZ, X, Y, and Z are respectively stored in different folders of the first database.

[0148] Exemplary: obtaining a preset data cube model, the data cube model includes a dimension information column and an indicator information column, and converting the dimension information column and the indicator information column into a logical table based on a preset Schema service; based on the logical table, cleaning, aggregating, grouping and filtering the dimensional data to obtain a pre-calculated result, and storing the pre-calculated result in a first database.

[0149] The user configures the data cube model on the BI analysis platform, such as Figure 5-1 As shown, the dimensional data includes mobile phone brand type (mobile phone brand 1-3), time (2021-2023) and regional dimensions (region 1, region 2, region 3). The indicator information is specific sales data. After configuring the Cube, you can count the sales data of each data brand in the corresponding time and region.

[0150] The construction indicator calculation process sets up pre-loaded statistical results for users, uses the space-for-time method to store the calculated results into a Cube and GroupingSet dimension combination, outputs UV (unique visitor number), new and retained indicators from multiple angles such as time, geography, and category, and finally configures the output of results to the relevant analysis system at regular intervals.

[0151] In a possible implementation, storing the dimensional data in the first database and storing the factual data in the first message system includes:

[0152] Convert the above dimension data into dimension logic table, and convert the counted fact data into fact logic table;

[0153] The dimension logic table is stored in the first database, and the fact logic table is stored in the first message system.

[0154] According to dimensional modeling theory, the fact table is designed only around the business process. The fact table includes the finest granularity and measurement values. The dimension table is designed around the environment in which the business process is located. The dimension table includes the primary key and various dimension fields.

[0155] The design process of the fact logic table is: declare the business → declare the granularity → determine the dimension → determine the fact. Take the transaction business as an example: placing an order is the business process, each row of data in the order fact logic table is the granularity, the region table or user table is the dimension, and the data with measurement values ​​(number of times, number of items, quantity, amount...) is the fact.

[0156] The design process of dimension logic table is: determine the dimension → determine the main dimension table and related dimension tables → determine the dimension attributes. Take the product dimension as an example: the product dimension logic table is the main dimension table, and the first-level category, second-level category and other product-related dimension tables are weakly normalized.

[0157] The dimension data is converted into a dimension logic table, the fact data is converted into a fact logic table, and then the dimension logic table is stored in the first database, and the fact logic table is stored in the first message system.

[0158] In a possible implementation, after obtaining the metric value corresponding to the specified dimension from the fact data according to the data aggregation requirement, performing calculations to obtain aggregated data, and storing the aggregated data in the first database, the method further includes:

[0159] Obtaining a data cube model pre-built by a user through a preset BI analysis platform; the pre-built data cube model includes a combination of dimensions that need to be aggregated;

[0160] For each dimension combination, aggregate the dimension combination to obtain the metric value corresponding to the dimension combination;

[0161] For each dimension combination, the dimension combination and the metric value corresponding to the dimension combination are stored in the folder corresponding to the dimension combination in the first database in the form of a data cube model.

[0162] The user connects to the first database through the BI analysis platform to build a data cube model on the aggregated data, pre-calculates the measurement values ​​that may be obtained from the multidimensional analysis, returns the calculation results to the first database for storage, and returns the calculation results to the BI platform.

[0163] The design of the BI analysis platform is essentially a cache system, which requires users to define pre-calculated content in advance according to business rules. The pre-calculated content can be in the form of a data cube model. The data cube model includes dimension information and metric information that need to be aggregated. Specifically, the data cube model can include aggregated dimension columns and metric columns.

[0164] After obtaining the data cube model pre-built by the user through the preset BI analysis platform, the results of various queries in the first database can be calculated first, and the calculated results can be saved as a data cube model and stored in the first database for subsequent query use, so as to realize the conversion of highly complex aggregation operations such as multi-table association into pre-calculated query results. The specific implementation process is as follows: determine the metric value, determine the dimension value, build the data cube to return the result, and query the result.

[0165] like Figure 5-2 As shown in the figure, the query results are obtained through full-dimensional calculation and dimensionality reduction calculation. The full-dimensional calculation is divided into two processes. First, the dimensions are scattered for pre-aggregation, and the same dimensions are hashed and dispersed. In the case where UV indicators need to be deduplicated, a deduplication mode is designed for deduplication, and then the deduplication results of the data are dimensionally aggregated. In the dimensionality reduction calculation, the return value of the full-dimensional pre-calculation is fully pre-aggregated, and the final result is output to KUDU.

[0166] Suppose there are four dimensions A, B, C, and D. First, the dimensions are scattered and pre-aggregated to obtain dimension combinations under different dimensions, that is, the following combinations are obtained: (A, B, C), (A, B, D), (A, C, D), (B, C, D), (A, B), (A, C), (A, D), (B, C), (B, D), (C, D), (A), (B), (C), (D), and then the same dimensions are hashed and dispersed. Finally, different dimension combinations are stored in different folders of the first database. In an example, each dimension combination corresponds to a folder.

[0167] If there are duplicate UV indicators, the UV indicators need to be deduplicated. In this case, Flink is used to design a window function every minute to perform deduplication, and then the deduplication results are output (SINK) to the next step for dimensional aggregation.

[0168] In a possible implementation manner, storing the dimension logic table under the dimension combination in a folder corresponding to the dimension combination in the first database includes:

[0169] For each dimension combination, determine a first folder corresponding to the dimension combination in the first database, and determine a first storage block to which the first folder belongs;

[0170] Obtaining a local preset deduplication mode corresponding to the first storage block, and using the local preset deduplication mode corresponding to the first storage block to deduplicate the dimension logic table under the dimension combination, to obtain a deduplication logic table under the dimension combination;

[0171] The deduplication logic table under the dimension combination is stored in the first folder.

[0172] For example, when counting the information of each user's login to an account, if there are consecutive logins within a time range, a deduplication rule can be set to retain only the first login information and remove other login information. That is, it is considered that within a time range, except for the first login, all others are repeated logins.

[0173] Alternatively, if user A's daily itinerary needs to be analyzed, when user A enters and exits a scenic spot multiple times a day, only the data of entering the scenic spot once is recorded. The user's itinerary can be deduplicated based on the user's ID, the scenic spot's ID, and the time dimension.

[0174] For different users, you can set their corresponding deduplication modes for deduplication. For example, the deduplication mode corresponding to the first user is a, and the deduplication mode corresponding to the second user is b. Different deduplication modes correspond to different user IDs. For each user, after obtaining the user's data, you can determine the corresponding deduplication mode based on the user ID of the user, and then deduplicate the data based on the deduplication mode.

[0175] Specifically, for each dimension combination, the first folder corresponding to the dimension combination in the first database is determined, and the first storage block to which the first folder belongs is determined, the local preset deduplication mode corresponding to the first storage block is obtained, and the dimension logic table under the dimension combination is deduplicated using the local preset deduplication mode corresponding to the first storage block to obtain the deduplication logic table under the dimension combination, and the deduplication logic table under the dimension combination is stored in the first folder. That is, in the dimensionality reduction calculation, the full-dimensional pre-calculated return value is fully pre-aggregated, and the final result is output to the KUDU database.

[0176] In a possible implementation, for each dimension combination, the dimension logic table of the dimension combination is aggregated to obtain the dimension logic table under the dimension combination, including:

[0177] Obtain each maximum subset of the dimension set including each dimension, and obtain multiple dimension combinations of the current level;

[0178] For each dimension combination at the current level, the dimension logic table of the dimension combination is aggregated to obtain the dimension logic table under the dimension combination;

[0179] For each dimension combination of the current level, obtain each maximum subset of the dimension combination of the current level, and obtain the dimension combination of the next level of the current level;

[0180] Take the next level dimension combination as the current level dimension combination, and return to the execution step: for each current level dimension combination, aggregate the dimension logic table of the dimension combination to obtain the dimension logic table under the dimension combination, until there are no subsets for each current level dimension combination, and obtain the dimension logic table under each dimension combination.

[0181] For example, if the dimension set is (A, B, C, D), then multiple dimension combinations at the current level are (A, B, C), (A, B, D), (A, C, D), (B, C, D); for (A, B, C), (A, B, D), (A, C, D), (B, C, D), the dimension logic table of the dimension combination can be aggregated to obtain the dimension logic table under the dimension combination, and then (A, B), (A, C), (A, D), (B, C), (B, D)(C, D) can be used as the dimension combination of the current level and the execution is repeated until there is no subset for the dimension combinations of the current level, that is, (A), (B), (C), (D), and the dimension logic table under each dimension combination is obtained.

[0182] In a possible implementation, for each current-level dimension combination, the dimension logic table of the dimension combination is aggregated to obtain the dimension logic table under the dimension combination, including:

[0183] For each dimension combination at the current level, based on the deduplication logic table at the previous level of the dimension combination at the current level, the dimension logic table for the dimension combination at the current level is aggregated to obtain the dimension logic table under the dimension combination.

[0184] In the case where UV indicators need to be deduplicated, the CureMap mode is designed in the embodiment of the present application for deduplication, and a window function is designed every minute using Flink to perform deduplication. Then the deduplicated data of the current level dimension combination is Sinked to the next level dimension combination for dimensional aggregation.

[0185] In addition, to avoid the problem of data skew, a state function can be introduced, and CureMapState can be customized to store the deduplication pattern (CureMap) in blocks. A blockid (block identifier) ​​corresponds to a local CureMap, which in turn corresponds to the spatial partition in the KUDU database to solve the problem of data skew.

[0186] Finally, the results are stored in the KUDU database, the dimensions are encoded, and time + dimension combination + dimension value combination is used as the primary key. Finally, partitioning is performed by dimension combination, dimension value combination, and time to obtain high performance and improve online query.

[0187] Users only need to configure the logical table and the real-time data warehouse will clean, aggregate, group and filter the data.

[0188] Based on the above embodiments, see Figure 5-3 , the user configures the data cube model on the BI (Business Intelligence) analysis platform, where the cube includes dimension information columns and indicator information columns. After the cube is configured, a metadata transfer station is configured in the Flink computing platform, which provides a unified Schema (a collection of database objects) service. The Schema service is used to abstract the dimension information columns and indicator information columns into logical tables, including Kafka topics, Redis cache database metadata, etc. In this way, the user only needs to configure the logical table and then the real-time data warehouse will clean, aggregate, group and filter the data.

[0189] After Flink pre-calculation (dimensionality reduction calculation) and modeling index calculation, it is stored in the KUDU database and the results are fed back to the BI platform for visualization. This meets the interactive query requirements for low latency, millisecond-level response, high concurrency, and strong grouping and aggregation capabilities. In other words, the introduction of the data cube preprocessing mechanism can meet the requirements of high concurrency and low latency for multi-dimensional data analysis in real-time scenarios.

[0190] The dimension data layer uses KUDU+Impala instead of Hbase+Phoenix storage to achieve idempotent writing. Compared with Hbase, KUDU not only has high-performance random read and write performance, but also far exceeds Hbase scanning performance. It saves space and is more suitable for efficient OLAP queries. Its deep integration with Impala provides a guarantee for reducing data latency. In other words, because the commonly used OLAP tools on the market (Druid, Clinkhouse) have their own limitations, such as: Clickhouse has poor indexing capabilities and poor table association capabilities, and cannot support high-concurrency queries; Druid's grouping and aggregation capabilities are average, and functions such as count distinst are not supported. The Flink pre-computation processing + Kudu storage solution in this application can meet the interactive query requirements for low latency, millisecond-level response, high concurrency, and strong grouping and aggregation capabilities.

[0191] Configure pre-calculated storage based on Kudu to achieve high-performance OLAP real-time analysis.

[0192] Based on the above embodiments, Figure 6 As shown, Figure 6 An interactive schematic diagram of a data processing method provided for the implementation of the present application.

[0193] The original business data is stored in the ODS layer (original data layer), illustratively, in Kafka, and the original business data is read from the business data ODS layer of Kafka;

[0194] Configuration information (configuration table) is stored in Mysql. The administrator maintains the configuration information. Then Flink CDC reads the configuration information, parses the configuration information, and verifies the configuration information writing status, so as to obtain the change of the configuration information. Then the change of the configuration information is generated into a broadcast stream. Flink diverts the original business data according to the configuration information to obtain dimension data and fact data, wherein the dimension data is written into the DIM layer and the fact data is written into the DWD layer. Specifically, the dimension data is stored in the KUDU database and the fact data is stored in Kafka. In addition, an independent cache database is set. Exemplarily, the independent cache database is Redis. The data aggregation requirement is preset. According to the above data aggregation requirement, the fact data under the above specified dimension is calculated to obtain the aggregation data:

[0195] According to the data aggregation requirement, determine whether the preset cache database has the aggregation data corresponding to the above data aggregation requirement, and if so, send the data aggregation requirement to the above preset cache database, and obtain the aggregation data of the above data aggregation requirement from the above preset cache database;

[0196] If not, a data aggregation requirement is sent to the KUDU database and the first message system, and the fact data under the specified dimension is calculated according to the data aggregation requirement to obtain the aggregated data.

[0197] The summary results are written to the DWS (Summary Data Layer) layer, and the analysis results are finally sent to the ADS (Application Data Layer), for example, a WEB interface or a BI analysis tool.

[0198] like Figure 7 As shown, Figure 7 A schematic diagram of a data processing device provided in an embodiment of the present application, the device comprising:

[0199] The acquisition module 710 is used to acquire original business data and configuration information;

[0200] A distribution module 720 is used to distribute the original business data based on the configuration information to obtain dimension data and fact data;

[0201] A storage module 730 is used to store the dimension data in a first database and store the fact data in a first message system;

[0202] The calculation module 740 is used to obtain data aggregation requirements; the data aggregation requirements represent the specified dimension that needs to be aggregated and the metric value corresponding to the specified dimension; the specified dimension belongs to the dimension in the dimensional data; the metric value corresponding to the specified dimension is calculated based on the fact data;

[0203] The analysis module 750 is used to obtain the measurement value corresponding to the above-mentioned specified dimension from the above-mentioned fact data, perform calculations to obtain summary data, and store the above-mentioned summary data in the above-mentioned first database.

[0204] In a possible implementation manner, the above-mentioned flow distribution module 720 is specifically used for:

[0205] Filter the above raw business data to obtain mainstream data;

[0206] Parsing the configuration information to obtain a configuration information broadcast stream, and setting the configuration information broadcast stream to a broadcast state;

[0207] Connecting the mainstream data and the configuration information broadcast stream to obtain connection stream data;

[0208] Based on the streaming computing program, the above connection stream data is split and processed to obtain dimension data and fact data.

[0209] In a possible implementation manner, the above-mentioned flow distribution module 720 is specifically used for:

[0210] For any original business data, determine whether the original business data is in a preset mainstream data format. If so, treat the original business data as mainstream data; if not, treat the original business data as dirty data.

[0211] In a possible implementation, the first database is a database that supports random reading and writing and real-time analysis, and the first message system is a message system that supports a distributed subscription function.

[0212] In a possible implementation manner, the first database includes a cache database and a KUDU database;

[0213] The calculation module 740 is specifically used for:

[0214] According to the data aggregation requirement, when the fact data and the dimension data are associated, it is determined whether the preset cache database has the specified dimension corresponding to the above data aggregation requirement. If yes, an association request for the above specified dimension is sent to the above preset cache database, and the above specified dimension is obtained from the above preset cache database to aggregate the data to obtain the aggregated data;

[0215] If not, a correlation request of the specified dimension is sent to the first database based on the fact data in the first message system, and the measurement value of the specified dimension is calculated according to the data aggregation requirement to obtain the aggregated data.

[0216] In a possible implementation manner, the above device further includes:

[0217] The cache module is used to obtain the measurement value corresponding to the above-mentioned specified dimension from the above-mentioned fact data according to the above-mentioned data aggregation requirements, and calculate, after obtaining the summarized data, load the specified dimension corresponding to the above-mentioned data aggregation requirements from the first database to the preset cache database.

[0218] In a possible implementation manner, the storage module 730 is specifically used for:

[0219] Convert the above dimension data into dimension logic table, and convert the counted fact data into fact logic table;

[0220] The dimension logic table is stored in the first database, and the fact logic table is stored in the first message system.

[0221] In a possible implementation manner, the above device further includes:

[0222] A preprocessing module is used to obtain a data cube model pre-built by a user through a preset BI analysis platform; the pre-built data cube model includes dimension combinations that need to be aggregated; for each dimension combination, the dimension combination is aggregated to obtain a measurement value corresponding to the dimension combination; for each dimension combination, the dimension combination and the measurement value corresponding to the dimension combination are stored in the form of a data cube model in a folder corresponding to the dimension combination in the first database.

[0223] In a possible implementation manner, the storage module 730 is specifically used for:

[0224] For each dimension combination, determine the first folder corresponding to the dimension combination in the first database, and determine the first storage block to which the above-mentioned first folder belongs; obtain the local preset deduplication mode corresponding to the above-mentioned first storage block, and use the local preset deduplication mode corresponding to the above-mentioned first storage block to deduplicate the dimension logic table under the dimension combination to obtain the deduplication logic table under the dimension combination; store the deduplication logic table under the dimension combination in the above-mentioned first folder.

[0225] In a possible implementation manner, the storage module 730 is specifically used for:

[0226] Obtain each maximum subset of the dimension set including each dimension, and obtain multiple dimension combinations of the current level;

[0227] For each dimension combination at the current level, the dimension logic table of the dimension combination is aggregated to obtain the dimension logic table under the dimension combination;

[0228] For each dimension combination of the current level, obtain each maximum subset of the dimension combination of the current level, and obtain the dimension combination of the next level of the current level;

[0229] Take the next level dimension combination as the current level dimension combination, and return to the execution step: for each current level dimension combination, aggregate the dimension logic table of the dimension combination to obtain the dimension logic table under the dimension combination, until there are no subsets for each current level dimension combination, and obtain the dimension logic table under each dimension combination.

[0230] In a possible implementation manner, the storage module 730 is specifically used for:

[0231] For each dimension combination at the current level, based on the deduplication logic table at the previous level of the dimension combination at the current level, the dimension logic table for the dimension combination at the current level is aggregated to obtain the dimension logic table under the dimension combination.

[0232] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0233] The present application also provides an electronic device, such as Figure 8 As shown, it includes a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804.

[0234] Memory 803, used for storing computer programs;

[0235] The processor 801 is used to execute the program stored in the memory 803, and implements the following steps:

[0236] Obtain original business data and configuration information;

[0237] Based on the configuration information, the original business data is divided to obtain dimension data and fact data;

[0238] The dimension data is stored in a first database, and the fact data is stored in a first message system;

[0239] Obtain data aggregation requirements; the data aggregation requirements indicate the specified dimension to be aggregated and the metric value corresponding to the specified dimension; the specified dimension belongs to the dimension in the dimensional data; the metric value corresponding to the specified dimension is calculated based on the fact data;

[0240] According to the above data aggregation requirements, the metric value corresponding to the above specified dimension is obtained from the above fact data, and is calculated to obtain the aggregated data, and the above aggregated data is stored in the above first database.

[0241] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0242] The communication interface is used for communication between the above electronic device and other devices.

[0243] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0244] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0245] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data processing methods are implemented.

[0246] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the data processing methods in the above embodiments.

[0247] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0248] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0249] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0250] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain original business data and configuration information; Divide the original business data based on the configuration information to obtain dimension data and fact data; storing the dimension data in a first database, and storing the fact data in a first message system; Acquire data aggregation requirements; the data aggregation requirements represent the specified dimension that needs to be aggregated and the metric value corresponding to the specified dimension; the specified dimension belongs to the dimension in the dimensional data; the metric value corresponding to the specified dimension is calculated based on the fact data; According to the data aggregation requirement, the metric value corresponding to the specified dimension is obtained from the fact data, and is calculated to obtain the aggregated data, and the aggregated data is stored in the first database.

2. The method according to claim 1, characterized in that The dividing the original business data based on the configuration information to obtain dimension data and fact data includes: Filtering the original business data to obtain mainstream data; Parsing the configuration information to obtain a configuration information broadcast stream, and setting the configuration information broadcast stream to a broadcast state; Connecting the mainstream data and the configuration information broadcast stream to obtain connected stream data; The connection stream data is split and processed based on a streaming computing program to obtain dimension data and fact data.

3. The method according to claim 2, characterized in that The filtering of the original business data to obtain mainstream data includes: For any original business data, determine whether the original business data is in a preset mainstream data format. If so, treat the original business data as mainstream data; if not, treat the original business data as dirty data.

4. The method according to claim 1, characterized in that The first database is a database that supports random reading and writing and real-time analysis, and the first message system is a message system that supports distributed subscription functions.

5. The method according to claim 4, characterized in that The first database includes a cache database and a KUDU database; The step of obtaining the metric value corresponding to the specified dimension from the fact data according to the data aggregation requirement and performing calculation to obtain the aggregated data includes: According to the data aggregation requirement, when the fact data and the dimension data are associated, it is determined whether the preset cache database has the specified dimension corresponding to the data aggregation requirement. If so, an association request for the specified dimension is sent to the preset cache database, and the specified dimension is obtained from the preset cache database to aggregate the data to obtain the aggregated data; If not, a correlation request of the specified dimension is sent to the first database based on the fact data in the first message system, and the metric value of the specified dimension is calculated according to the data aggregation requirement to obtain the aggregated data.

6. The method according to claim 5, characterized in that The method further comprises: According to the data aggregation requirement, the metric value corresponding to the specified dimension is obtained from the fact data, and after calculation is performed to obtain the aggregated data, the specified dimension corresponding to the data aggregation requirement is loaded from the first database to the preset cache database.

7. The method according to claim 1, characterized in that The storing the dimension data into the first database and the storing the fact data into the first message system comprises: Convert the dimension data into a dimension logic table, and convert the counted fact data into a fact logic table; The dimension logic table is stored in a first database, and the fact logic table is stored in a first message system.

8. The method according to claim 1, characterized in that After obtaining the metric value corresponding to the specified dimension from the fact data according to the data aggregation requirement, performing calculations to obtain the aggregated data, and storing the aggregated data in the first database, the method further includes: Obtaining a data cube model pre-built by a user through a preset BI analysis platform; the pre-built data cube model includes a combination of dimensions that need to be aggregated; For each dimension combination, aggregate the dimension combination to obtain the metric value corresponding to the dimension combination; For each dimension combination, the dimension combination and the metric value corresponding to the dimension combination are stored in the folder corresponding to the dimension combination in the first database in the form of a data cube model.

9. The method according to claim 8, characterized in that The method further comprises: For each dimension combination, determine the first folder corresponding to the dimension combination in the first database, and determine the first storage block to which the first folder belongs; obtain the local preset deduplication mode corresponding to the first storage block, and use the local preset deduplication mode corresponding to the first storage block to deduplicate the dimension logic table under the dimension combination to obtain the deduplication logic table under the dimension combination; store the deduplication logic table under the dimension combination in the first folder.

10. The method according to claim 9, characterized in that For each dimension combination, the dimension combination is aggregated to obtain a dimension logic table under the dimension combination, including: Obtain each maximum subset of the dimension set including each dimension, and obtain multiple dimension combinations of the current level; For each dimension combination at the current level, the dimension logic table of the dimension combination is aggregated to obtain the dimension logic table under the dimension combination; For each dimension combination of the current level, obtain each maximum subset of the dimension combination of the current level, and obtain the dimension combination of the next level of the current level; Take the next level dimension combination as the current level dimension combination, and return to the execution step: for each current level dimension combination, aggregate the dimension logic table of the dimension combination to obtain the dimension logic table under the dimension combination, until there are no subsets for each current level dimension combination, and obtain the dimension logic table under each dimension combination.

11. The method according to claim 10, characterized in that For each dimension combination at the current level, the dimension logic table of the dimension combination is aggregated to obtain the dimension logic table under the dimension combination, including: For each dimension combination at the current level, based on the deduplication logic table at the previous level of the dimension combination at the current level, the dimension logic table for the dimension combination at the current level is aggregated to obtain the dimension logic table under the dimension combination.

12. A data processing device, characterized in that: The device comprises: Acquisition module, used to obtain original business data and configuration information; A distribution module, used to distribute the original business data based on the configuration information to obtain dimension data and fact data; A storage module, used for storing the dimension data in a first database and storing the fact data in a first message system; A calculation module, used for obtaining data aggregation requirements; the data aggregation requirements represent the specified dimension to be aggregated and the metric value corresponding to the specified dimension; the specified dimension belongs to the dimension in the dimensional data; the metric value corresponding to the specified dimension is calculated based on the fact data; The analysis module is used to obtain the measurement value corresponding to the specified dimension from the fact data, perform calculations to obtain summary data, and store the summary data in the first database.

13. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1-11 when executing a program stored in a memory.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 11 are implemented.

Images