A real-time processing method for indicator data and big data cloud platform

Abstract

The present application discloses a real-time processing method for indicator data and a big data cloud platform. The big data cloud platform includes at least a data acquisition component, a stream computing component and an indicator factory. The processing method should be applied to the big data cloud platform, including: the data acquisition component acquires real-time data; the stream computing component pre-processes the real-time data to obtain a fact table; the fact table stores the data to be processed obtained after pre-processing the real-time data; the indicator factory associates the fact table with the dimension table based on the information of the configured statistical dimensions and metric fields to obtain associated data; the associated data includes the data stored in the fact table and the dimension table; the information of the statistical dimensions and metric fields is configured through the indicator factory; the indicator factory passes the associated data to the stream computing component; the stream computing component summarizes and calculates the associated data based on the indicator definition information configured by the indicator factory to obtain key indicators.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a real-time processing method for indicator data and a big data cloud platform. Background Art

[0002] With the rapid development of the economy, the banking industry has increasingly higher requirements for the timeliness of obtaining bank product indicator data.

[0003] With the promotion and application of emerging tools and technologies such as Storm, Spark, and Apache, banks have also begun to use these emerging tools and technologies to automate data processing, thereby gradually realizing the acquisition of real-time data on asset changes for simple banking products.

[0004] However, for the calculation process of key performance indicator data of complex banking products, since it involves more business components and pending data, it is necessary to search and obtain the proxy pending data, integrate it, and then perform statistical calculations. Therefore, using the existing method, the periodicity of obtaining these data is relatively long, and the timeliness of the indicators cannot be effectively guaranteed. Summary of the invention

[0005] Based on the above-mentioned deficiencies of the prior art, the present application provides a real-time processing method for indicator data and a big data cloud platform to solve the problem that the prior art cannot effectively guarantee the timeliness of indicators.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] In a first aspect, the present application provides a real-time processing method for indicator data, which is applied to a big data cloud platform, wherein the big data cloud platform at least includes a data acquisition component, a stream computing component, and an indicator factory, and the data processing method includes:

[0008] The data acquisition component acquires real-time data;

[0009] The stream computing component preprocesses the real-time data to obtain a fact table; wherein the fact table stores the data to be processed obtained after preprocessing the real-time data;

[0010] The indicator factory associates the fact table with the dimension table based on the information of the configured statistical dimensions and metric fields to obtain associated data; wherein the associated data includes the data stored in the fact table and the dimension table; the information of the statistical dimensions and metric fields is configured by the indicator factory;

[0011] The indicator factory passes the associated data to the stream computing component;

[0012] The stream computing component aggregates and calculates the associated data based on the indicator definition information configured by the indicator factory to obtain key indicators.

[0013] Optionally, in the method provided above, the data acquisition component acquires real-time data, including:

[0014] The data collection component collects the real-time message data provided by the event-driven architecture into Kafka; wherein the real-time data includes real-time flow data and real-time dimension data; the dimension table includes a preloaded dimension table and a real-time dimension table storing the real-time dimension data;

[0015] The stream computing component preprocesses the real-time data to obtain a fact table, including:

[0016] The stream computing component preprocesses the real-time stream data to obtain a fact table.

[0017] Optionally, in the method provided above, the big data cloud platform further includes an MPP database and an Hbase database, and the indicator factory associates the fact table with the dimension table based on the information of the configured statistical dimensions and metric fields, and before obtaining the associated data, further includes:

[0018] The MPP database synchronizes the preloaded dimension table in the Greenplum database, and synchronizes the preloaded dimension table to the Hbase database;

[0019] The stream computing component preprocesses the real-time dimension data to obtain the real-time dimension table, and stores the implementation dimension table in the Hbase database.

[0020] Optionally, in the method provided above, the stream computing component preprocesses the data to be processed to obtain a fact table, including:

[0021] The stream computing component parses the data to be processed into tabular data, and extracts data of specified fields from the tabular data to form a fact table.

[0022] Optionally, in the method provided above, the indicator factory associates the fact table with the dimension table to obtain associated data, including:

[0023] Encapsulating the dimension table into a first container, and encapsulating the associated field information into a second container; wherein the associated field information is stored in both the fact table and the dimension table;

[0024] By associating the first container with the second container, data of multiple fields are extracted from multiple target data in the fact table and the dimension table to form associated data; wherein the target data refers to data containing the associated field information.

[0025] Optionally, in the method provided above, before the stream computing component aggregates and calculates the associated data based on the indicator definition information configured by the indicator factory to obtain the key indicator, it also includes:

[0026] The indicator factory passes the data source information, target table information, operator information and job configuration information to the stream computing component by calling the application program interface;

[0027] The stream computing component forms a stream computing job based on the data source information, the target table information, the operator information and the job configuration information; wherein, when the stream computing job is started, it executes the indicator definition information configured based on the indicator factory, summarizes and calculates the related data, and obtains key indicators.

[0028] Optionally, in the method provided above, after the stream computing component aggregates and calculates the associated data based on the indicator definition information configured by the indicator factory to obtain the key indicator, the method further includes:

[0029] The stream computing component stores the key indicators in the database according to the mapping relationship of the target table fields defined by the indicator factory for display by the front end.

[0030] The second aspect of the present application provides a big data cloud platform, including:

[0031] Data collection component, used to collect real-time data;

[0032] The indicator factory is used to configure the information of the statistical dimension and the metric field, and to associate the fact table with the dimension table based on the configured statistical dimension and the metric field information to obtain associated data, and pass the associated data to the stream computing component; wherein the associated data includes the data stored in the fact table and the dimension table;

[0033] A stream computing component is used to preprocess the real-time data to obtain a fact table, and to summarize and calculate the related data based on the indicator definition information configured by the indicator factory to obtain key indicators; wherein the fact table stores the data to be processed obtained after preprocessing the real-time data.

[0034] Optionally, in the big data cloud platform provided above, when the data acquisition component collects real-time data, it is used to:

[0035] The real-time message data provided by the event-driven architecture is collected into Kafka; wherein the real-time data includes real-time flow data and real-time dimension data; the dimension table includes a preloaded dimension table and a real-time dimension table storing the real-time dimension data;

[0036] The stream computing component preprocesses the real-time data to obtain a fact table, which is used to:

[0037] The real-time flow data is preprocessed to obtain a fact table.

[0038] Optionally, the big data cloud platform provided above further includes: an MPP database and an Hbase database;

[0039] The MPP database is used to synchronize the preloaded dimension table in the Greenplum database, and synchronize the preloaded dimension table to the Hbase database;

[0040] The Hbase database is used to store the real-time dimension table obtained by preprocessing the real-time dimension data by the stream computing component, and to store the preloaded dimension table synchronized with the MPP database.

[0041] Optionally, in the big data cloud platform provided above, when the stream computing component pre-processes the data to be processed to obtain a fact table, it is used to:

[0042] The data to be processed is parsed into table data, and data of designated fields are extracted from the table data to form a fact table.

[0043] Optionally, in the big data cloud platform provided above, the indicator factory associates the fact table with the dimension table to obtain associated data, which is used to:

[0044] Encapsulating the dimension table into a first container, and encapsulating the associated field information into a second container; wherein the associated field information is stored in both the fact table and the dimension table;

[0045] By associating the first container with the second container, data of multiple fields are extracted from multiple target data in the fact table and the dimension table to form associated data; wherein the target data refers to data containing the associated field information.

[0046] Optionally, in the big data cloud platform provided above, the indicator factory is also used for:

[0047] The data source information, target table information, operator information and job configuration information are passed to the stream computing component by calling the application program interface.

[0048] Optionally, in the big data cloud platform provided above, the stream computing component is further used to:

[0049] Based on the data source information, the target table information, the operator information and the job configuration information, a stream computing job is formed; wherein, when the stream computing job is started, the indicator definition information configured based on the indicator factory is executed, and the related data is summarized and calculated to obtain key indicators.

[0050] Optionally, in the big data cloud platform provided above, the stream computing component is further used to:

[0051] According to the mapping relationship of the target table fields defined by the indicator factory, the key indicators are stored in the database for display by the front end.

[0052] The present application provides a real-time processing method for indicator data, which is applied to a big data cloud platform, wherein the big data cloud platform at least includes a data acquisition component, a stream computing component, and an indicator factory. The data processing method provided in the embodiment of the present application collects real-time data through a data acquisition component, and the stream computing component pre-processes the real-time data to obtain a fact table storing the data to be processed. In addition, the statistical dimensions and metric fields are configured through the indicator factory, so that the indicator factory associates the fact table with the dimension table based on the information of the configured statistical dimensions and metric fields to obtain associated data, thereby realizing rapid integration of data. Finally, the indicator factory passes the associated data to the stream computing component, and the stream computing component summarizes and calculates the associated data based on the indicator definition information configured by the indicator factory to obtain key indicators. Thus, the fast computing characteristics of the stream computing component are utilized, and the work of the stream computing component that simplifies the information configuration through the indicator factory can effectively provide data processing efficiency, thereby ensuring the timeliness of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0054] Figure 1 A flowchart of a method for real-time processing of indicator data provided in an embodiment of the present application;

[0055] Figure 2 A flowchart of a table association method provided in another embodiment of the present application;

[0056] Figure 3 A flowchart of another method for real-time processing of indicator data provided by another embodiment of the present application;

[0057] Figure 4 A schematic diagram of the structure of a big data cloud platform provided in another embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0060] The embodiment of the present application provides a real-time processing method for indicator data, which is applied to a big data cloud platform. The big data cloud platform includes at least a collection component, a stream computing component, and an indicator factory. Figure 1 As shown, the real-time processing method of indicator data provided by the embodiment of the present application includes the following steps:

[0061] S101, the data acquisition component collects real-time data.

[0062] Optionally, the collected real-time data can be stored in Kafka to achieve data integrity. Kafka is an open source stream processing platform developed by the Apache Software Foundation.

[0063] Optionally, an event-driven architecture (EDA) technology may be used to implement real-time data generated by monitoring product components, and then provide the generated real-time data to a data acquisition component, and the data acquisition component stores the real-time data in Kafka.

[0064] S102: The stream computing component preprocesses the real-time data to obtain a fact table; the fact table stores the data to be processed obtained after the real-time data is preprocessed.

[0065] It should be noted that, since only part of the real-time data may need to be processed, the data needs to be extracted. In addition, there may be data with inconsistent formats in the real-time data, or the current format cannot be processed. The real-time data needs to be formatted. Taking the authorization flow topic as an example, the 13- or 20-digit timestamp field needs to be converted to a standard timestamp acceptable to the stream computing component for subsequent use. Therefore, due to the existence of the above problems, the real-time data needs to be preprocessed first. Preprocessing can specifically include format conversion, data cleaning, data extraction, etc. After processing the real-time data, the data obtained for subsequent processing is used as the data to be processed.

[0066] Optionally, in another embodiment of the present application, a specific implementation of step S102 includes:

[0067] The stream computing component parses the data to be processed into tabular data and extracts the data of the specified fields from the tabular data to form a fact table.

[0068] It should be noted that, since the data is processed in the form of a table in the embodiment of the present application, and only part of the data needs to be processed, it is necessary to parse the data to be processed into table data, and extract the data of the specified field from the table data to form a fact table. For example, if the real-time data is data in the json format provided by EDA, it is necessary to parse the real-time data in the json format into table data, that is, to parse it into data in the table format. Then, the data in the fields required for processing can be extracted from the table data, for example, the installment amount, the installment type and other fields.

[0069] Optionally, the specified fields can be configured through the indicator factory. In addition, the resulting fact table can be stored in Kafka first and then extracted from it when needed later.

[0070] S103. The indicator factory associates the fact table with the dimension table based on the configured statistical dimensions and metric field information to obtain associated data.

[0071] The associated data includes the data stored in the fact table and the dimension table. The information of the statistical dimension and the metric field is configured through the indicator factory. The specific association method can be defined in a customized way.

[0072] It should be noted that in the embodiment of the present application, the statistical dimensions and metric field information of both the dimension table and the fact table are first configured through the indicator factory. Through the information of the statistical dimensions and metric fields, it can be determined which fields in the dimension table and the fact table need to be statistically analyzed and with which statistical dimensions.

[0073] It should also be noted that, since part of the data that may be used in the calculation of the specified key indicators exists in the fact table and part exists in the dimension table, it is necessary to associate the fact table with the corresponding dimension table to obtain associated data, which can be understood as obtaining all the data required for the calculation through the association method. It can be seen from this that the fact table may also contain all the data required for the calculation, and it is not necessary to associate it with the dimension table at this time, so at this time step S103 can be to associate the fact table with an empty dimension table to obtain associated data, that is, only extract data from the fact table to perform the subsequent step S105.

[0074] Optionally, in another embodiment of the present application, a specific implementation of step S103 is as follows: Figure 2 As shown, the specific steps include:

[0075] S201: Encapsulate the dimension table into a first container, and encapsulate the associated field information into a second container.

[0076] Among them, the associated field information is stored in both the fact table and the dimension table.

[0077] Optionally, both the first container and the second container may be MAP containers. Specifically, the dimension table is encapsulated into the first container, and then the associated fields in the fact table in Kafka are extracted through the SQL operator and encapsulated into the second container.

[0078] S202: By associating the first container with the second container, data of multiple fields are extracted from multiple target data in the fact table and the dimension table to form associated data.

[0079] The target data refers to the data containing the associated field information.

[0080] By associating the first container with the second container, each target data in the dimension table is matched. Then, since the target data in the fact table can be determined based on the key field, data of multiple fields can be extracted from multiple target data in the fact table and the dimension table to form associated data.

[0081] For example, the fact table stores credit card numbers, installment amounts, and product numbers, while the dimension table stores credit card numbers and organization information. The credit card number can be used as the associated field to extract the installment amount and product number from the fact table and the organization information from the dimension table to form the associated data.

[0082] S104. The indicator factory passes the associated data to the stream computing component.

[0083] Optionally, the indicator factory can automatically call the stream computing component by calling the RestAPI, thereby passing the associated data to the stream computing component, and processing the stream computing component to execute step S105.

[0084] S105. The stream computing component aggregates and calculates the associated data based on the indicator definition information configured by the indicator factory to obtain key indicators.

[0085] Optionally, the indicator definition information may include metrics, statistical dimensions, statistical methods, and calibers, etc. That is, it can be simply understood as the indicator factory pre-defining which data to be counted, from which statistical dimensions to be counted, the statistical methods to be adopted, and the statistical scope, etc. Therefore, the stream computing component aggregates and calculates the associated data based on the indicator definition information configured by the indicator factory to obtain the key indicators to be obtained.

[0086] Then the indicator factory can associate the fact table with the dimension table by establishing a data model, and finally map the indicator definition information to the data model to complete the indicator configuration.

[0087] Optionally, in another embodiment of the present application, before executing step S105, it can be further performed: the indicator factory passes the data source information, target table information, operator information and job configuration information to the stream computing component by calling the application program interface.

[0088] The stream computing component forms a stream computing job based on data source information, target table information, operator information, and job configuration information.

[0089] When the stream computing job is started, it executes the indicator definition information based on the indicator factory configuration, summarizes and calculates the related data, and obtains the key indicators.

[0090] It should be noted that the data source information, target table information, operator information, and job configuration information are all configured through the indicator factory, and the stream computing component only needs to perform summary calculations, which can greatly speed up the efficiency of data processing.

[0091] Optionally, the data source information, target table information, operator information, and job configuration information may be transmitted to the stream computing component in an asynchronous manner, and the stream computing component may parse and generate a stream computing job.

[0092] Optionally, data source information may include data source name, title name, kafka consumption location, etc. Target table information may include sink source name, table name, etc. Operator information includes completed SQL statements, which are complete expressions of indicator definition information. Therefore, based on the SQL statements, it is possible to aggregate and calculate related data to obtain key indicators. Job configuration information also includes related dimension table information and statistical window information.

[0093] The embodiment of the present application provides a real-time processing method for indicator data, which is applied to a big data cloud platform, wherein the big data cloud platform at least includes a data acquisition component, a stream computing component, and an indicator factory. The data processing method provided by the embodiment of the present application collects real-time data through the data acquisition component, and the stream computing component pre-processes the real-time data to obtain a fact table storing the data to be processed. In addition, the statistical dimensions and metric fields are configured through the indicator factory, so that the indicator factory associates the fact table with the dimension table based on the information of the configured statistical dimensions and metric fields to obtain associated data, thereby realizing rapid integration of data. Finally, the indicator factory passes the associated data to the stream computing component, and the stream computing component summarizes and calculates the associated data based on the indicator definition information configured by the indicator factory to obtain key indicators. Thus, the fast computing characteristics of the stream computing component are utilized, and the work of the stream computing component that simplifies the information configuration through the indicator factory can effectively provide data processing efficiency, thereby ensuring the timeliness of the data.

[0094] Another embodiment of the present application provides another real-time processing method for indicator data, which is also applied to a big data cloud platform. The big data cloud platform includes at least a data acquisition component, a stream computing component, an indicator factory, an MPP database, and an Hbase database. Figure 3 As shown, the real-time processing method of indicator data provided by the embodiment of the present application includes the following steps:

[0095] S301. The data collection component collects the real-time message data provided by the event-driven architecture into Kafka.

[0096] The real-time data includes real-time flow data and real-time dimension data. The dimension table includes a preloaded dimension table and a real-time dimension table storing real-time dimension data.

[0097] It should be noted that the preloaded dimension table is a preloaded dimension table, and the number of preloaded dimension tables is usually relatively large. The real-time dimension data is added later, and the user's dimension data is required in subsequent calculations.

[0098] S302: The stream computing component preprocesses the real-time stream data to obtain a fact table.

[0099] Optionally, the specific implementation of step S302 may refer to step S102 in the above method embodiment, which will not be described in detail here.

[0100] S303. The MPP database synchronizes the preloaded dimension table in the Greenplum database, and synchronizes the preloaded dimension table to the Hbase database.

[0101] S304: The stream computing component preprocesses the real-time dimension data to obtain a real-time dimension table, and stores the implementation dimension table in the Hbase database.

[0102] It should be noted that step S304 may also be executed simultaneously with step S302, and the execution order in the embodiment of the present application is only one of the optional methods.

[0103] S305. The indicator factory associates the fact table with the dimension table based on the configured statistical dimension and metric field information to obtain associated data.

[0104] The associated data includes the data stored in the fact table and the dimension table. The information of the statistical dimension and the metric field is configured through the indicator factory.

[0105] S306: The indicator factory passes the associated data to the stream computing component.

[0106] S307. The stream computing component aggregates and calculates the associated data based on the indicator definition information configured by the indicator factory to obtain key indicators.

[0107] S308. The stream computing component stores the key indicators in the database according to the mapping relationship of the target table fields defined by the indicator factory for display on the front end.

[0108] The database may be an Oracle database.

[0109] The currently calculated key indicators are stored in the database and can be further summarized with the previously calculated key indicators.

[0110] Another embodiment of the present application provides a big data cloud platform, such as Figure 4 As shown, including:

[0111] The data collection component 401 is used to collect real-time data.

[0112] The indicator factory 402 is used to configure the information of the statistical dimensions and metric fields, and to associate the fact table with the dimension table based on the configured statistical dimensions and metric field information, obtain the associated data, and pass the associated data to the stream computing component.

[0113] The associated data includes the data stored in the fact table and the dimension table.

[0114] The stream computing component 403 is used to preprocess the real-time data to obtain a fact table, and to summarize and calculate the related data based on the indicator definition information configured by the indicator factory to obtain key indicators; wherein the fact table stores the data to be processed obtained after preprocessing the real-time data.

[0115] Optionally, in the big data cloud platform provided in another embodiment of the present application, when the data collection component collects real-time data, it is used to:

[0116] Collect real-time message data provided by the event-driven architecture into Kafka.

[0117] Optionally, Kafka can be set in the data collection component.

[0118] The real-time data includes real-time flow data and real-time dimension data. The dimension table includes a preloaded dimension table and a real-time dimension table storing real-time dimension data.

[0119] In the embodiment of the present application, the stream computing component preprocesses the real-time data to obtain the fact table, which is used to:

[0120] Preprocess the real-time flow data to obtain the fact table.

[0121] Optionally, in the big data cloud platform provided in another embodiment of the present application, see also Figure 4 , and may further include:

[0122] MPP database 404 and Hbase database 405 .

[0123] The MPP database 404 is used to synchronize the preloaded dimension table in the Greenplum database, and synchronize the preloaded dimension table to the Hbase database 405 .

[0124] The Hbase database 405 is used to store the real-time dimension table obtained by preprocessing the real-time dimension data by the stream computing component 403, and to store the preloaded dimension table synchronized with the MPP database.

[0125] Optionally, in the big data cloud platform provided in another embodiment of the present application, when the stream computing component pre-processes the data to be processed and obtains the fact table, it is used to:

[0126] Parse the data to be processed into tabular data, and extract the data of the specified fields from the tabular data to form a fact table.

[0127] Optionally, in a big data cloud platform provided in another embodiment of the present application, the indicator factory associates the fact table with the dimension table to obtain associated data, which is used to:

[0128] The dimension table is encapsulated into a first container, and the associated field information is encapsulated into a second container.

[0129] Among them, the associated field information is stored in both the fact table and the dimension table.

[0130] By associating the first container with the second container, data of multiple fields are extracted from multiple target data in the fact table and the dimension table to form associated data; wherein the target data refers to data containing associated field information.

[0131] Optionally, in the big data cloud platform provided in another embodiment of the present application, the indicator factory is further used for:

[0132] The data source information, target table information, operator information, and job configuration information are passed to the stream computing component by calling the application program interface.

[0133] Optionally, in the big data cloud platform provided in another embodiment of the present application, the stream computing component is further used to:

[0134] Based on the data source information, target table information, operator information, and job configuration information, a stream computing job is formed.

[0135] When the stream computing job is started, it executes the indicator definition information based on the indicator factory configuration, summarizes and calculates the related data, and obtains the key indicators.

[0136] Optionally, in the big data cloud platform provided in another embodiment of the present application, the stream computing component is further used to:

[0137] According to the mapping relationship of the target table fields defined by the indicator factory, the key indicators are stored in the database for display on the front end.

[0138] It should be noted that the specific working process of each component in the big data cloud platform provided in the above embodiment of the present application can refer to the implementation method of the corresponding steps in the above method embodiment, and will not be repeated here.

[0139] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0140] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A real-time processing method for index data, characterized in that: Applied to a big data cloud platform, wherein the big data cloud platform includes at least a data acquisition component, a stream computing component, and an indicator factory, and the processing method includes: The data acquisition component acquires real-time data; The stream computing component preprocesses the real-time data to obtain a fact table; wherein the fact table stores the data to be processed obtained after preprocessing the real-time data; The indicator factory associates the fact table with the dimension table based on the information of the configured statistical dimensions and metric fields to obtain associated data; wherein the associated data includes the data stored in the fact table and the dimension table; the information of the statistical dimensions and metric fields is configured by the indicator factory; The indicator factory passes the associated data to the stream computing component; The stream computing component aggregates and calculates the associated data based on the indicator definition information configured by the indicator factory to obtain key indicators.

2. The method according to claim 1, characterized in that The data acquisition component collects real-time data, including: The data collection component collects the real-time message data provided by the event-driven architecture into Kafka; wherein the real-time data includes real-time flow data and real-time dimension data; the dimension table includes a preloaded dimension table and a real-time dimension table storing the real-time dimension data; The stream computing component preprocesses the real-time data to obtain a fact table, including: The stream computing component preprocesses the real-time stream data to obtain a fact table.

3. The method according to claim 2, characterized in that The big data cloud platform also includes an MPP database and an Hbase database. The indicator factory associates the fact table with the dimension table based on the information of the configured statistical dimensions and metric fields, and before obtaining the associated data, it also includes: The MPP database synchronizes the preloaded dimension table in the Greenplum database, and synchronizes the preloaded dimension table to the Hbase database; The stream computing component preprocesses the real-time dimension data to obtain the real-time dimension table, and stores the real-time dimension table in the Hbase database.

4. The method according to claim 1, characterized in that The stream computing component preprocesses the data to be processed to obtain a fact table, including: The stream computing component parses the data to be processed into tabular data, and extracts data of specified fields from the tabular data to form a fact table.

5. The method according to claim 1, characterized in that The indicator factory associates the fact table with the dimension table to obtain associated data, including: Encapsulating the dimension table into a first container, and encapsulating the associated field information into a second container; wherein the associated field information is stored in both the fact table and the dimension table; By associating the first container with the second container, data of multiple fields are extracted from multiple target data in the fact table and the dimension table to form associated data; wherein the target data refers to data containing the associated field information.

6. The method according to claim 1, characterized in that The stream computing component aggregates and calculates the associated data based on the indicator definition information configured by the indicator factory to obtain the key indicator, and further includes: The indicator factory passes the data source information, target table information, operator information and job configuration information to the stream computing component by calling the application program interface; The stream computing component forms a stream computing job based on the data source information, the target table information, the operator information and the job configuration information; wherein, when the stream computing job is started, it executes the indicator definition information configured based on the indicator factory, summarizes and calculates the related data, and obtains key indicators.

7. The method according to claim 1, characterized in that The stream computing component aggregates and calculates the associated data based on the indicator definition information configured by the indicator factory to obtain the key indicators, and further includes: The stream computing component stores the key indicators in the database according to the mapping relationship of the target table fields defined by the indicator factory for display by the front end.

8. A big data cloud platform, characterized in that: include: Data collection component, used to collect real-time data; The indicator factory is used to configure the information of the statistical dimension and the metric field, and to associate the fact table with the dimension table based on the configured statistical dimension and the metric field information to obtain the associated data, and pass the associated data to the stream computing component; wherein the associated data includes the data stored in the fact table and the dimension table; A stream computing component is used to preprocess the real-time data to obtain a fact table, and to summarize and calculate the related data based on the indicator definition information configured by the indicator factory to obtain key indicators; wherein the fact table stores the data to be processed obtained after preprocessing the real-time data.

9. The big data cloud platform according to claim 8, characterized in that: When the data collection component collects real-time data, it is used to: The real-time message data provided by the event-driven architecture is collected into Kafka; wherein the real-time data includes real-time flow data and real-time dimension data; the dimension table includes a preloaded dimension table and a real-time dimension table storing the real-time dimension data; The stream computing component preprocesses the real-time data to obtain a fact table, which is used to: The real-time flow data is preprocessed to obtain a fact table.

10. The big data cloud platform according to claim 9, characterized in that: Also includes: MPP database and Hbase database; The MPP database is used to synchronize the preloaded dimension table in the Greenplum database, and synchronize the preloaded dimension table to the Hbase database; The Hbase database is used to store the real-time dimension table obtained by preprocessing the real-time dimension data by the stream computing component, and to store the preloaded dimension table synchronized with the MPP database.

Images