An index calculation method, device, storage medium and index calculation platform

Abstract

In a technical solution of an indicator calculation method, device, storage medium and indicator calculation platform provided by an embodiment of the present invention, multiple collected basic data are preprocessed, and a data label corresponding to each basic data is determined. Based on the preset basic indicators, target data for calculating real-time indicators are obtained from the basic data according to the data labels corresponding to each basic data, and the target data is calculated to obtain real-time indicators. A stream computing engine is used, and based on the dimensions corresponding to the real-time indicators, each indicator is calculated layer by layer to determine the real-time indicators of all dimensions. Only one stream computing is required to complete the calculation of real-time indicators of all dimensions, thereby reducing development costs and operation and maintenance costs.

Description

[Technical field]

[0001] The present invention relates to the field of data service technology, and in particular to an indicator calculation method, device, storage medium and indicator calculation platform. [Background technology]

[0002] Currently, offline big data architecture, Lambda architecture or Kappa architecture are usually used to calculate indicators. However, for the offline big data architecture, this solution is mainly for offline indicator calculation and does not support real-time indicator calculation. If the indicator system is unreasonable, it will lead to data redundancy and repeated development. For the solution using Lambda architecture for indicator calculation, although it meets the real-time indicator calculation, it brings more development and operation and maintenance costs. For the solution using Kappa architecture for indicator calculation, it seems simple, but the implementation difficulty is relatively high, especially the data replay part. Business demand modification or historical data reprocessing needs to be completed through upstream replay. The throughput capacity of reprocessing historical data will be lower than offline processing, although it can be compensated by increasing computing resources. [Summary of the invention]

[0003] In view of this, the present invention provides an indicator calculation method, device, storage medium and indicator calculation platform. Through the stream computing engine, only one stream computing is required to complete the real-time indicator calculation of all dimensions, thereby reducing the development cost and operation and maintenance cost.

[0004] On the one hand, an embodiment of the present invention provides an indicator calculation method, including:

[0005] Preprocessing the collected multiple basic data and determining a data label corresponding to each of the basic data;

[0006] Based on the preset basic indicators, target data for calculating the real-time indicators are obtained from the basic data according to the data tags corresponding to the basic data;

[0007] Calculating the target data to obtain the real-time indicator;

[0008] A stream computing engine is used, and based on the dimensional relationship corresponding to the real-time indicators, each indicator is calculated upward layer by layer to determine the real-time indicators of all dimensions.

[0009] Optionally, the basic indicators include atomic indicators, time granularity and dimension;

[0010] Based on the preset basic indicators, target data for calculating the real-time indicators are obtained from the basic data according to the data tags corresponding to the basic data, including:

[0011] Determining a data label corresponding to the atomic indicator;

[0012] Target data corresponding to the time granularity and the dimension are determined from basic data corresponding to the data tag.

[0013] Optionally, it also includes:

[0014] A non-real-time indicator is determined from data generated after preprocessing the plurality of basic data.

[0015] Optionally, it also includes:

[0016] According to the preset real-time indicator classification, the real-time indicator is stored in a designated classification pair storage system;

[0017] According to the preset non-real-time indicator classification, the non-real-time indicator is stored in a designated classification pair storage system.

[0018] Optionally, the real-time indicator classification includes key indicators, thermal indicators or temperature indicators.

[0019] Optionally, the non-real-time index includes a cold index, an ultra-long-time granularity aggregation, and an index OLAP cube.

[0020] Optionally, the preprocessing of the collected multiple basic data includes:

[0021] The collected multiple basic data are preprocessed through an abnormal protection mechanism, and the abnormal protection mechanism includes a data loss response mechanism, a data disorder / late arrival response mechanism, a special indicator response mechanism and a high availability mechanism.

[0022] On the other hand, an embodiment of the present invention provides an indicator calculation device, the device comprising:

[0023] A preprocessing module, used to preprocess the collected multiple basic data and determine the data label corresponding to each of the basic data;

[0024] An acquisition module, used to acquire target data for calculating real-time indicators from the basic data according to data tags corresponding to each basic data based on preset basic indicators;

[0025] A calculation module, used to calculate the target data to obtain the real-time index;

[0026] The determination module is used to adopt a stream computing engine and, based on the dimensional blood relationship corresponding to the real-time indicators, calculate each indicator layer by layer to determine the real-time indicators of all dimensions.

[0027] On the other hand, an embodiment of the present invention provides a storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above-mentioned indicator calculation method.

[0028] On the other hand, an embodiment of the present invention provides an indicator calculation platform, including a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions, and the program instructions are loaded by the processor and execute the steps of the above-mentioned indicator calculation method.

[0029] In the technical solution provided by the embodiment of the present invention, the collected multiple basic data are preprocessed, and the data label corresponding to each basic data is determined. Based on the preset basic indicators, the target data for calculating the real-time indicator is obtained from the basic data according to the data label corresponding to each basic data, and the target data is calculated to obtain the real-time indicator. A stream computing engine is used, and based on the dimension corresponding to the real-time indicator, each indicator is calculated layer by layer to determine the real-time indicator of all dimensions. Only one stream computing is required to complete the calculation of the real-time indicator of all dimensions, thereby reducing the development cost and operation and maintenance cost.

Brief Description of the Drawings

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 is a flow chart of an indicator calculation method provided by an embodiment of the present invention;

[0032] Figure 2 is a flow chart of an indicator calculation method provided by another embodiment of the present invention;

[0033] Figure 3 It is a structural schematic diagram of an indicator calculation device provided by an embodiment of the present invention;

[0034] Figure 4 A schematic diagram of an indicator calculation platform provided in an embodiment of the present invention. [Specific implementation method]

[0035] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0036] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0038] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0039] Currently, offline big data architecture, Lambda architecture or Kappa architecture are usually used to calculate indicators. However, for the offline big data architecture, this solution is mainly for offline indicator calculation and does not support real-time indicator calculation. If the indicator system is unreasonable, it will lead to data redundancy and repeated development. For the solution using Lambda architecture for indicator calculation, although it meets the real-time indicator calculation, it brings more development and operation and maintenance costs. For the solution using Kappa architecture for indicator calculation, it seems simple, but the implementation difficulty is relatively high, especially the data replay part. Business demand modification or historical data reprocessing needs to be completed through upstream replay. The throughput capacity of reprocessing historical data will be lower than offline processing, although it can be compensated by increasing computing resources.

[0040] In order to overcome the above-mentioned problems existing in the prior art, the present invention provides an indicator calculation method, which integrates the dual-engine architecture of stream computing and interactive analysis on the basis of the existing Kappa architecture. In this architecture, the stream computing engine is responsible for the real-time indicator calculation of all dimensions, and the interactive analysis engine is responsible for the on-demand calculation of real-time indicators. The two work together to quickly respond to changes in business needs while ensuring high throughput and low latency, and reduce the problems caused by data replay. Specifically, an indicator calculation method provided by the present invention is introduced through the following embodiments:

[0041] Figure 1 A flowchart of an indicator calculation method provided by an embodiment of the present invention is as follows: Figure 1 As shown, the method includes:

[0042] Step 102: pre-process the collected multiple basic data, and determine the data label corresponding to each of the basic data.

[0043] Step 104: Based on the preset basic indicators, target data for calculating the real-time indicators are obtained from the basic data according to data tags corresponding to each basic data.

[0044] Step 106: Calculate the target data to obtain the real-time index.

[0045] Step 108: Using a stream computing engine, and based on the dimensional relationship corresponding to the real-time indicators, calculate each indicator layer by layer to determine the real-time indicators of all dimensions.

[0046] In the technical solution provided by the embodiment of the present invention, the collected multiple basic data are preprocessed, and the data label corresponding to each basic data is determined. Based on the preset basic indicators, the target data for calculating the real-time indicator is obtained from the basic data according to the data label corresponding to each basic data, and the target data is calculated to obtain the real-time indicator. A stream computing engine is used, and based on the dimension corresponding to the real-time indicator, each indicator is calculated layer by layer to determine the real-time indicator of all dimensions. Only one stream computing is required to complete the calculation of the real-time indicator of all dimensions, thereby reducing the development cost and operation and maintenance cost.

[0047] Figure 2 A flowchart of another indicator calculation method provided by an embodiment of the present invention is as follows: Figure 2 As shown, the method includes:

[0048] Step 202: pre-process the collected multiple basic data through an abnormal protection mechanism, and determine the data label corresponding to each of the basic data.

[0049] In this step, the abnormal protection mechanism includes data loss response mechanism, data disorder / late response mechanism, special indicator response mechanism and high availability mechanism. Specifically, the data loss response mechanism is used to receive, clean and convert data, push it to the message queue in real time for real-time calculation, and push data in batches to HDFS in the form of detailed files for persistence, retain the source data, and provide limited data playback based on the message queue Kafka consumption displacement mechanism, supporting the specification of any playback position.

[0050] Among them, the data disorder / late arrival response mechanism is based on the event time recorded when the data actually occurs, so as to effectively handle any data disorder situation.

[0051] Among them, the special indicator response mechanism can satisfy the commutative law / associative law. Indicators do not need special processing, such as MAX / MIN / SUM / COUNT. Indicators that do not satisfy the commutative law / associative law need special processing. The average is indirectly realized through SUM / COUNT. The variance / standard deviation must rely on the full set of data and can only be calculated and processed separately. Cardinality count can be calculated and processed separately, or it can be stored and processed.

[0052] Among them, the high availability mechanism is used for distributed components of the entire link to provide high availability guarantees. For example, real-time computing provides a failover mechanism. Flink provides a Checkpoint snapshot mechanism. When a real-time computing job exits abnormally due to reasons such as the machine, network, or dirty data, it ensures that the calculation continues based on the calculation results before the exit, ensuring the accuracy of the data, and then providing high availability guarantees such as At-least-Once / Exactly-Once.

[0053] In the embodiment of the present invention, the accuracy and reliability of the indicators can be guaranteed by the above-mentioned abnormality protection mechanism.

[0054] In this step, preprocessing may include data cleaning and conversion, data exchange, and data preprocessing. Among them, data cleaning and conversion and data exchange mainly complete real-time data collection and send it to the message queue (Kafka can be selected as the technical model). For different business systems, existing real-time data collection technologies can be used to complete real-time data collection. Before outputting to the message queue, data is received and cleaned and converted, and unified data cleaning and conversion rule definitions and management are provided to avoid data cleaning and conversion being scattered in different business logics and eliminate data ambiguity. In addition to outputting to the message queue, batch data exchange can be performed to store detailed data on disk (HDFS can be selected as the technical model) to support data compensation and playback for real-time calculations and for data verification.

[0055] Data preprocessing is to optimize the data preprocessing implemented by stream computing, simplify data definition, retain only the attributes that stream computing strongly depends on, and reduce IO consumption. For example, if you need to calculate the download rate for 1 minute, for stream computing, since the rate is equal to the flow divided by the delay, the strongly dependent attributes include flow and delay.

[0056] In the embodiment of the present invention, each step of the embodiment of the present invention can be performed based on the stream-batch integrated indicator calculation platform of Flink.

[0057] Furthermore, after step 202, the method further includes:

[0058] Step 203: Determine a non-real-time indicator from the data generated after preprocessing the plurality of basic data.

[0059] In this step, the non-real-time indicators can be understood as full indicators, that is, the non-real-time indicators can include historical indicators.

[0060] Furthermore, after step 202, the method further includes:

[0061] Step 204: Based on the preset basic indicators, target data for calculating the real-time indicators are obtained from the basic data according to data tags corresponding to each basic data.

[0062] In this step, the basic indicators include atomic indicators, time granularity and dimension. In addition, the basic indicators can also include dimension lineage, dimension attributes, derived indicators, etc., to build a unified indicator matrix and indicator algorithm library. In addition, the basic indicators can be flexibly defined, following the definition method of "atomic indicators + time granularity + dimension", such as the number of 4G users in a certain city area every 1 minute.

[0063] In the embodiment of the present invention, step 204 may specifically include:

[0064] Step 2041, determining a data label corresponding to the atomic indicator;

[0065] In this step, atomic indicators, also called metrics, may include order volume or user volume, etc. Data tags are used to indicate data types, for example, data types may include user information or other information.

[0066] In an embodiment of the present invention, for example, taking the atomic indicator as the number of 4G users, since it involves the number of 4G users, the data tag corresponding to the atomic indicator should be user information, and then the subsequent step 206 is executed to determine the target data corresponding to the time granularity and the dimension from the basic data corresponding to the data tag.

[0067] Step 2042: Determine the target data corresponding to the time granularity and the dimension from the basic data corresponding to the data tag.

[0068] In the embodiment of the present invention, taking the data tag as user information, the time granularity includes 1 minute, and the dimension includes the A city area as an example, by determining the basic data whose data tag is the user information, the number of 4G users in the A city area corresponding to each minute is filtered out from the basic data.

[0069] Step 206: Calculate the target data to obtain the real-time index.

[0070] In an embodiment of the present invention, this step can be executed by an interactive analysis engine. Specifically, based on the above step 204, the number of 4G users in the A city area every minute can be obtained based on the number of 4G users in the A city area every minute, that is, a real-time indicator can be obtained.

[0071] Step 208: Using a stream computing engine, and based on the dimensional relationship corresponding to the real-time indicators, each indicator is calculated upward layer by layer to determine the real-time indicators of all dimensions.

[0072] In this step, the stream computing engine may include the stream computing engine Flink, which can realize the calculation of real-time indicators of all dimensions. The basic indicators may also include dimension lineage relationships, where the dimension lineage relationships are used to indicate the relationship between dimensions, for example, there is a lineage relationship between seconds, minutes, and hours.

[0073] In the implementation of the present invention, for example, taking the dimension corresponding to the real-time indicator as 1 minute as an example, the real-time indicator with a dimension of hour can be calculated based on the accumulation of 60 minutes of traffic in the real-time indicator, that is, 1 hour of traffic.

[0074] It should be noted that non-real-time indicators can also be generated by determining multiple full-dimensional real-time indicators.

[0075] Furthermore, we provide easy-to-use, low-cost, stable and warm indicator services to empower application systems.

[0076] Furthermore, after step 202 and step 208, the following steps are further included:

[0077] Step 209: According to the preset real-time indicator classification, the real-time indicator is stored in a designated classification pair storage system; according to the preset non-real-time indicator classification, the non-real-time indicator is stored in a designated classification pair storage system.

[0078] In this step, the real-time index classification includes key index, hot index or warm index. The non-real-time index includes cold index, ultra-long time granularity aggregation and index OLAP cube.

[0079] The stored content includes indicator results and timestamps (for example, taking minute granularity as an example, the timestamp of the stored content may include: 202002181844), and stores indicators within a preset time.

[0080] Among them, hot indicators include the latest data, warm indicators include the last hour, key indicators include indicators that are focused on in business scenarios, and add labels to ensure key protection in storage, monitoring, etc. Cold indicators in non-real-time indicators include full historical indicators, and ultra-long-term granularity aggregation can include indicator aggregation based on any time and free expansion. The indicator OLAP cube relies on an interactive analysis engine to provide more three-dimensional and comprehensive on-demand calculation of non-real-time indicators.

[0081] In the implementation of the present invention, indicators can be divided into real-time indicators and non-real-time indicators according to their timeliness. By storing real-time indicators separately from non-real-time indicators, all-dimensional real-time indicators can be asynchronously written into the storage system corresponding to non-real-time indicators, and cooperate with the interactive analysis engine (Kylin can be selected as the technology) to complete the on-demand calculation of indicators that meet business needs. Only the subset of key indicators that are truly required for real-time business scenarios will be synchronously written into the real-time indicator storage (Redis / HBase can be selected as the technology), supporting temperature indicator empowerment. In addition, a complete exception protection mechanism is provided to ensure the accuracy and reliability of indicators.

[0082] In the embodiment of the present invention, non-real-time indicators are stored, relying on an interactive analysis engine to be responsible for on-demand calculation of indicators, quickly responding to changes in business needs, and reducing problems caused by data replay.

[0083] In the technical solution provided by the embodiment of the present invention, the collected multiple basic data are preprocessed, and the data label corresponding to each basic data is determined. Based on the preset basic indicators, the target data for calculating the real-time indicator is obtained from the basic data according to the data label corresponding to each basic data, and the target data is calculated to obtain the real-time indicator. A stream computing engine is used, and based on the dimension corresponding to the real-time indicator, each indicator is calculated layer by layer to determine the real-time indicator of all dimensions. Only one stream computing is required to complete the calculation of the real-time indicator of all dimensions, thereby reducing the development cost and operation and maintenance cost.

[0084] Furthermore, the technical solution provided by the embodiment of the present invention realizes the integration of stream and batch processing. There is no need to maintain two links of offline computing and real-time computing, and there is no need to develop two sets of identical codes. Based on the stream computing engine Flink, only the real-time computing link needs to be maintained and one set of codes needs to be developed, which reduces the development and operation and maintenance costs. The dual-engine architecture of stream computing and interactive analysis, with real-time indicators first, realizes real-time and on-demand calculation of indicators, quickly responds to changes in business needs, and stores hot and cold indicators separately, providing richer, more stable, and warmer indicator services, empowering application systems, and providing rich exception protection mechanisms to ensure the accuracy and reliability of indicators.

[0085] Figure 3 is a schematic diagram of the structure of an indicator calculation device provided by an embodiment of the present invention. Figure 3 As shown, the device comprises:

[0086] A preprocessing module 11 preprocesses the collected multiple basic data and determines a data label corresponding to each of the basic data;

[0087] An acquisition module 12 is used to acquire target data for calculating real-time indicators from the basic data according to data tags corresponding to each basic data based on preset basic indicators;

[0088] A calculation module 13, used to calculate the target data to obtain the real-time index;

[0089] The determination module 14 is used to adopt a stream computing engine and, based on the dimensional relationship corresponding to the real-time indicators, calculate each indicator layer by layer to determine the real-time indicators of all dimensions.

[0090] In an embodiment of the present invention, the basic indicators include atomic indicators, time granularity and dimension; the acquisition module 12 of the device is specifically used to determine the data label corresponding to the atomic indicator; and determine the target data corresponding to the time granularity and the dimension from the basic data corresponding to the data label.

[0091] In the embodiment of the present invention, the determination module 14 of the device is further used to determine the non-real-time indicator from the data generated after pre-processing the plurality of basic data.

[0092] In the embodiment of the present invention, the device further includes: a storage module 15.

[0093] The storage module 15 is used to store the real-time indicators into a designated classification pair storage system according to the preset real-time indicator classification; and store the non-real-time indicators into a designated classification pair storage system according to the preset non-real-time indicator classification.

[0094] In the embodiment of the present invention, the real-time indicator classification includes key indicators, thermal indicators or temperature indicators.

[0095] In the embodiment of the present invention, the non-real-time indicators include cold indicators, ultra-long time granularity aggregation, and indicator OLAP cubes.

[0096] In an embodiment of the present invention, the preprocessing module 11 of the device is specifically used to preprocess the collected multiple basic data through an abnormal protection mechanism, and the abnormal protection mechanism includes a data loss response mechanism, a data disorder / late arrival response mechanism, a special indicator response mechanism and a high availability mechanism.

[0097] In the technical solution provided by the embodiment of the present invention, the collected multiple basic data are preprocessed, and the data label corresponding to each basic data is determined. Based on the preset basic indicators, the target data for calculating the real-time indicator is obtained from the basic data according to the data label corresponding to each basic data, and the target data is calculated to obtain the real-time indicator. A stream computing engine is used, and based on the dimension corresponding to the real-time indicator, each indicator is calculated layer by layer to determine the real-time indicator of all dimensions. Only one stream computing is required to complete the calculation of the real-time indicator of all dimensions, thereby reducing the development cost and operation and maintenance cost.

[0098] Furthermore, the technical solution provided by the embodiment of the present invention realizes the integration of stream and batch processing. There is no need to maintain two links of offline computing and real-time computing, and there is no need to develop two sets of identical codes. Based on the stream computing engine Flink, only the real-time computing link needs to be maintained and one set of codes needs to be developed, which reduces the development and operation and maintenance costs. The dual-engine architecture of stream computing and interactive analysis, with real-time indicators first, realizes real-time and on-demand calculation of indicators, quickly responds to changes in business needs, and stores hot and cold indicators separately, providing richer, more stable, and warmer indicator services, empowering application systems, and providing rich exception protection mechanisms to ensure the accuracy and reliability of indicators.

[0099] An embodiment of the present invention provides a storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the steps of the embodiment of the above-mentioned indicator calculation method. For a specific description, please refer to the embodiment of the above-mentioned indicator calculation method.

[0100] The embodiment of the present invention provides an indicator calculation platform, including a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions, and the program instructions are loaded and executed by the processor to implement the steps of the above-mentioned indicator calculation method. For a specific description, please refer to the embodiment of the above-mentioned indicator calculation method.

[0101] Figure 4 A schematic diagram of an indicator calculation platform provided by an embodiment of the present invention. Figure 4 As shown, the indicator calculation platform 4 of this embodiment includes: a processor 41, a memory 42, and a computer program 43 stored in the memory 42 and executable on the processor 41. When the computer program 43 is executed by the processor 41, the indicator calculation method applied in the embodiment is implemented. To avoid repetition, it is not described one by one here. Alternatively, when the computer program is executed by the processor 41, the functions of each model / unit in the indicator calculation device applied in the embodiment are implemented. To avoid repetition, it is not described one by one here.

[0102] The indicator calculation platform 4 includes, but is not limited to, a processor 41 and a memory 42. Those skilled in the art will understand that Figure 4 It is only an example of the indicator calculation platform 4 and does not constitute a limitation of the indicator calculation platform 4. It can include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the indicator calculation platform 4 can also include input and output devices, network access devices, buses, etc.

[0103] The processor 41 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0104] The memory 42 may be an internal storage unit of the indicator computing platform 4, such as a hard disk or memory of the indicator computing platform 4. The memory 42 may also be an external storage device of the indicator computing platform 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the indicator computing platform 4. Further, the memory 42 may also include both an internal storage unit and an external storage device of the indicator computing platform 4. The memory 42 is used to store computer programs and other programs and data required by the indicator computing platform 4. The memory 42 may also be used to temporarily store data that has been output or is to be output.

[0105] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0106] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0108] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0109] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (Processor) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0110] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for calculating an indicator, characterized in that: include: Preprocessing the collected multiple basic data and determining a data label corresponding to each of the basic data; Based on the preset basic indicators, target data for calculating the real-time indicators are obtained from the basic data according to the data tags corresponding to the basic data; Calculating the target data to obtain the real-time indicator; Using a stream computing engine, and based on the dimensional relationship corresponding to the real-time indicators, calculating each indicator layer by layer upwards to determine the real-time indicators of all dimensions; The preprocessing of the collected multiple basic data includes: Preprocessing the collected multiple basic data through an abnormal protection mechanism, wherein the abnormal protection mechanism includes a data loss response mechanism, a data disorder / late arrival response mechanism, a special indicator response mechanism, and a high availability mechanism; The preprocessing includes data cleaning and conversion, data exchange and data preprocessing.

2. The method according to claim 1, characterized in that The basic indicators include atomic indicators, time granularity and dimensions; Based on the preset basic indicators, target data for calculating the real-time indicators are obtained from the basic data according to the data tags corresponding to the basic data, including: Determining a data label corresponding to the atomic indicator; Target data corresponding to the time granularity and the dimension are determined from basic data corresponding to the data tag.

3. The method according to claim 1, characterized in that Also includes: A non-real-time indicator is determined from data generated after preprocessing the plurality of basic data.

4. The index calculation method according to claim 1 or 3, characterized in that: Also includes: According to the preset real-time indicator classification, the real-time indicator is stored in a designated classification pair storage system; According to the preset non-real-time indicator classification, the non-real-time indicator is stored in a designated classification pair storage system.

5. The index calculation method according to claim 4, characterized in that: The real-time indicator classification includes key indicators, thermal indicators or temperature indicators.

6. The index calculation method according to claim 4, characterized in that: The non-real-time indicators include cold indicators, ultra-long-time granularity aggregation, and indicator OLAP cubes.

7. An index calculation device, characterized in that: include: A preprocessing module, used to preprocess the collected multiple basic data and determine the data label corresponding to each of the basic data; An acquisition module, used to acquire target data for calculating real-time indicators from the basic data according to data tags corresponding to each basic data based on preset basic indicators; A calculation module, used to calculate the target data to obtain the real-time index; A determination module, which is used to use a stream computing engine and, based on the dimensional relationship corresponding to the real-time indicators, calculate each indicator layer by layer to determine the real-time indicators of all dimensions; The preprocessing of the collected multiple basic data includes: Preprocessing the collected multiple basic data through an abnormal protection mechanism, wherein the abnormal protection mechanism includes a data loss response mechanism, a data disorder / late arrival response mechanism, a special indicator response mechanism, and a high availability mechanism; The preprocessing includes data cleaning and conversion, data exchange and data preprocessing.

8. A storage medium comprising a stored program, wherein: When the program is running, the device where the storage medium is located is controlled to execute the indicator calculation method described in any one of claims 1 to 6.

9. An indicator calculation platform, comprising a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions, characterized in that: When the program instructions are loaded and executed by the processor, the indicator calculation method described in any one of claims 1 to 6 is implemented.

Images