Indicator data pre-aggregation method

Abstract

The application discloses an index data pre-aggregation method, which comprises the following steps: collecting index data in real time, and reporting collected burying point information to a cat monitoring server; storing burying point information details in a clickhouse database and storing aggregated index data in an InfluxDB database, wherein the index data is generated by aggregating the burying point information according to a predetermined rule; storing the burying point information in the clickhouse database in the form of a burying point detail table, querying and analyzing data according to the burying point detail table, and storing index data generated by re-aggregating the burying point detail table data on demand in the InfluxDB.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and more specifically, to a method for pre-aggregating indicator data. Background Technology

[0002] As the wave of enterprise digitalization advances, data has become the most important asset for most companies, and the world's top companies rely on data to support their business decisions. Statistics show that the top 40% of companies make most of their business decisions based on data. In other words, organizations that drive business decisions through data are more likely to reach the top of their industry compared to those driven by experience and instinct.

[0003] Traditional enterprises often rely on tools like Excel to organize, analyze, and generate charts. However, with the exponential growth of business data and the increasingly rapid changes in the market environment, traditional data analysis models often only reflect outdated data and support fixed-dimensional data displays, failing to meet the data analysis needs of enterprises. Decision-makers require truly agile data insights and a real-time, accurate data analysis platform.

[0004] As a result, more and more companies are trying to collect, analyze, and use data. At the same time, there are a large number of related technologies on the market. For data collection, there are Google's GA, Baidu Statistics, Sensors Data, Zhuge IO, etc. For data analysis, there are real-time streaming computing technologies such as Storm, Spark Streaming, S4, Heron, etc. For query analysis, there are ClickHouse, Hive, Impala, Pig, Presto, Phoenix, SparkSQL, Drill, Flink, Kylin, Druid, and offline computing technologies such as Hadoop MapReduce and Spark.

[0005] By combining the above technologies, we can fully integrate the data collected from various systems, build a streamlined and unified data collection standard, and ultimately form a systematic real-time data indicator pool without affecting daily R&D efficiency. This provides digital decision support for various business scenarios in the future and is an important guarantee for the success of enterprise digital transformation.

[0006] However, existing solutions have the following three major drawbacks:

[0007] 1) Building a big data system is complex and costly:

[0008] If built according to the big data system, the cost of investing in technical personnel is high, and the cost of investing in machine resources is also high. For medium-sized enterprises with a data scale and personnel scale, this will inevitably result in a huge financial burden.

[0009] 2) The collected indicator data has a high barrier to entry for use:

[0010] After the collected metric data flows into the analytical data system, it is difficult for product managers, operations and other partners who do not have technical knowledge to use it directly to create reports. This is because they need to understand the table structure and fields, write SQL-like query statements, and perform aggregate queries on the metric data. This often requires the support of BI personnel to complete.

[0011] 3) Indicator data cannot be pre-aggregated on demand:

[0012] The collected and stored metric data is perfectly adequate for real-time querying and analysis by employees within an enterprise. However, when extended to customer-facing (B2B, B2C) or real-time / near-real-time transaction processing, pre-aggregation is necessary; otherwise, big data technology struggles to support it (unless resources are heavily invested). Yet, pre-aggregation often fails to aggregate historical data, leading to a dilemma regarding whether or not this metric should be pre-aggregated at all. Summary of the Invention

[0013] To address the aforementioned problems in related technologies, this invention proposes an indicator data pre-aggregation method and system, which can realize real-time aggregation of predefined indicators and on-demand aggregation of undefined indicators, thereby achieving on-demand calculation.

[0014] The technical solution of this invention is implemented as follows:

[0015] According to one aspect of the present invention, a method for pre-aggregating indicator data is provided, comprising: real-time acquisition of indicator data; reporting the acquired tracking information to a CAT server; the CAT server converting the tracking information stored in the ClickHouse database into pre-aggregated indicator data and storing it in an InfluxDB database, wherein the indicator data is generated by aggregating the tracking information according to predetermined rules; the tracking information is stored in the ClickHouse database in the form of a tracking detail table, and data is queried and analyzed based on the tracking detail table, and indicators are re-aggregated as needed based on the tracking detail table data and stored in the InfluxDB database.

[0016] According to an embodiment of the present invention, the step of the cat server converting the tracking information stored in the clickhouse database into pre-aggregated metric data and storing it in the InfluxDB database is executed by the following four threads: Minute-level metric report generation thread: The cat server processes each received tracking information using stream processing, sequentially matching tracking configuration, matching metric sets, filtering aggregation rules, and updating minute-level metric reports, thereby generating and updating minute-level metric reports in memory in real time as stream processing occurs; Minute-level metric report persistence thread: This thread aggregates metric data from all nodes into memory, then merges it into a single metric report and writes it to the local disk; Metric generation and hourly / daily metric aggregation thread: This thread converts the minute-level metric report file into metric data and adds it to the asynchronous upload queue of the InfluxDB database, while simultaneously merging the metric data into hourly metrics; When the hour ends, the hourly metric data is retrieved from the cache database Redis, added to the asynchronous upload queue of the InfluxDB database, and merged into the daily metrics; When the day ends, the daily metrics are retrieved and added to the asynchronous upload queue of the InfluxDB database, where Redis stores the current hourly and daily metric data. Metrics upload thread: It reads metric data from the upload queue of the InfluxDB database in a loop and uploads the metric data to the InfluxDB database in batches.

[0017] According to an embodiment of the present invention, indicator data collection includes front-end collection, which includes PC websites, H5 websites and mini-programs, and indicator data collection is performed by embedding Arms JavaScript probes.

[0018] According to an embodiment of the present invention, the steps of collecting event tracking data include: custom events collected by the Arms JavaScript probe are reported to the log library in real time, and the log library provides the ability to consume custom event tracking log data in real time through a consumer group; the consumer captures and parses the custom event tracking information in the original log.

[0019] According to an embodiment of the present invention, the indicator data collection includes front-end collection, which includes front-end Android and iOS clients, and indicator data collection is performed through the Emas SDK.

[0020] According to an embodiment of the present invention, the steps for collecting event tracking data include: integrating Postbe SDK to customize business tracking data on the front-end Android and iOS clients, Postbe reporting tracking log data to the Kafka message queue, and then collecting real-time Kafka data from the front-end Android and iOS clients to parse the tracking information.

[0021] According to an embodiment of the present invention, the indicator data collection includes backend collection, which includes: the backend application collecting data by integrating the cat client tool and using code-based data tracking.

[0022] According to an embodiment of the present invention, the steps of querying, analyzing, and re-aggregating include: creating a new application and creating a tracking detail table under the corresponding database of the ClickHouse database; if there are front-end metrics under the application, creating a new front-end tracking detail table; if there are special metrics, separating them from other metrics of the application and specifying a custom detail database and data table.

[0023] According to an embodiment of the present invention, the indicator data pre-aggregation method further includes the step of storing the tracking information into the ClickHouse database, including: the Cat client pushes the tracking detail JSON structured data associated with the tracking information into Kafka; the Cat server consumes Kafka messages, parses the JSON data and stores it in batches into the corresponding field columns of the tracking detail table in the ClickHouse database. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of an indicator data pre-aggregation method according to an embodiment of the present invention;

[0026] Figure 2 This is a flowchart of data acquisition according to an embodiment of the present invention;

[0027] Figure 3 This is the Arms acquisition and processing flow for data acquisition according to an embodiment of the present invention;

[0028] Figure 4 This is a flowchart illustrating how a server converts information into pre-aggregated metrics and stores them in InfluxDB, according to an embodiment of the present invention.

[0029] Figure 5 This is a flowchart of the data falling into ClickHouse process according to an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0031] According to an embodiment of the present invention, a method for pre-aggregating indicator data is provided. Figure 1 This is a flowchart of a method for pre-aggregating indicator data according to an embodiment of the present invention. Figure 1 As shown, the indicator data pre-aggregation method may include the following steps S12-S18.

[0032] First, in step S12, indicator data is collected in real time. Figure 2 This is a flowchart of data acquisition according to an embodiment of the present invention. (In conjunction with...) Figure 2 As shown, metric data collection can be divided into front-end collection and back-end collection. Front-end collection includes PC websites, H5 (short for HTML5) websites, and mini-programs, which collect metric data by embedding Arms JavaScript probes. Furthermore, front-end Android and iOS clients can collect metric data using the Emas SDK. For back-end collection, back-end applications can integrate Tomcat client tools to collect data through code-based tracking.

[0033] Then, at S14, event tracking is performed, and the collected tracking information is reported to the Tomcat server (hereinafter referred to as the cat server). Arms is a real-time monitoring platform for front-end applications. Leveraging the APIs provided by Arms, it can collect front-end system and page-level metric data with minute-level response times. For custom event tracking in the Arms front-end, custom events collected by the Armsjavascript (a high-level programming language) probe are reported to the log library in real time. The log library provides the ability to consume custom event log data in real time through consumer groups. Consumers can capture and parse the custom event information in the raw logs. For Android and iOS front-ends, the Android and iOS clients integrate Postbe SDK (Software Development Kit) to customize business event tracking. Postbe reports event log data to the Kafka (a distributed publish-subscribe messaging system) message queue, and then the Android and iOS clients consume Kafka data in real time to parse the event information.

[0034] At S16, the cat server converts the tracking information stored in the clickhouse database into pre-aggregated metric data and stores it in the InfluxDB database. The metric data is generated by aggregating the tracking information according to predetermined rules.

[0035] At point S18, the event tracking information is stored in the ClickHouse database as a detailed event tracking table. Data is queried and analyzed based on this table, and metrics are re-aggregated as needed and stored in the InfluxDB database. In some embodiments, the following steps can be used for querying, analysis, and re-aggregation: Create a new application and a detailed event tracking table under the corresponding database in the ClickHouse database; if the application has front-end metrics, create a new front-end event tracking table. If there are special metrics, separate them from other metrics in the application and specify a custom detailed database and table.

[0036] This invention combines common database technologies, including the time-series database InfluxDB, the caching database Redis, and the columnar database Clickhouse, to achieve real-time aggregation of predefined metrics and on-demand real-time aggregation of undefined metrics with minimal server resource investment (undefined metrics can obtain the latest data in real time after the requirements are defined, while their historical data is queried, aggregated, and dumped into the InfluxDB time-series database through the Clickhouse database), thereby enabling on-demand computation.

[0037] The technical solution of the present invention will be described below through specific embodiments. In general, the technical solution is implemented in the following three aspects: First, by using technologies such as CAT, ARMS, and EMAS, the indicator data is collected, transferred, and aggregated; second, by using Cat, the indicator data is pre-aggregated; and third, by using Clickhouse, real-time on-demand query analysis and on-demand re-aggregation are achieved.

[0038] First, by using technologies such as CAT, ARMS, and EMAS, we can achieve the collection, transfer, and aggregation of indicator data.

[0039] Data collection is a crucial and important part of the entire indicator monitoring system. Data collection is equivalent to the sensors in the industrial system and is the root of data-driven operations. Only by ensuring the comprehensiveness, timeliness, and accuracy of the collected data can the final data analysis results be credible and valuable.

[0040] Data acquisition clients are mainly divided into front-end acquisition and back-end acquisition. (Reference) Figure 2As shown: PC websites, H5 websites, and mini-programs collect data by embedding Alibaba Arms JavaScript probes; front-end Android and iOS clients collect data through the Emas SDK; and back-end applications collect data by integrating Cat client tools and using code-based tracking.

[0041] (1) Data collection of Arms front-end system and page metrics

[0042] Automatically obtains the PID of the front-end application. The scheduled task automatically obtains the PID of all front-end monitoring applications through the API. It supports data splitting, that is, different PIDs are hashed and then distributed to different collection servers for execution. It supports dynamic configuration of metric dimensions, and can configure the metrics and dimensions to be captured as needed.

[0043] Data collection resumes from where it was interrupted. Every 30 seconds, the time is automatically checked and collected into the database. After the application recovers from a failure, data will be retrieved from the most recent successful time point to ensure the integrity of the collected data.

[0044] To achieve failover, the instances automatically perform a health check by sending heartbeats between them every 60 seconds. If the heartbeat check fails, the instance is removed from the instance configuration table and reassigned to a normal instance using its PID, and data capture continues from the checkpoint.

[0045] For metric matching, page-based metrics are matched according to the page dimension, with "All" being the default for all pages; system-based metrics are matched according to the application, and the matched metrics are stored in the time-series database InfluxDB according to different storage strategies for minute, hour, and day.

[0046] (2) Arms front-end custom event tracking

[0047] The Arms data collection and processing flow has been redefined, meaning that custom events collected by the front-end Arms JavaScript probe are reported in real time to Alibaba Cloud Log Service (SLS). SLS provides real-time log consumption capabilities, allowing the consumption of custom event log data through ConsumerGroups. A ConsumerGroup contains multiple Consumers, each named according to its IP address and sequence number (ensuring no duplicate names within the consumer group). Different consumers create multiple threads (workers) on multiple servers. A consumer can consume multiple Shard data fragments, and a Shard can only be held by one consumer, as follows: Figure 3 As shown.

[0048] The consumer uses a poll method to fetch a set of logs (LogGroup), parses the custom tracking data in the raw logs (Log), and reports it to the Cat server in real time. The Cat server can then aggregate the tracking data in real time. To enable drill-down analysis of tracking metrics, we have also improved the Arms JavaScript probe, adding custom tag functionality. By default, it supports custom summation and averaging with three tags. Furthermore, to accommodate business expansion needs, the probe also allows adding multiple tags and metrics via a HashMap.

[0049] (3) Data collection of Emas front-end Android / iOS system and interface metrics

[0050] Eams is an Android / iOS front-end SDK provided by Alibaba, which comes with built-in monitoring and data collection capabilities. Front-end developers use Python web crawlers to scrape Eams system and UI-related metric monitoring data (such as system-related monitoring data: number of logged-in devices, average startup time, startup time percentile, startup time compliance rate, main link sliding frame rate compliance rate, overall crash rate, etc.; UI-related monitoring data: number of visits, page dwell time, exit rate, etc.), generating hourly and daily metric files and uploading them to OSS object storage. The metric collection instance then downloads the OSS metric files, parses them, and stores them in the time-series database InfluxDB.

[0051] (4) Front-end Android / iOS custom event tracking

[0052] This requires integration with Alibaba Cloud Mobile Development Platform (EMAS) (access to Alibaba Cloud EMAS Android / iOS SDK), creation of products and applications, and encapsulation of functions such as real-time sending of single data items and waiting for triggering after pushing data into a queue (triggering timings include app background activation, scheduled activation, and queue full activation). Considering factors such as data usage, power consumption, and server load, the mobile app defaults to batch reporting via push queue to avoid frequent network requests from the client. However, for particularly important data points with real-time requirements, individual instant reporting is also possible.

[0053] The reported event tracking log data first flows into the Kafka message queue, then the client instance consumes Kafka data in real time, parses the event tracking metrics, and reports them to the Cat server for aggregation calculation.

[0054] (5) Cat client data collection

[0055] The application uses the cat-clientjar package to implement data tracking via encoding. The Cat client places the tracked data into a local asynchronous message queue. The sender thread retrieves messages from the message queue and reports them to the Cat server, thus completing data collection. For performance reasons, the underlying layer uses a Netflix TCP long connection to transmit data with the Cat server.

[0056] Therefore, the following beneficial effects can be achieved:

[0057] 1) Implemented the servlet filter interface to parse the HTTP header: global trace ID, UID (user ID, device ID under Android / iOS), SID (current session ID), event tracking number, baggage data, etc.

[0058] 2) Implement HttpClient interception: Add HTTP header link ID, client call address and domain, and set baggage to be passed down with the data.

[0059] 3) Implement data transfer via MQ and microservice RPC. The MQ producer header contains a baggage, which is then parsed by the consumer. The RPC implementation principle is the same as that of MQ.

[0060] 4) Implement log.error() to upload application error logs to the Cat server. In the log4j log, inherit the AppenderSkeleton class and override the append method to complete the error log reporting.

[0061] 5) Implemented custom summation, averaging, and counting statistics: Provided new methods logMetricForSum(), logMetricForAvg(), logMetricForCount, and the general metric collection method logMetric(). The underlying layer extended the tag functionality for Metric messages. Each method corresponds to a different Metric type. The Cat server receives the message and performs aggregation calculations according to the Metric type.

[0062] 6) API calls support the addition of tags.

[0063] 7) Business processing results, including business return codes, return messages, status, etc. The Cat client's MessageTree supports extended business attributes, and business processing results are set to be reported to the message tree.

[0064] 8) Log events for requests and results, including modifications to Memcached, Redis, Controllers, Services, URLs, etc.

[0065] Second, implement indicator data pre-aggregation through Cat.

[0066] InfluxDB stores pre-aggregated metric data, while Clickhouse stores raw event tracking information. Metric data is generated by aggregating event tracking information according to predetermined rules. Event tracking content includes event tracking number and name, business attributes, and data values. All event tracking points must be applied for and configured on the event tracking and metric management platform for unified standardization, management, and recording. On this platform, one or more metrics can be added to the applied event tracking points. Each metric corresponds to a pre-aggregation rule, which is a logical expression derived from the combination of business attributes.

[0067] The client transmits the tracking information to the server in real time via TCP protocol. The server then converts this information into pre-aggregated metrics and stores them in InfluxDB. This operation is mainly completed by four threads.

[0068] refer to Figure 4 As shown, the minute-level indicator report generation thread: The server uses stream processing to process each received tracking information. The general processing flow is: matching tracking configuration -> matching indicator set -> aggregation rule filtering -> updating minute-level indicator report; the minute-level indicator report is generated and updated in memory in real time as the stream processing proceeds. To avoid the risk of memory overflow due to excessive indicator data, the retention period of the indicator report in memory is configurable. The minute-level indicator report records statistical data and some auxiliary information for each indicator.

[0069] Minute-level metric report persistence thread: Data for the same metric may be distributed across multiple server nodes. Each server node is responsible for processing and uploading a portion of the metrics. This thread will aggregate the metric data it is responsible for from all nodes into memory, and then merge them into a single metric report and write it to the local disk. This approach can prevent the loss of a large amount of metric data in memory due to service crashes, and it can also accumulate and save metric data when InfluxDB is unavailable, and then compensate and upload it after InfluxDB recovers.

[0070] The thread responsible for generating and aggregating hourly and daily metrics stores the current hourly and daily metric data in a Redis cache database. It uses Redis commands to calculate these metrics, employing both numerical and set-based methods. Numerical data is incrementally calculated using the `inc` Redis command, while set-based data is adaptively selected or converted to Redis's `set`, `bit`, or `hyperLogLog` storage structures based on its volume, and then its intersection and union are calculated. This thread reads and parses minute-level metric report files from the local disk into memory, converts them into individual metric data entries, adds them to the InfluxDB asynchronous upload queue, and merges these entries into the hourly metrics. At the end of the hour, the thread retrieves the hourly metrics from Redis, adds them to the InfluxDB asynchronous upload queue, and merges them into the daily metrics. At the end of the day, the thread retrieves the daily metrics and adds them to the InfluxDB asynchronous upload queue.

[0071] Metrics upload thread: This thread continuously reads metric data from the InfluxDB upload queue and uploads this data to InfluxDB in batches. To avoid metric data loss, the upload queue is a blocking queue. When there is no free space in the queue, add operations will block and wait until there is free space. If InfluxDB becomes unavailable, the upload operation stops and begins to continuously check the InfluxDB status. Metric data will only continue to be uploaded after the InfluxDB status returns to normal.

[0072] Third, ClickHouse enables real-time on-demand query analysis and on-demand re-aggregation.

[0073] The mapping relationship between CAT monitoring platform elements and ClickHouse data tables:

[0074] ClickHouse data tables are partitioned by date and sorted by tracking point number.

[0075] Tracking points are categorized by type: system / page / interface / event. By default, the details of tracking points of the same type within a system are stored in a single table in ClickHouse.

[0076] Details of the same type of event tracking points from multiple systems can also be stored in one table.

[0077] When the amount of data in a ClickHouse event tracking table becomes very large, it impacts machine performance. Therefore, the table should be upgraded to a distributed table.

[0078] If the amount of data at a single tracking point in the system is very large, the tracking point can be extracted and treated as a separate distributed table.

[0079] Establish a route configuration mapping for the correspondence between cat and clickhouse to meet the data storage needs of event tracking details in various scenarios.

[0080] (1) Data entry into ClickHouse process

[0081] like Figure 5 As shown, Kafka message middleware is used for streaming data writing. Kafka creates different partitions (functions) for different applications. The CAT client pushes the event tracking details JSON structured data into Kafka. The CAT server consumes Kafka messages, parses the JSON data, and batches it into the corresponding fields of the ClickHouse event tracking details table.

[0082] (2) Data Query

[0083] The event tracking details table can be aggregated based on any column. A specific time period must be selected before aggregation for the data to be effective. The essence of data querying can be viewed as a process of continuously reducing the data range, thereby speeding up the query. Furthermore, data tagging further minimizes the range of data that needs to be decompressed and calculated.

[0084] minmax.idx (partition index) --- primary.idx (first-level index) --- skip_idx.idx (second-level index) --- .mrk (tag file) ---- .bin (compressed data file).

[0085] For queries on non-indexed columns, the system scans columns according to the smallest data range. Since the content of the ClickHouse data table is compressed, the data range can be quickly located, and the data needed for the query can be retrieved: querying lists and specific individual data; performing aggregation operations such as summation, average, maximum and minimum values, total, frequent occurrence, and existence; and performing sample approximation calculations.

[0086] If a query does not specify any WHERE clause, or if a WHERE clause is specified but the condition does not match any index (partition index, primary index, or secondary index), then the data scan range cannot be reduced beforehand. Subsequent data queries will then scan all partition directories and the maximum range of index segments within those directories. Although the scan range cannot be reduced, query performance can still be improved by using data tagging and multi-threading to read multiple compressed data blocks simultaneously.

[0087] (3) Multidimensional analysis

[0088] ClickHouse's column-oriented storage model is suitable for analyzing columnar data. InfluxDB has a limited number of tag business attributes that can be stored, and it also has a limit on the number of groups due to performance issues. When performing multi-dimensional data analysis, ClickHouse can extract and transform detailed tracking data and build various data models according to themes.

[0089] Business results are calculated from detailed data and data models according to time periods (minutes, hours, days, weeks, months, etc.) and stored in the analysis table to meet various types of big data analysis scenarios.

[0090] (4) Re-aggregation

[0091] For aggregation operations that InfluxDB cannot perform, we still need to query detailed data and use SQL to customize aggregation scenarios to obtain the aggregation results we need.

[0092] for example:

[0093] Class A and Class B have 20 and 30 students respectively. Today's average performance scores are 85 and 90 respectively. The goal is to obtain the average score of all students.

[0094] InfluxDB stores aggregated data and doesn't include the number of students in each class: (85+90) / 2 is clearly not the answer we want. ClickHouse stores the score record data and can calculate the number of students in each class, similar to the following operation:

[0095]

[0096]

[0097] (5) Historical Review

[0098] When an event tracking point was created a long time ago, but the metric is newly created, the historical data for that metric in InfluxDB cannot be retrieved. ClickHouse allows you to use event tracking details to backtrack historical data for metrics.

[0099] The technical solution of this invention, on the one hand, makes reasonable use of the time-series database InfluxDB: a) Instance partitioning, supporting the creation of multiple InfluxDB instances, which can be InfluxDB cluster instances or single-machine versions depending on business access; b) Database partitioning, creating databases by system number, routing to specific InfluxDB instances + specific databases based on the system to which the metric belongs, with configurable routing rules, and the routing rules can be refined to specific metrics when a hot data point appears; c) Table partitioning, where one metric corresponds to one Measurement in the InfluxDB database, stored according to storage strategies of minute, hour, and day. On the other hand, by reasonably using InfluxDB (storing daily, hourly, and minute-level statistical data) and the caching database Redis (real-time calculation of hourly and daily-level metric statistical data, with minute-level data processed in memory), a low-cost data storage and monitoring time-series solution is achieved without the need to build a big data system. It also requires fewer and cheaper resources. Clickhouse is more expensive but is not used to support real-time scenarios; instead, it is used to calculate a small amount of historical data on demand. In addition, by cleverly combining pre-aggregation and on-demand aggregation, the query requirements of high-concurrency scenarios for analyzing data can be met, thereby achieving unified output of predefined indicators and unknown indicators (calculated on demand).

[0100] In summary, the beneficial effects of the above-mentioned technical solution of the present invention include:

[0101] 1) In terms of data collection, make full use of existing technologies and achievements, focusing on data extraction and aggregation rather than replacing existing data collection solutions and rebuilding them.

[0102] 2) Regarding metric aggregation, whatever metrics are needed, the corresponding pre-aggregation rules can be configured on the unified tracking metric configuration platform. The configured metrics are all calculated in real time and generate results in real time. By making reasonable use of the full-link monitoring Cat, combined with mature lightweight technologies such as InfluxDB and Redis, real-time stream computing effects can be achieved with a small amount of server resources. Moreover, it has strong scalability and can be horizontally expanded indefinitely as needed.

[0103] 3) Pre-aggregate the required data on demand, instead of pre-aggregating all data at once. When new aggregation needs arise later, the data can be converted into pre-aggregated metric data based on historical data to provide high-concurrency query services.

[0104] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for pre-aggregation of indicator data, characterized by, include: Real-time collection of indicator data; Collect event tracking data and report the collected tracking information to the CAT server. Through multiple threads, the cat server transforms the event tracking information into pre-aggregated metric data and stores it in the InfluxDB database. The metric data is generated by aggregating the event tracking information according to predetermined rules. The metric data is then matched by application, and the matched metric data is stored in the InfluxDB database according to different time storage strategies. The multiple threads include a metric generation and hourly / daily metric aggregation thread: converting minute-level metric report files into metric data and adding it to the InfluxDB database asynchronous upload queue, while merging the metric data into hourly metrics; when the hour ends, retrieving the hourly metric data from the cache database Redis, adding it to the InfluxDB database asynchronous upload queue, and merging it into the daily metrics; when the day ends, retrieving the daily metrics and adding them to the InfluxDB database asynchronous upload queue, wherein Redis stores the current hourly and daily metric data; The event tracking information is stored in the ClickHouse database in the form of event tracking detail tables. Data is queried and analyzed based on the event tracking detail tables, and indicators are re-aggregated and stored in the InfluxDB as needed. Specifically, the Kafka message middleware Cat client is used to push the structured event tracking detail data into Kafka. The Cat server consumes Kafka messages and parses the structured data into the corresponding field columns of the event tracking detail table in the ClickHouse database. After selecting a specified time, the event tracking detail table is aggregated based on any column to query data. The steps for querying, analyzing, and re-aggregating include: creating a new application and creating the event tracking detail table under the corresponding database of the ClickHouse database; if there are front-end metrics under the application, creating a new front-end event tracking detail table; if there are special metrics, separating them from other metrics of the application and specifying a custom detail database and data table. For aggregation operations that cannot be completed using the InfluxDB aggregation data, the corresponding aggregation results are obtained by querying the detailed data stored in the ClickHouse database.

2. The indicator data pre-aggregation method according to claim 1, characterized in that, The plurality of threads also includes: Minute-level indicator report generation thread: The cat server uses stream processing to process each piece of the received tracking information, and sequentially performs matching tracking configuration, matching indicator set, aggregation rule filtering, and updating minute-level indicator report, thereby generating and updating minute-level indicator report in memory in real time as stream processing occurs. Minute-level indicator report persistence thread: used to aggregate indicator data from all nodes into memory, and then merge it into an indicator report and write it to the local disk; Metrics upload thread: Iteratively reads the metric data from the upload queue of the InfluxDB database and uploads the metric data to the InfluxDB database in batches.

3. The indicator data pre-aggregation method according to claim 1, characterized in that, Indicator data collection includes front-end collection, which includes: PC websites, H5 websites, and mini-programs collect metric data by embedding Arms JavaScript probes.

4. The indicator data pre-aggregation method according to claim 3, characterized in that, The steps for collecting the event tracking points include: The custom events collected by the Arms javascript probe are reported to the log library in real time, and the log library provides the ability to consume custom event log data in real time through consumer groups; Consumers capture and parse custom event tracking information from raw logs.

5. The indicator data pre-aggregation method according to claim 1, characterized in that, Indicator data collection includes front-end collection, which includes: The front-end Android and iOS clients collect metric data using the Emas SDK.

6. The indicator data pre-aggregation method according to claim 5, characterized in that, The steps for collecting the event tracking points include: The front-end Android and iOS clients integrate Postbe SDK to customize business event tracking. Postbe reports event tracking log data to a Kafka message queue, and then collects real-time Kafka data from the front-end Android and iOS clients to parse the event tracking information.

7. The indicator data pre-aggregation method according to claim 1, characterized in that, Indicator data collection includes backend collection, which includes: The backend application collects data by integrating Tomcat client tools and embedding data points in the code.

8. The indicator data pre-aggregation method according to claim 1, characterized in that, It also includes the step of storing the tracking information in the ClickHouse database, including: The CAT client pushes the structured JSON data of the tracking details associated with the tracking information into Kafka; The cat server consumes Kafka messages, parses JSON data, and batches it into the corresponding field columns of the event tracking details table in the ClickHouse database.

Images