Event-driven data hierarchical model statistical method

Abstract

The invention discloses an event-driven data hierarchical model statistical method, which relates to the technical field of data processing, and adopts a self-adaptive processing mechanism capable of sensing context characteristics of a data stream, and a system performs data hierarchical model statistical analysis according to real-time throughput and data attributes. According to the method, an event-driven mode, a timed task scheduling mode and a micro-batch processing mode are flexibly and automatically switched, so that the high throughput demand of historical data backtracking is effectively considered while low-delay real-time calculation is ensured, and the optimal balance between the resource utilization rate and the calculation efficiency is realized. And the user retention and active indexes are processed by introducing a bitmap-based combined storage structure and displacement arithmetic logic, so that the storage space occupation is greatly compressed, the complex set operation is converted into efficient binary bit operation, and the query performance of cross-cycle statistics is remarkably improved. In addition, a multi-level cache mechanism is utilized to optimize a streaming association process, and the access pressure on an external dimension database is effectively reduced.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and specifically relates to an event-driven statistical method for data hierarchical models. Background Technology

[0002] With the rapid development of mobile internet and IoT technologies, the amount of data generated by various business systems has exploded. Real-time processing and value mining of massive amounts of data have become the core driving force for enterprise digital transformation. Existing data statistics technologies still have many limitations when facing complex and ever-changing business scenarios. On the one hand, traditional architectures usually adopt a single processing mode, either relying solely on offline batch processing, resulting in high data latency and difficulty in meeting real-time requirements, or using full streaming computing, which consumes huge resources and lacks stability when facing historical data backtracking or ultra-high throughput, and lacks the flexibility to automatically switch processing modes based on data context characteristics, leading to wasted computing resources or system blockage. On the other hand, when calculating core indicators such as user retention and activity, conventional methods often directly perform real-time correlation queries based on massive detailed data or use simple set operations, which not only occupy expensive storage space, but also cause the performance of cross-period queries and aggregation calculations to drop sharply as the user base expands, seriously affecting the response speed of data services and the decision-making efficiency of downstream businesses. In addition, when performing dimensional correlations during data flow, frequent queries back to the source database can easily cause I / O bottlenecks, thereby causing delays in the overall computing chain. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides an event-driven data hierarchical model statistical method to solve the aforementioned technical problems.

[0004] An event-driven statistical method for hierarchical data models includes the following steps:

[0005] Establish a data model that includes a raw data layer, a static configuration data layer, a detailed data layer, and an application data service layer, and configure the message topics and data structure definitions corresponding to the data flow between each layer;

[0006] The system receives raw events from the business system via an event bus, monitors the contextual characteristics of the current data stream in real time, and determines the current data processing mode based on the contextual characteristics; the data processing mode includes at least an event-driven mode.

[0007] When the event-driven mode is determined, the message topic is monitored in real time and the raw events are written to the raw data layer.

[0008] Read the data from the original data layer, and based on the streaming association algorithm, obtain the corresponding dimension information from the static configuration data layer for association and completion, generate wide table data and write it into the detail data layer;

[0009] Real-time statistical algorithms are executed based on the detailed data layer to generate business metrics and update them to the application data service layer.

[0010] The statistical results of the application data service layer are stored, and result events containing updated results are published for downstream subscription and consumption.

[0011] Preferably, determining the current data processing mode based on the context features specifically includes the following steps:

[0012] The acquired data stream context features should include at least data throughput, business latency tolerance, and whether it is historical backtracking data;

[0013] When the data throughput exceeds the preset batch processing threshold or the data attribute is determined to be historical backtracking data, switch to the scheduled task mode and call the offline batch computing engine for processing;

[0014] When the business latency tolerance of the data stream is determined to be low latency, the system switches to the event-driven mode and calls the real-time stream computing engine for processing.

[0015] When the data characteristics fall between the trigger conditions of the scheduled task mode and the event-driven mode, switch to micro-batch processing mode.

[0016] Preferably, in the application data service layer, the storage of user retention metrics adopts a bitmap-based combined storage structure, specifically including the following steps:

[0017] Define a user retention data model that includes a unique user identifier, registration date, and a binary bit set for recording active status over multiple consecutive days;

[0018] The binary bit group is divided into at least two long integer numerical fields for storage;

[0019] The combination of bits in the numerical field is used to map the user's daily activity status within a preset time period after registration;

[0020] Specifically, the retention status of the first period is stored in the first long integer field, and the retention status of subsequent periods is stored in the second long integer field.

[0021] Preferably, when the real-time statistical algorithm updates the user retention metric, it specifically includes the following steps:

[0022] When a user behavior event is detected in the detailed data layer, the existing user retention data model is queried from the application data service layer based on the user's unique identifier.

[0023] If the query result is empty, initialize a new user retention data model and set both long integer numeric fields to zero;

[0024] Calculate the difference in days between the current event time and the user's registration date;

[0025] Based on the difference in the number of days, the corresponding long integer numerical field and the specific binary bit position are located through displacement operation, and the binary bit position is set to 1.

[0026] Write the updated user retention data model back to the application data service layer.

[0027] Preferably, the process of converting the data from the original data layer into wide table data using the streaming association algorithm specifically includes the following steps:

[0028] Listen to the message topic stream corresponding to the raw data layer;

[0029] When processing message topic stream data, query the dimension table data of the static configuration data layer based on the preset primary key information;

[0030] The query process employs a multi-level caching mechanism: it prioritizes querying the local memory cache with automatic expiration; if the local memory cache is not found, it queries the external database and uses the query results to populate the local memory cache.

[0031] Write the associated wide table records to the detail data layer and simultaneously publish the corresponding wide table message topic.

[0032] Preferably, a cache consistency maintenance mechanism is adopted for queries on the static configuration data layer, wherein the mechanism includes at least one of the following steps:

[0033] Set the write expiration time for the local memory cache to ensure that the cached dimension data is refreshed regularly;

[0034] Listen to the change message stream of the static configuration data layer. When a message indicating a change in dimension data is captured, proactively invalidate the primary key or update the corresponding entry in the local memory cache.

[0035] Preferably, the active user statistics in the application data service layer are obtained using a bitmap compression algorithm, specifically including the following steps:

[0036] Map the set of unique identifiers of active users within the same statistical period to a bitmap object;

[0037] The bitmap object is serialized to generate a compressed byte array;

[0038] Store the byte array in a statistics table of the application data service layer;

[0039] When performing cross-period statistics, the corresponding byte array is read, deserialized into a bitmap object, and subjected to a set of logical AND and logical OR operations to calculate the continuous active user or cumulative active user metrics.

[0040] Preferably, when the scheduled task mode is selected, the following steps are performed:

[0041] Confirm that the data to be processed belongs to historical backtracking data or a large-scale data set;

[0042] Initiate batch processing logic and input the data to be processed into the offline batch computing engine;

[0043] The offline batch computing engine is used to perform full aggregation calculations on the data, generate statistical indicators, and write them into the application data service layer.

[0044] Preferably, the pre-configuration of the message topic and data structure mapping relationship for data flow between layers specifically includes the following steps:

[0045] Define message topic identifiers to represent different business types;

[0046] For each message topic, a corresponding data structure definition is established, which includes business attribute fields and their data types;

[0047] During the layered processing, the received message topic is parsed into a structured data object according to the data structure definition, and the fields of the structured data object are mapped to the storage fields of each layer's database.

[0048] Preferably, when storing the statistical results of the application data service layer and publishing result events containing the updated results for downstream subscription and consumption, the method further includes the following steps:

[0049] The generated business metrics are written to the physical database of the application data service layer via the storage adapter;

[0050] After the data writing is complete, construct a message object containing the updated indicator data;

[0051] The message object is published through the event bus, and the message object is exposed to downstream components through a preset interface to trigger subsequent business logic.

[0052] The beneficial effects of this invention are as follows: Employing an adaptive processing mechanism capable of sensing data flow context characteristics, the system flexibly switches automatically between event-driven mode, scheduled task mode, and micro-batch processing mode based on real-time throughput and data attributes. This ensures low-latency real-time computation while effectively addressing the high-throughput requirements of historical data backtracking, achieving an optimal balance between resource utilization and computational efficiency. Furthermore, by introducing a bitmap-based combined storage structure and bitwise operation logic to handle user retention and activity metrics, storage space usage is significantly reduced, transforming complex set operations into efficient binary bit operations, and significantly improving query performance for cross-period statistics. In addition, a multi-level caching mechanism optimizes the streaming association process, effectively reducing access pressure to external dimension databases and ensuring data flow stability in high-concurrency scenarios. A publish-subscribe mechanism enables loose coupling and real-time linkage between upstream and downstream businesses. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart illustrating the steps of an event-driven data hierarchical model statistical method provided by this invention;

[0055] Figure 2 The flowchart illustrates the steps for determining the current data processing mode in an event-driven data hierarchical model statistical method provided by this invention. Detailed Implementation

[0056] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0057] The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. To simplify the disclosure of the present invention, the components and arrangements of specific examples are described below. Of course, these are merely examples and are not intended to limit the present invention.

[0058] The embodiments of the invention will now be described in detail with reference to the accompanying drawings.

[0059] like Figure 1 As shown, an event-driven data hierarchical model statistical method includes the following steps:

[0060] Establish a data model that includes a raw data layer, a static configuration data layer, a detailed data layer, and an application data service layer, and configure the message topics and data structure definitions corresponding to the data flow between each layer;

[0061] The system receives raw events from the business system via an event bus, monitors the contextual characteristics of the current data stream in real time, and determines the current data processing mode based on the contextual characteristics; the data processing mode includes at least an event-driven mode.

[0062] When the event-driven mode is determined, the message topic is monitored in real time and the raw events are written to the raw data layer.

[0063] Read the data from the original data layer, and based on the streaming association algorithm, obtain the corresponding dimension information from the static configuration data layer for association and completion, generate wide table data and write it into the detail data layer;

[0064] Real-time statistical algorithms are executed based on the detailed data layer to generate business metrics and update them to the application data service layer.

[0065] The statistical results of the application data service layer are stored, and result events containing updated results are published for downstream subscription and consumption.

[0066] In practical implementation, taking real-time data monitoring of e-commerce promotions as an example, the system receives massive transaction events through the event bus and monitors the current data stream in real time to detect contextual characteristics requiring high concurrency and low latency. Therefore, the current processing mode is determined to be event-driven. The streaming association algorithm uses a stream computing engine like Flink. This engine consumes the mainstream events from the raw layer, parses and extracts the association keys: the system reads order stream data, accesses the dimension data layer through asynchronous I / O and multi-level caching mechanisms to obtain user membership levels and product category information, and integrates the scattered data fields to generate wide table data, which is then written to the detailed data layer. The wide table data defined here is essentially a denormalized data table structure. Its technical feature is that it pre-associates and flattens the raw factual data with attribute data from multiple dimensions before writing it to storage, forming a single-row wide record containing all the fields required for subsequent analysis. The basic principle of this design is to trade space for time. By consuming computing resources and storage space in advance during the data writing stage to complete data assembly, it completely eliminates the huge IO overhead and computational latency when performing complex multi-table association queries in the application data service layer or OLAP engine, thereby reducing the query response time from seconds to milliseconds.

[0067] In scenarios requiring adjustments to last quarter's algorithm logic or data reruns, if the system identifies that the input data has historical backtracking attributes or that the message backlog exceeds a preset threshold, it automatically switches to a scheduled task mode. This utilizes an offline batch processing engine like Spark to perform high-throughput parallel computation on historical data shards stored in a distributed file system, avoiding overloading the real-time computing pipeline. Based on the wide table data in the detailed layer, the system executes real-time aggregation algorithms to generate business metrics such as GMV and conversion rates, updating them to the ADS layer. The results are persisted using storage adapters, and a result event containing the updated results is published via a message queue, directly triggering downstream marketing systems to automatically issue coupons or update recommendation strategies.

[0068] Compared to existing technologies, this solution achieves dynamic adaptation of computing patterns through context feature awareness, which not only ensures millisecond-level real-time visibility of hot data, but also solves the resource efficiency and stability issues when backtracking cold data. Furthermore, the wide table preprocessing mechanism, combined with the layered architecture, significantly reduces the query complexity of the data warehouse, and the publish-subscribe mechanism achieves complete decoupling of data production and consumption operations.

[0069] like Figure 2 As shown, more specifically, determining the current data processing mode based on the context features includes the following steps:

[0070] The acquired data stream context features should include at least data throughput, business latency tolerance, and whether it is historical backtracking data;

[0071] When the data throughput exceeds the preset batch processing threshold or the data attribute is determined to be historical backtracking data, switch to the scheduled task mode and call the offline batch computing engine for processing;

[0072] When the business latency tolerance of the data stream is determined to be low latency, the system switches to the event-driven mode and calls the real-time stream computing engine for processing.

[0073] When the data characteristics fall between the trigger conditions of the scheduled task mode and the event-driven mode, switch to micro-batch processing mode.

[0074] The system first collects contextual features in real time through monitoring probes integrated into the message middleware and computing cluster. The collection methods include: reading the number of writes per second and the backlog of consumer groups in the message queue through the JMX interface to quantify data throughput; determining the business latency tolerance, such as "second-level visibility" or "T+1 delivery", by reading the SLA parameters in the metadata configuration; and determining whether it is historical backtracking data by parsing specific identifiers in the data packet header or the difference between the occurrence time and processing time of the comparison item.

[0075] In normal real-time streaming, when the system detects that the data stream is generated incrementally in real time, the LA value is below the preset safety level, and the SLA requires millisecond-level response, the system locks into event-driven mode and calls streaming engines such as Flink for low-latency processing of each data item. Under high load or during data supplementation, if the LA value exceeds the batch processing threshold or a "REPLAY" backtracking flag is detected (the batch processing threshold is over 1 million data items), the controller automatically switches to scheduled task mode and starts offline engines such as Spark SQL to perform high-throughput full computation on the sharded data stored on HDFS. It is worth noting that when the detected traffic characteristics fall between the two, the system will switch to micro-batch processing mode. This mode is defined as discretizing the continuous, unbounded real-time data stream into tiny time windows in the time dimension, usually set to 500ms to 5s, and performing atomic batch processing on the RDD or data set within the window. Its design principle is to take advantage of the memory vectorization computing of small batch data to establish a buffer between the low latency of the real-time stream and the high throughput of the offline batch, similar to the running mechanism of Spark Streaming. It sacrifices a little real-time performance to achieve several times the throughput improvement and effectively resist back pressure.

[0076] More specifically, in the application data service layer, the storage of user retention metrics adopts a bitmap-based composite storage structure, specifically including the following steps:

[0077] Define a user retention data model that includes a unique user identifier, registration date, and a binary bit set for recording active status over multiple consecutive days;

[0078] The binary bit group is divided into at least two long integer numerical fields for storage;

[0079] The combination of bits in the numerical field is used to map the user's daily activity status within a preset time period after registration;

[0080] Specifically, the retention status of the first period is stored in the first long integer field, and the retention status of subsequent periods is stored in the second long integer field.

[0081] The system predefines a storage model that includes a user's unique identifier, registration date, and several long integer fields. In Example 1, when the system receives an activity event of a user on the 5th day after registration, it calculates the day difference DayDiff=5, and according to the bitwise operation formula Index=Day Diff / 64, it obtains Index=0, which locates the first long integer field, which stores the status of days 0-63. The offset is obtained by Offset=Day Diff%64, which is 5. Then, the bit setting operation Long[0]=Long[0]|(1<<5) is executed to set the 5th bit of the first field from 0 to 1, thus completing the status update. In Example 2, when an active event on the 70th day after a user's registration is received, Day Diff = 70 is calculated. At this time, Index = 70 / 64 = 1. The system automatically locates the second long integer field, which stores the status from day 64 to day 127. The offset Offset = 70%64 = 6 is calculated, and Long[1] = Long[1]|(1<<6) is executed, thereby realizing cross-cycle status recording. The basic principle of this design is to utilize the efficiency and compactness of the computer's underlying binary bit operations to map the user's daily active status to a single bit. Compared to using Set <string>This solution can store 70 date strings or JSON objects, and only requires two Long type fields to fully express the user's activity trajectory over 128 days, with a storage space compression ratio of more than 50 times. At the same time, the speed of retention rate calculation using bit logic operations is much faster than collection traversal.

[0082] More specifically, when the real-time statistical algorithm updates the user retention metric, it includes the following steps:

[0083] When a user behavior event is detected in the detailed data layer, the existing user retention data model is queried from the application data service layer based on the user's unique identifier.

[0084] If the query result is empty, initialize a new user retention data model and set both long integer numeric fields to zero;

[0085] Calculate the difference in days between the current event time and the user's registration date;

[0086] Based on the difference in the number of days, the corresponding long integer numerical field and the specific binary bit position are located through displacement operation, and the binary bit position is set to 1.

[0087] Write the updated user retention data model back to the application data service layer.

[0088] As a wide table stream that listens to the detailed data layer of the consumer, the real-time computing engine receives the first behavioral event generated by a newly registered user. When the system queries the application data service layer based on the user's unique identifier and returns an empty result, it initializes the retention model containing two long integer fields and initializes all the values ​​to zero. It calculates the difference between the current time and the registration date, which is 0. It uses the shift instruction to locate the 0th bit of the first numerical field and marks it as 1, thus completing the initialization and writing of the model.

[0089] For existing users who become active again on the 35th day after registration, the system queries and retrieves the existing model, calculates DayDiff=35, uses integer division 35 / 64=0 to locate the first field, and performs modulo operation 35%64=35 to locate the 35th bit. It then performs a logical OR operation Value|(1<<35), sets the corresponding bit to 1, and writes the updated model back to the database. The core principle of this design is to transform the traditional complex set state change operation into a CPU bit instruction operation with a constant time complexity of O(1), avoiding the cumbersome process of full deserialization and element appending and reserialization of variable-length lists or sets in traditional schemes. Furthermore, the fixed-length numerical field results in extremely low network transmission bandwidth usage, significantly improving the transaction processing capacity per second of the real-time link.

[0090] More specifically, the process of transforming the data from the original data layer into wide table data using the streaming association algorithm includes the following steps:

[0091] Listen to the message topic stream corresponding to the raw data layer;

[0092] When processing message topic stream data, query the dimension table data of the static configuration data layer based on the preset primary key information;

[0093] The query process employs a multi-level caching mechanism: it prioritizes querying the local memory cache with automatic expiration; if the local memory cache is not found, it queries the external database and uses the query results to populate the local memory cache.

[0094] Write the associated wide table records to the detail data layer and simultaneously publish the corresponding wide table message topic.

[0095] The system monitors the message stream of the raw data layer. When processing events containing frequently accessed hot keys, such as popular product IDs, the operator prioritizes querying the local memory cache integrated within the compute node's JVM process, such as an LRU cache implemented based on Guava or Caffeine, to achieve data hit and association, completely avoiding network overhead. For scenarios where long-tail data or cold-start data causes local cache misses, the system automatically triggers a back-to-origin query process. In this embodiment, this back-to-origin query process specifically refers to the behavior where, when the high-speed cache layer cannot provide the required data, the system degrades and initiates a network request to the persistent storage layer to obtain the data, and then fills the obtained result back into the local cache for subsequent reuse. The core principle of this design is to leverage the temporal locality of data access, using high-speed memory access to shield against low-speed database I / O, thus decoupling computational throughput from external storage performance.

[0096] More specifically, a cache consistency maintenance mechanism is adopted for queries on the static configuration data layer, wherein the mechanism includes at least one of the following steps:

[0097] Set the write expiration time for the local memory cache to ensure that the cached dimension data is refreshed regularly;

[0098] Listen to the change message stream of the static configuration data layer. When a message indicating a change in dimension data is captured, proactively invalidate the primary key or update the corresponding entry in the local memory cache.

[0099] The system first employs a passive expiration strategy. For low-frequency change data such as administrative division codes, an expiration time is configured after writing to local heap memory, for example, set to 30 minutes. This forces cached entries to expire periodically and be reloaded from the source, thus ensuring eventual data consistency with minimal system overhead. Simultaneously, for high-sensitivity data such as product prices and promotional status, the system uses an active expiration strategy. By subscribing to database change log streams, such as using Canal to collect CDC messages generated by MySQLBinlog, once the listener captures an UPDATE or DELETE event for a specific primary key, it immediately performs cache eviction or hot reloading operations for that primary key in the compute node's memory. The core principle of this design lies in using an event-driven mechanism to break cache isolation, compressing the time window of data inconsistency to millisecond-level message transmission latency while maintaining microsecond-level read speeds in local memory.

[0100] More specifically, the active user statistics in the application data service layer are obtained using a bitmap compression algorithm, specifically including the following steps:

[0101] Map the set of unique identifiers of active users within the same statistical period to a bitmap object;

[0102] The bitmap object is serialized to generate a compressed byte array;

[0103] Store the byte array in a statistics table of the application data service layer;

[0104] When performing cross-period statistics, the corresponding byte array is read, deserialized into a bitmap object, and subjected to a set of logical AND and logical OR operations to calculate the continuous active user or cumulative active user metrics.

[0105] In its implementation, the system first performs an object mapping and serialization process: mapping the hundreds of millions of active user integer IDs for the day to a bitmap container, automatically compressing sparse data segments using run-length encoding, generating a compact binary byte array, and persistently storing it in an analytical database as a BLOB, compressing the original storage space of several gigabytes to the MB level. In cross-period calculation scenarios, when calculating the active user metric for seven consecutive days, the system reads binary data blocks from the past seven days in parallel, deserializes them into memory bitmap objects, and directly executes logical AND and set operations on Bitmap_Day1&Bitmap_Day2...&Bitmap_Day7 using CPU bit logic instructions, outputting the intersection result. The core principle of this design lies in utilizing a hierarchical compressed bitmap data structure, simultaneously solving the space waste problem of traditional raw bitmaps under sparse data and the full traversal overhead of traditional list structures when performing intersection and union operations.

[0106] More specifically, when the scheduled task mode is selected, the following steps are performed:

[0107] Confirm that the data to be processed belongs to historical backtracking data or a large-scale data set;

[0108] Initiate batch processing logic and input the data to be processed into the offline batch computing engine;

[0109] The offline batch computing engine is used to perform full aggregation calculations on the data, generate statistical indicators, and write them into the application data service layer.

[0110] In its implementation, the system first determines triggering conditions through metadata tags or backlog thresholds. For historical backtracking scenarios where all business metrics from the previous year need to be recalculated due to changes in algorithm logic, the system automatically suspends the real-time consumption thread and starts batch processing jobs based on Spark or MapReduce. It then reads historical raw data shards stored in a columnar storage format in a distributed file system in parallel, constructing a directed acyclic graph to execute the Map-Shuffle-Reduce computation process for the entire dataset. For scenarios where severe backlogs in the real-time pipeline lead to large-scale data set remediation, such as when lag exceeds five times the system's processing capacity, the system dumps the raw messages from the backlog period into offline files. Leveraging the high-bandwidth I / O advantage of the batch processing engine, it performs a one-time batch aggregation, generates statistical metrics, and writes them to the application data service layer via batch loading. The core principle of this design lies in optimizing sequential disk read / write and memory vectorized computation using batch processing, sacrificing second-level real-time performance for maximum data throughput and computational stability.

[0111] More specifically, the pre-configuration of the message topic and data structure mapping relationship between data flows between layers includes the following steps:

[0112] Define message topic identifiers to represent different business types;

[0113] For each message topic, a corresponding data structure definition is established, which includes business attribute fields and their data types;

[0114] During the layered processing, the received message topic is parsed into a structured data object according to the data structure definition, and the fields of the structured data object are mapped to the storage fields of each layer's database.

[0115] In its implementation, the system first establishes a unified metadata management center. For user payment transactions, the message topic identifier is defined as ODS_PAYMENT_EVENT, and its strict data contract is defined using Avro or Protobuf format, explicitly including fields such as long integers and double-precision floating-point numbers, along with their constraints. When the producer sends a message, only the SchemaID and serialized binary data need to be transmitted. When handling layered data flow, the ETL job does not hardcode field mapping relationships in the code. Instead, it reads JSON or YAML mapping files stored in the configuration center. The general parsing operator, based on this configuration, uses Java reflection or dynamic SQL generation technology to automatically parse the unstructured input stream into strongly typed Row objects and precisely route the fields to the corresponding column storage of the DWD layer wide table. The core principle of this design is to completely decouple data form description from data processing logic, guiding the operation of the computing engine through externalized metadata configuration, rather than relying on code logic.

[0116] More specifically, when storing the statistical results of the application data service layer and publishing result events containing the updated results for downstream subscription and consumption, the process further includes the following steps:

[0117] The generated business metrics are written to the physical database of the application data service layer via the storage adapter;

[0118] After the data writing is complete, construct a message object containing the updated indicator data;

[0119] The message object is published through the event bus, and the message object is exposed to downstream components through a preset interface to trigger subsequent business logic.

[0120] The system first introduces a storage adapter as an abstract interface for the persistence layer, shielding the differences in underlying physical storage. After the computing engine generates real-time user retention metrics, it writes them to a high-concurrency key-value database via the adapter. Upon successful write confirmation, the system constructs a standard JSON message object containing {UID:1001, Retention Day:7, Status:1} and submits it to the event bus. In this solution, the event bus specifically refers to distributed message queue middleware, such as Apache Kafka, RocketMQ, or Apache Pulsar, rather than the hardware transmission bus on the computer motherboard. For large-screen display scenarios, after the batch processing task completes the aggregation of all metrics and writes them to the OLAP database, the system publishes a lightweight signal event containing only the data version number and update timestamp to the event bus, triggering the front-end Web Socket service to actively pull the latest report data. The core principle of this design is a distributed extension based on the observer pattern, i.e., an event-driven architecture, which completely decouples data producers and consumers in time and space through asynchronous message passing. It enables push-to-pull data processing, eliminating invalid polling requests to the database from downstream systems and reducing end-to-end latency for data updates from minutes to milliseconds. New downstream services only need to subscribe to the corresponding topic without intruding on or modifying the upstream code.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.< / string>

Claims

1. A statistical method for an event-driven data hierarchical model, characterized in that, Includes the following steps: Establish a data model comprising a raw data layer, a static configuration data layer, a detailed data layer, and an application data service layer, and configure the message topics and data structure definitions corresponding to the data flow between each layer; receive raw events from the business system through an event bus, monitor the context characteristics of the current data flow in real time, and determine the current data processing mode based on the context characteristics; the data processing mode includes at least an event-driven mode; when the event-driven mode is determined, listen to the message topic in real time and write the raw events into the raw data layer; Read the data from the original data layer, and based on the streaming association algorithm, obtain the corresponding dimension information from the static configuration data layer for association and completion, generate wide table data and write it into the detail data layer; Real-time statistical algorithms are executed based on the detailed data layer to generate business metrics and update them to the application data service layer. The statistical results of the application data service layer are stored, and result events containing updated results are published for downstream subscription and consumption.

2. The event-driven data hierarchical model statistical method according to claim 1, characterized in that, The process of determining the current data processing mode based on the context features specifically includes the following steps: the acquired data stream context features include at least data throughput, business latency tolerance, and whether it is historical backtracking data; when the data throughput exceeds the preset batch processing threshold or the data attribute is determined to be historical backtracking data, the process switches to the scheduled task mode and calls the offline batch computing engine for processing; when the business latency tolerance of the data stream is determined to be low latency requirement, the process switches to the event-driven mode and calls the real-time stream computing engine for processing; when the data features are between the trigger conditions of the scheduled task mode and the event-driven mode, the process switches to the micro-batch processing mode.

3. The event-driven data hierarchical model statistical method according to claim 1, characterized in that, In the application data service layer, the storage of user retention metrics adopts a bitmap-based combined storage structure, specifically including the following steps: defining a user retention data model, which includes a unique user identifier, registration date, and a binary bit group for recording active status over multiple consecutive days; dividing the binary bit group into at least two long integer numerical fields for storage; using the bit combination of the numerical fields to map the user's daily active status within a preset time period after registration; wherein, the retention status of the first period is stored through the first long integer numerical field, and the retention status of subsequent periods is stored through the second long integer numerical field.

4. The event-driven data hierarchical model statistical method according to claim 3, characterized in that, When the real-time statistical algorithm updates user retention metrics, it specifically includes the following steps: When a user behavior event is detected in the detailed data layer, the existing user retention data model is queried from the application data service layer based on the user's unique identifier; if the query result is empty, a new user retention data model is initialized, and both long integer numerical fields are set to zero; the difference in the number of days between the current event occurrence time and the user's registration date is calculated; based on the difference in the number of days, the corresponding long integer numerical field and the specific binary bit position are located through shift operations, and the binary bit position is set to 1; the updated user retention data model is written back to the application data service layer.

5. The event-driven data hierarchical model statistical method according to claim 1, characterized in that, The process of transforming the data in the original data layer into wide table data using the streaming association algorithm includes the following steps: monitoring the message topic stream corresponding to the original data layer; when processing the message topic stream data, querying the dimension table data of the static configuration data layer according to the preset primary key information; the query process adopts a multi-level caching mechanism: firstly querying the local memory cache with automatic expiration function; when the local memory cache is not hit, querying the external database and using the query results to fill the local memory cache; writing the associated wide table records into the detail data layer, and simultaneously publishing the corresponding wide table message topic.

6. The event-driven data hierarchical model statistical method according to claim 5, characterized in that, A cache consistency maintenance mechanism is adopted for queries on the static configuration data layer. The mechanism includes at least one of the following steps: setting the write expiration time of the local memory cache to ensure that the cached dimension data is refreshed regularly; listening to the change message stream of the static configuration data layer, and when a message of dimension data change is captured, actively invalidating or updating the corresponding entry in the local memory cache according to the changed primary key.

7. The event-driven data hierarchical model statistical method according to claim 1, characterized in that, The active user statistics in the application data service layer are obtained using a bitmap compression algorithm, specifically including the following steps: mapping the unique identifier set of active users within the same statistical period to a bitmap object; performing a serialization operation on the bitmap object to generate a compressed byte array; storing the byte array in the statistics table of the application data service layer; when performing cross-period statistics, reading the corresponding byte array, deserializing it into a bitmap object, and performing a set operation of logical AND and logical OR to calculate the continuous active user or cumulative active user metrics.

8. The event-driven data hierarchical model statistical method according to claim 2, characterized in that, When the scheduled task mode is selected, the following steps are performed: Confirm that the data to be processed belongs to historical backtracking data or a large batch of data; Initiate batch processing logic and input the data to be processed into the offline batch computing engine; The offline batch computing engine is used to perform full aggregation calculations on the data, generate statistical indicators, and write them into the application data service layer.

9. The event-driven data hierarchical model statistical method according to claim 1, characterized in that, The pre-configuration of the message topic and data structure mapping relationship between the data flow between each layer specifically includes the following steps: defining message topic identifiers representing different business types; establishing a corresponding data structure definition for each message topic, wherein the data structure definition includes business attribute fields and their data types; during the layered processing, parsing the received message topic into a structured data object according to the data structure definition, and mapping the fields of the structured data object to the storage fields of each layer's database.

10. The event-driven data hierarchical model statistical method according to claim 1, characterized in that, The process of storing the statistical results of the application data service layer and publishing result events containing the updated results for downstream subscription and consumption further includes the following steps: writing the generated business metrics into the physical database of the application data service layer through the storage adapter; constructing a message object containing the updated metric data after the data is written; publishing the message object through the event bus, and exposing the message object to downstream components through a preset interface to trigger subsequent business logic.

Images