A method for dynamic grouping of real-time streaming data

Abstract

The application belongs to the technical field of real-time big data, and particularly relates to a real-time stream data dynamic grouping method, which comprises the following steps: configuring rules, configuring rules on a front-end page according to actual requirements, and configuring grouping fields at the same time, wherein the actual requirements include but are not limited to real-time recommendation, real-time risk control and market operation requirements; analyzing real-time burying point log data, wherein the burying point log data is user operation log, mainly including user terminal attributes, time types and time details data; loading rules and obtaining grouping fields; copying event data and marking values of each grouping field; and grouping according to the values of the grouping fields. The real-time stream data dynamic grouping method can improve the processing efficiency of stream application based on event driving and increase flexibility.

Description

Technical Field

[0001] This invention belongs to the field of real-time big data technology, specifically relating to a method for dynamic grouping of real-time streaming data. Background Technology

[0002] Traditional "precision marketing platforms" rely on marketers to select target audiences based on profile tag data for targeted marketing, which has its limitations. This traditional approach often fails to capture fleeting marketing opportunities. For example, the most effective approach is to promptly push information about bags on sale to a user who is keen on promotions while browsing bags; or to push coupons to a price-sensitive user who is hesitating and repeatedly checking a target item in their shopping cart.

[0003] These scenarios obviously cannot be discovered manually by marketers. Software systems are needed to automatically monitor user behavior in real time, make judgments based on pre-configured rules, and then drive the marketing push system to push relevant marketing content. Different rules require data to be grouped and calculated according to different fields, because rules can be dynamically added, updated, or deleted in real time. This requires data to be dynamically grouped according to rules in real time. This allows multiple types of rules to be processed in real time in a single streaming application, saving resources while improving processing efficiency and increasing flexibility. Summary of the Invention

[0004] The purpose of this invention is to provide a method for dynamic grouping of real-time streaming data, which can improve the processing efficiency and increase the flexibility of event-driven streaming applications.

[0005] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for dynamically grouping real-time streaming data, the method comprising:

[0007] S1: Configure rules. Configure rules on the front-end page according to actual needs, and configure grouping fields at the same time.

[0008] S2: Parse real-time event tracking log data;

[0009] S3: Load rules and retrieve grouping fields;

[0010] S4: Copy the event data and mark the values ​​of each grouping field;

[0011] S5: Group by grouping field value.

[0012] To further define, the actual needs include, but are not limited to, real-time recommendations, real-time risk control, and market operation needs.

[0013] Further specifying, the parsing of real-time event tracking log data involves using a stream computing engine to read the event tracking log data in real time and parsing it into EventBean POJO objects.

[0014] Furthermore, the data collected in the event tracking system is user operation logs, which mainly include user terminal attributes, time type, and time details.

[0015] Further specifying, the loading rule, obtaining the grouping field is: using broadcast in the stream computing engine to dynamically load all rules configured in step S1 in real time, and adding the grouping field key of the rule to the data structure HashSet.

[0016] Further specifying, the copied event data, marking the values ​​of each grouping field as follows: iterate through the HashSet, copy the data for each of the aforementioned tracking log data according to the grouping field key in the HashSet, generate a new DynamicKeyedBeanPOJO class object, and use Java reflection mechanism to obtain the value corresponding to the grouping field key.

[0017] Further specifying, the grouping based on the grouping field value is as follows: grouping is performed based on the attribute value of the DynamicKeyedBean POJO, and data with the same value are grouped into the same group for corresponding rule calculations, thereby realizing dynamic grouping of real-time streaming data.

[0018] The invention employing the above technical solution has the following advantages:

[0019] The present invention provides a method for dynamic grouping of real-time streaming data. Based on pre-configured rules, the real-time data stream is judged and then dynamically grouped. This method can improve the processing efficiency of event-driven streaming applications and increase flexibility while saving resources. Attached Figure Description

[0020] The present invention can be further illustrated by the non-limiting embodiments given in the accompanying drawings; it should be understood that the following drawings only show some embodiments of this application and therefore should not be regarded as limiting the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a method for dynamic grouping of real-time streaming data according to the present invention;

[0022] Figure 2 This is a flowchart illustrating the principle of a real-time streaming data dynamic grouping method according to the present invention. Detailed Implementation

[0023] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that similar or identical parts are referred to by the same reference numerals in the drawings or description. Implementations not shown or described in the drawings are forms known to those skilled in the art. In addition, directional terms mentioned in the embodiments, such as "up," "down," "top," "bottom," "left," "right," "front," and "back," are only for reference to the directions in the drawings and are not intended to limit the scope of protection of the present invention.

[0024] Example:

[0025] like Figure 1 As shown, the present invention provides a method for dynamic grouping of real-time streaming data, including...

[0026] S1: Configure rules. Based on your needs, including but not limited to real-time recommendations, real-time risk control, and market operation requirements, configure the rules on the front-end page. At the same time, configure the grouping field. The grouping field is the basis of the rule. Data with the same grouping field value will be grouped into the same group for calculation.

[0027] S2: Parse real-time event tracking log data.

[0028] Even if the stream computing engine reads the event tracking log data in real time and parses it into EventBean POJO class objects, POJO (Plain Ordinary Java Object) is a simple Java object, which is actually just a regular JavaBean. It can be understood as a simple entity class, which can be easily used as an object and its get and set methods can be easily called. The event tracking log data is the user operation log, which mainly includes user terminal attributes, time type, and time details data.

[0029] S3: Load rules and retrieve grouping fields.

[0030] This involves using broadcasting in the stream computing engine to dynamically load all rules configured in step S1 in real time, and adding the grouping field key of the rules to the HashSet data structure.

[0031] S4: Copy the event data and mark the values ​​of each group field.

[0032] Iterate through the HashSet, copy the data for each incoming logging data according to the grouping field key in the HashSet, generate a new DynamicKeyedBean POJO class object. DynamicKeyedBean refers to the POJO class grouped by the grouping field, and use Java reflection mechanism to get the value of the field corresponding to the grouping field key.

[0033] S5: Group by grouping field value.

[0034] Grouping is performed based on the value attribute of the DynamicKeyedBean POJO. Data with the same value will be grouped into the same group for corresponding rule calculations. This achieves dynamic grouping of real-time streaming data.

[0035] The following is a detailed description of a method for dynamic grouping of real-time streaming data according to the present invention:

[0036] S1: Configure rules. Based on your needs, including but not limited to real-time recommendations, real-time risk control, and market operation requirements, configure the rules on the front-end page. At the same time, configure the grouping field. The grouping field is the basis of the rule. Data with the same grouping field value will be grouped into the same group for calculation.

[0037] For example, a real-time risk control requirement could be:

[0038] An IP address has registered more than 10 accounts in the last hour, with the group field being IP.

[0039] A device has logged in more than 5 times in the last 3 minutes, and the group field is deviceId.

[0040] If the above rules are met, it is highly likely that fraudulent registration or account theft has occurred.

[0041] S2: Parse real-time event tracking log data.

[0042] This means using a stream processing engine to read the event tracking log data in real time and parse it into EventBean POJO objects.

[0043] Specifically, the structure of the parsed POJO class object is as follows:

[0044]

[0045]

[0046] The event tracking log data consists of user action logs, and its overall structure is a nested JSON string, as shown below:

[0047]

[0048] The main components include: user terminal attributes (common attributes), time type (eventId), and time details data (properties, map structure).

[0049] The example data above indicates that a user browsed product number 157 on a device with deviceId WEISLD0 and IP address 192.168.0.1.

[0050] S3: Load rules and retrieve grouping fields.

[0051] This involves using broadcasting in the stream computing engine to dynamically load all rules configured in step S1 in real time, and adding the grouping field key of the rules to the HashSet data structure.

[0052] For example, if the risk control requirements above are configured with two grouping fields, IP and device, then the data stored in the HashSet will be [IP, device].

[0053] S4: Copy the event data and mark the values ​​of each group field.

[0054] Iterate through the HashSet, copy the data for each incoming logging data according to the grouping field key in the HashSet, generate a new DynamicKeyedBean POJO object, and use Java reflection to get the value corresponding to the grouping field key.

[0055] The data structure of DynamicKeyedBean is shown below:

[0056]

[0057]

[0058] For example, if the grouping field key in the rule HashSet has two grouping field keys, IP and device, then the data will be copied separately by IP and device. The original EventBean POJO object would be:

[0059] EventBean(deviceId="WEISLD0", ip="192.168.0.1", timeStamp="1594534406220", eventId="productView")

[0060] After dynamic copying, it becomes two DynamicKeyedBean POJO objects:

[0061] DynamicKeyedBean(key="deviceId", value="WEISLD0", eventBean=EventBean)

[0062] DynamicKeyedBean(key="ip", value="192.168.0.1", eventBean=EventBean)

[0063] To retrieve the value of a grouping field key using Java reflection, follow these steps:

[0064]

[0065] S5: Group by grouping field value.

[0066] Grouping is performed based on the value attribute of the DynamicKeyedBean POJO. Data with the same value will be grouped into the same group for corresponding rule calculations. This achieves dynamic grouping of real-time streaming data.

[0067] For example, data with value="WEISLD0" and value="192.168.0.1" will be assigned to two different groups for corresponding rule calculations.

[0068] The above provides a detailed description of a method for dynamic grouping of real-time streaming data provided by the present invention. The specific embodiments described are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A method for dynamic grouping of real-time streaming data, characterized in that: The method includes: S1: Configure rules. Configure rules on the front-end page according to actual needs, and configure grouping fields at the same time. S2: Parse real-time event tracking log data; S3: Load rules and retrieve grouping fields; S4: Copy the event data and mark the values ​​of each grouping field; S5: Group by grouping field value; The actual needs mentioned include real-time recommendations, real-time risk control, and market operation requirements; Specifically, the copied event data is marked with the following values ​​for each grouping field: traverse the HashSet, copy the data for each of the event log data according to the grouping field key in the HashSet, generate a new DynamicKeyedBean POJO object, and use Java reflection mechanism to obtain the value corresponding to the grouping field key. The process of parsing real-time event tracking log data involves using a stream processing engine to read the event tracking log data in real time and parsing it into EventBean POJO objects. The loading rule obtains the grouping field as follows: In the stream computing engine, all rules configured in step S1 are dynamically loaded in real time using broadcast, and the grouping field key of the rule is added to the data structure HashSet.

2. The method for dynamic grouping of real-time streaming data according to claim 1, characterized in that: The data collected in the event tracking system is user operation logs, including user terminal attributes, time type, and time details.

3. A method for dynamic grouping of real-time streaming data according to any one of claims 1-2, characterized in that: The grouping based on the grouping field value is as follows: grouping is performed based on the value attribute of DynamicKeyedBean POJO. Data with the same value are grouped into the same group and the corresponding rules are calculated to achieve dynamic grouping of real-time streaming data.

Images