A data pushing method

Abstract

The present application relates to data push method technical field, especially a kind of data push method, by mirror index tree engine, the engine adopts hierarchical index tree to carry out hierarchical management to rule, ensure that even under high data volume, it can realize fast, accurate rule matching, improve the efficiency of data push.For the potential problems caused by the dynamic modification of rule, mirror index tree engine also contains shadow hierarchical index tree, when rule is added, deleted or modified, reduce the interference to the ongoing rule matching task, ensure the stability and response speed of system, effectively reduce the negative influence of traditional rule engine on system performance when the number of rules grows and frequently changes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data pushing method, in particular to a data pushing method. BACKGROUND

[0002] Data pushing has become the core link of many business scenarios, such as real-time notification, information update, state change notification, etc. The rule engine plays a role in sorting and processing data in data pushing, and can free complex and variable business rules from hard coding, so that the rules are stored in the form of scripts in files or special storage media, so that the change of business rules can take effect immediately in online environment without modifying project code and restarting server.

[0003] In a large access control system, access control data needs to be pushed to multiple platforms for monitoring, management and data processing, which leads to the demand for large data volume and high rule expansion. The RETE algorithm used by the traditional rule engine such as drools will lead to a huge network structure when the number of rules increases, thus consuming a large amount of memory, and the newly added rules need to be compiled. In the case of large data volume or frequent rule expansion, frequent rule compilation and evaluation lead to a significant decline in data pushing performance. SUMMARY

[0004] To solve the above problems, the present application provides a data pushing method.

[0005] To achieve the above purpose, the technical solution adopted by the present application is:

[0006] A data pushing method, comprising the following steps:

[0007] The canal is used to simulate the interactive protocol of MySQL slave, and the canal is disguised as MySQL slave to request data from MySQL master, parse the binary log pushed by MySQL master and deliver it to the message queue;

[0008] Data is exported from the message queue, enters the first buffer and is preprocessed, and a wide table is generated according to the preprocessed data;

[0009] The rule is managed hierarchically using the mirror index tree engine, the user configured rule is loaded and updated in real time, and the rule is applied to the preprocessed data for sorting, the protocol package is used to process the sorted data according to the task demand, the processed data is packaged as a task package and stored in the task scheduling center;

[0010] The second buffer obtains the task package from the task scheduling center and pushes the data;

[0011] The mirror index tree includes a hierarchical index tree and a shadow hierarchical index tree.

[0012] Further, the hierarchical index tree is constructed in the following manner:

[0013] A field representing a rule characteristic in the rule is determined as a key;

[0014] In the first layer of the hierarchical index tree, all rules are classified according to different values of the determined key, and each value is used as a node of the hierarchical index tree; each node of the first layer of the hierarchical index tree is associated with a group of rules having the same key value.

[0015] For each group of rules associated with a node, a key is selected again for classification to construct an index of the next layer.

[0016] Further, the mirror index tree engine performs hierarchical management of rules in the following manner:

[0017] A use counter is associated with each rule in each hierarchical index tree, and when a rule is applied, the value of the use counter corresponding to the rule is increased;

[0018] The use frequency of all rules is periodically evaluated, and when the value of the use counter corresponding to a rule exceeds a preset high-frequency threshold, the rule is identified as a high-frequency rule; when the value of the use counter corresponding to a rule is lower than a preset low-frequency threshold, the rule is identified as a low-frequency rule;

[0019] The hierarchical index tree is reconstructed according to the use frequency of the rules, and the high-frequency rules are arranged in the upper layer of the hierarchical index tree, and the low-frequency rules are arranged in the lower layer of the hierarchical index tree.

[0020] Further, the shadow hierarchical index tree is the same as the hierarchical index tree, the hierarchical index tree is used for rule matching operation, and the shadow hierarchical index tree is used for rule modification.

[0021] After the rule modification is completed, the roles of the hierarchical index tree and the shadow hierarchical index tree are exchanged to realize updating.

[0022] Further, the protocol package is dynamically loaded and updated through a plug-in framework, including:

[0023] A class loader is introduced to dynamically load, unload or update the protocol package at runtime; each protocol package is provided with a version identifier, and the version identifier is used to check and judge new and old versions.

[0024] Further, the processed data is packaged as a task package and stored in a task scheduling center, specifically:

[0025] After the processed data is encapsulated into the task package, a priority value is determined for each task package according to the nature, urgency and business requirements of the task.

[0026] Further, the task scheduling center manages the task packages using a priority queue to ensure that the task packages with high priority values are located at the front of the queue.

[0027] Further, when the task scheduling center performs task scheduling, it always takes the task package with the highest priority value from the priority queue for scheduling.

[0028] Further, for tasks with the same priority value, a first-in-first-out strategy is used to ensure that the tasks are scheduled in the order in which they arrive at the task scheduling center.

[0029] The present application has the beneficial effects that: the present application uses canal to simulate the interactive protocol of MySQL slave, directly obtains real-time data from MySQL master, thereby ensuring the integrity and real-time performance of the data. Subsequently, the data is preprocessed through the message queue and the first buffer, generates the required wide table format data, and the wide table format data is sorted through the mirror index tree engine, and is processed through the protocol package, encapsulated into a task package and stored in the task scheduling center. The second buffer pulls the task package from the task scheduling center for real-time data pushing. In order to solve the performance problem of the traditional rule engine in the big data and high concurrency scene, the present application designs a mirror index tree engine, adopts a hierarchical index tree to manage the rules in a hierarchical manner, realizes fast rule matching under high data volume, uses a shadow hierarchical index tree, so that the rule adding, deleting and modifying operation does not affect the ongoing rule matching task, thereby reducing the influence of the rule engine on the performance when the rules are frequently expanded. BRIEF DESCRIPTION OF DRAWINGS

[0030] Fig. 1 The step flowchart of the data pushing method in the present application.

[0031] Fig. 2 The step flowchart of the initialization of the mirror index tree engine.

[0032] Fig. 3 The flowchart of step S3. DETAILED DESCRIPTION

[0033] Referring to Figs. 1-3 The present application relates to a data pushing method, which comprises the following steps:

[0034] S1, using canal to simulate the interaction protocol of MySQL slave, disguising as MySQL slave, requesting data from MySQL master, parsing binary log pushed by MySQL master and delivering to message queue;

[0035] Specifically, binary log is a log file of MySQL, which records all data modification operations of the database and can be used for data replication and recovery.

[0036] Canal is an open source tool that can simulate the slave protocol of MySQL, capture binary log from MySQL, deploy and configure Canal to simulate the role of MySQL slave and interact with MySQL master to capture binary log.

[0037] Kafka is an open source stream processing platform for processing and analyzing data streams, which is used as a receiver of binary log in this embodiment.

[0038] In order to ensure that MySQL can generate binary log and allow external tools to read these logs, the MySQL is configured and adjusted, including enabling binary log function, selecting appropriate log format, and assigning a unique server ID to Canal. After obtaining the permission and configuration of MySQL, the deployment of Canal is carried out. Canal is configured to simulate the role of MySQL slave, which enables it to communicate with MySQL master and request and receive binary log. Selecting Kafka as message queue can transmit binary log data to target system or application. Canal is configured to connect to a specific Kafka cluster and send messages for each captured binary log event. After starting Canal service, it will start interacting with MySQL master, capturing binary log and forwarding it to Kafka, which realizes efficient capture of data changes without zero-invasive business system code. Avoid data point missing caused by too many data entrances, loss of push data.

[0039] S2, data is exported from the message queue, enters the first buffer and is preprocessed, and a wide table is generated according to the preprocessed data;

[0040] Specifically, in the data processing flow, data is first exported from the message queue. As an intermediate storage system, the message queue ensures that the data to be processed is transmitted in order according to the first-in, first-out principle. The exported data is temporarily stored in the first buffer, providing a short stopover point to ensure data integrity and consistency.

[0041] Subsequently, the data is preprocessed in the first buffer. This preprocessing step involves data cleaning, standardization, and conversion, among other operations, to improve data quality and applicability. For example, deleting duplicate records, filling in missing values, or converting data formats.

[0042] After preprocessing is complete, the data will be used to construct a wide table. A wide table is a specific data structure where each record contains features and columns, suitable for subsequent data analysis and reporting.

[0043] The entire process from data export from the message queue to the generation of the wide table ensures the continuity, consistency, and accuracy of the data flow.

[0044] S3, using the mirror index tree engine to hierarchically manage rules, real-time loading and updating of user-configured rules, and applying rules to the preprocessed data for sorting. The protocol package processes the sorted data according to task requirements, and the processed data is packaged as a task package and stored in the task scheduling center. Specifically, step S3 includes:

[0045] S31, using the mirror index tree engine to hierarchically manage rules, real-time loading and updating of user-configured rules, and applying rules to the preprocessed data for sorting.

[0046] Specifically, the mirror index tree engine includes constructing a hierarchical index tree and introducing a shadow hierarchical index tree,

[0047] Wherein, the process of constructing a hierarchical index tree is specifically:

[0048] First, determine the field in the rule that best represents the characteristics of the rule as the key. The key should represent the core features of the rule, allowing classification based on the key to effectively group rules.

[0049] Based on the determined key, in the first layer of the hierarchical index tree, all rules are classified by different values of the key. Each value is constructed as a node of the hierarchical index tree. At this time, each node is associated with a group of rules with the same key value.

[0050] For each node in the first layer, continue to select a key to further classify the rule group associated with it to construct the next layer of the index. This key can be the same as the first layer or other fields, and the specific selection depends on the content of the rule and the query requirements.

[0051] Consider a user table with basic fields such as "Name", "Gender", "Mobile Number", etc. Additionally, each user is associated with additional information such as "Address", "Hobbies", "Workplace", etc. For efficient querying, first select a primary field from the user table as the primary key, for example, "Gender", as it might be a common condition in queries.

[0052] In the first layer of the index tree, use "Gender" as the key to classify users, creating two main nodes: "Male" and "Female". Each node is associated with a list of users of the corresponding gender.

[0053] For the "Male" and "Female" nodes in the first layer, further classification of the associated user list is needed. Here, "Workplace" can be chosen as the second key. For example, under the "Male" node, further classification can be made into sub-nodes such as "IT Company", "Financial Institution", "Educational Institution", etc. Similarly, the "Female" node can also be classified according to the same second key. For deeper classification, another key such as "Hobbies" can be chosen, further classifying "IT Company" into "Football Enthusiasts" and "Music Enthusiasts".

[0054] To improve query efficiency and optimize the application of rules, this embodiment introduces the use of counters. Each rule is associated with a usage counter in the hierarchical index tree. When a rule is applied, the corresponding usage counter value increases. By regularly assessing the frequency of rule usage, rules can be divided into high-frequency and low-frequency categories. When a rule's usage counter value exceeds a pre-set high-frequency threshold, the rule is identified as a high-frequency rule; conversely, when the value is below a pre-set low-frequency threshold, it is identified as a low-frequency rule.

[0055] Taking into account the frequency of rule usage, the hierarchical index tree is dynamically reconstructed. This ensures that high-frequency rules can be quickly retrieved during queries, while low-frequency rules are placed in lower layers of the tree.

[0056] Continuing with the user table example, during the reconstruction process, "Gender" remains as the first layer key, but under the "Male" node, "Financial Institution" may be promoted to the first sub-node, while other workplaces such as "IT Company" or "Educational Institution" may be placed below due to being identified as low-frequency rules. Further, for the "Financial Institution" node, "Football Enthusiasts" will be set as the topmost sub-node due to its highest frequency of use. This multi-level classification ensures the orderliness of data, greatly reducing the number of data items that need to be checked when querying large-scale data sets.

[0057] When a query is performed, the relevant nodes of the first layer are first searched, and then the sub-nodes of the second layer are quickly entered, and so on. This hierarchical query strategy greatly reduces the amount of data that needs to be traversed, thereby realizing efficient data sorting. Through multi-level logical classification and index structure, it is ensured that even in a large-scale data set, the required data can be quickly found, and the sorting speed is improved.

[0058] The process of introducing the shadow hierarchical index tree is specifically as follows:

[0059] The shadow hierarchical index tree is completely the same as the hierarchical index tree structure, and the hierarchical index tree is mainly responsible for real-time rule matching operations, while the shadow hierarchical index tree is specially used for rule modification and update.

[0060] When the system needs to add, delete or modify rules, these operations are not directly performed on the hierarchical index tree that is currently performing matching operations. Instead, all the add, delete and modify operations are first performed on the shadow hierarchical index tree, ensuring the continuity of rule matching operations and the consistency of data.

[0061] After completing the rule modification on the shadow hierarchical index tree, in order to make these modifications effective, the roles of the shadow hierarchical index tree and the hierarchical index tree need to be exchanged. This exchange operation is atomic, ensuring that at any moment, there is only one index tree in the system that is performing rule matching operations.

[0062] After the role exchange is completed, the original hierarchical index tree becomes the shadow hierarchical index tree, which is now in a waiting state, preparing for the next rule modification. The original shadow hierarchical index tree becomes the new hierarchical index tree and starts to perform rule matching tasks.

[0063] Through the design of the hierarchical index tree and the shadow hierarchical index tree, not only the efficiency of rule matching is realized, but also the real-time update of rules and the stable operation of the system are ensured. The effective design is particularly suitable for high-demand scenarios that require frequent rule updates and cannot interrupt service.

[0064] S32, processing the sorted data according to task requirements, and the processed data is packaged as a task package and stored in a task scheduling center;

[0065] Specifically, the core of the framework is the application class loader. The class loader can not only introduce new protocol packages in the form of plug-ins during system runtime, but also can update or uninstall existing protocol packages in real time. When a new protocol package is added to the predetermined plug-in directory, the class loader will automatically identify and immediately load it into the system, so that the new protocol takes effect immediately. Similarly, when an existing protocol package needs to be updated or is no longer used, the class loader can also ensure that these changes can be seamlessly removed in real time in the system without affecting other running services or processes.

[0066] To meet the changing and real-time requirements of different data push protocols in the business, first of all, each protocol package is explicitly marked with a version identifier. When a new protocol package is ready to be added to the system, the system will first compare its version identifier. If the new version number is higher than the existing version, the system will replace the existing version; if the version is the same but the content is different, it can choose to run in parallel, or take other strategies according to the specific business needs.

[0067] The data processed in a specific way is integrated and packaged into a task package. Each task package is assigned a specific priority value according to the nature of its internal data, the urgency of processing, and the actual business needs.

[0068] To ensure that tasks can be processed in a timely manner according to their urgency and importance, the task scheduling center adopts a priority queue to manage task packages. This queue structure ensures that task packages with high priority values are always at the front of the queue, so they can be processed first.

[0069] In the actual task scheduling process, the scheduling center will always select the task package with the highest priority value from the priority queue for processing and scheduling. This ensures that important or urgent tasks can be processed first and in a timely manner.

[0070] Finally, for task packages with the same priority value, to ensure fairness and avoid potential starvation, the scheduling center uses a first-in, first-out strategy. This ensures that the task that arrives at the scheduling center first will be scheduled and processed first. This effectively handles the loading and updating of dynamic protocol packages, while ensuring efficient, fair, and orderly scheduling of tasks, meeting the real-time and efficiency needs of the business.

[0071] S4, the second buffer obtains the task package from the task scheduling center and performs data push;

[0072] Specifically, once the task packages in the task scheduling center are ready to be pushed, the second buffer will obtain these task packages from the scheduling center in batches or individually according to the predetermined strategy and time window and perform push. This effectively reduces the number of I / O operations and the overhead of network transmission.

[0073] The above embodiments only describe the preferred embodiments of the present application and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by ordinary engineering technicians in the field shall fall within the protection scope determined by the claims of the present application.

Claims

1. A data push method, characterized in that, Includes the following steps: Canal is used to simulate the interaction protocol of a MySQL slave, pretending to be a MySQL slave, requesting data from the MySQL master, parsing the binary log pushed by the MySQL master, and delivering it to the message queue; Data is exported from the message queue, enters the first buffer and is preprocessed, and a wide table is generated based on the preprocessed data; The mirror index tree engine is used to manage rules hierarchically, load and update user-configured rules in real time, and apply rules to the preprocessed data for sorting. Protocol packets are used to process the sorted data according to task requirements. The processed data is encapsulated into task packets and stored in the task scheduling center. The second buffer retrieves the task package from the task scheduling center and pushes the data. The mirrored index tree includes a hierarchical index tree and a shadow hierarchical index tree; The hierarchical index tree is constructed as follows: Determine the fields in the rule that represent its characteristics as keywords; In the first layer of the hierarchical index tree, all rules are classified according to the different values ​​of the determined keywords, and each value serves as a node in the hierarchical index tree; wherein each node in the first layer of the hierarchical index tree is associated with a set of rules with the same keyword value. For each rule group associated with a node, select a keyword again for classification and build the next level of index; The mirrored index tree engine performs hierarchical management of rules, including: A usage counter is associated with each rule in the hierarchical index tree. When a rule is applied, the value of the corresponding usage counter is incremented. The usage frequency of all rules is evaluated periodically. When the value of the usage counter corresponding to a rule exceeds a preset high-frequency threshold, it is identified as a high-frequency rule; when the value of the usage counter corresponding to a rule is lower than a preset low-frequency threshold, it is identified as a low-frequency rule. The hierarchical index tree is reconstructed based on the frequency of rule usage, with the high-frequency rules set at the upper level of the hierarchical index tree and the low-frequency rules set at the lower level of the hierarchical index tree. The shadow hierarchical index tree is the same as the hierarchical index tree. The hierarchical index tree is used for rule matching operations, and the shadow hierarchical index tree is used for rule modification. After the rule is modified, the roles of the hierarchical index tree and the shadow hierarchical index tree are swapped to achieve the update.

2. The data push method according to claim 1, characterized in that, The protocol packet is dynamically loaded and updated through a plug-in framework, including: A class loader is introduced to dynamically load, unload, or update the protocol packets at runtime; each protocol packet is assigned a version identifier, which is used to check and determine whether the version is new or old.

3. The data push method according to claim 1, characterized in that, The processed data is packaged into a task package and stored in the task scheduling center, specifically as follows: After the processed data is packaged into task packages, a priority value is determined for each task package based on the nature, urgency, and business requirements of the task.

4. The data push method according to claim 3, characterized in that, The task scheduling center uses a priority queue to manage the task packages to ensure that high-priority task packages are at the front of the queue.

5. The data push method according to claim 4, characterized in that, When scheduling tasks, the task scheduling center always retrieves the task package with the highest priority value from the priority queue for scheduling.

6. The data push method according to claim 5, characterized in that, For tasks with the same priority value, a first-in, first-out (FIFO) strategy is used to ensure that tasks are scheduled in the order they arrive at the task scheduling center.

Images