Index acceleration mode matching method and device for event query and storage medium
By designing an index accelerated pattern matching method in IoT device data, using buffer pools and range bitmap indexes, the high latency problem of traditional tree indexes in complex event queries is solved, and efficient pattern matching and query response is achieved.
Patent Information
- Application Number
- CN202411294433.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2024-12-17
AI Technical Summary
Among massive IoT device data, traditional tree indexes are difficult to detect complex event patterns quickly, resulting in high latency problems, mainly due to expensive disk reading and memory computing overhead.
Design an index accelerated pattern matching method, by allocating buffer pool storage events, building ordered index blocks, using range bitmap indexes, compressing index data, and optimizing variable selection rate during query, avoiding unrelated index block access, and reducing disk Read.
It effectively reduces index space overhead and query delay, avoids sorting operations, and improves query efficiency and response speed.
Smart Images
Figure CN119149542A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of time series data mining, and in particular to an index acceleration pattern matching method, device and storage medium for event query. Background Art
[0002] The rapid growth of IoT devices allows us to collect more real-world physical data, thereby enhancing our perception of the surrounding environment. In the massive amount of IoT device data, identifying specific pattern information is particularly important to help users make better decisions. Consider the following three application scenarios:
[0003] (1) Smart grid monitoring scenario: Each charging pile / box is equipped with current, voltage, and temperature sensors. At a certain moment, if a current sensor shows a sharp increase in current, and the temperature of the temperature sensor near the sensor rises to a higher threshold in the following period of time, this may indicate that the wires near the charging pile are broken and cause a fire. Therefore, the monitoring equipment needs to issue an alarm to promptly notify relevant personnel to deal with the problem of wire breakage and fire.
[0004] (2) Intelligent tracking scenario for mining vehicles: Each transport vehicle is equipped with sensors, and the workshop is equipped with automatic weighing and height measurement equipment. The driver of the mining vehicle needs to complete the mining, weighing, measuring and unloading of minerals within the specified time to ensure that each transportation complies with safety regulations and unloads the minerals to the designated location. The intelligent dispatching center will monitor the vehicle status information in real time (here the information monitored at the same time is regarded as an event), and automatically settle the driver's daily remuneration based on the total weight of the minerals in the number of compliant transportation trips.
[0005] (3) Cluster monitoring scenario: The management node will track the usage of key hardware resources during the scheduling of each task in real time. If it is found that most of the tasks on a machine have experienced continuous events of submission, scheduling, termination, or eviction in a short period of time, and the CPU and memory usage of the machine are high, this indicates that there may be computationally intensive tasks running. Therefore, before the task is completed, the management node should transfer other tasks to other machines for execution.
[0006] In the above three application scenarios, multiple raw events are combined with attribute values and time window constraints to form a complex event pattern. To detect the above patterns, pattern matching technology is needed. However, in the case of massive data, pattern matching technology faces the challenge of high latency in mining user-defined patterns. This high latency is mainly due to expensive disk reads and in-memory computing operations. Therefore, in the process of complex event recognition, how to effectively reduce disk reads and in-memory computing overhead to reduce query latency has become a problem that needs to be solved urgently.
[0007] Well-known relational databases, such as MySQL, PostgreSQL, and Oracle, basically use tree-type index B+ trees to accelerate attribute predicate filtering. However, when querying predicates with high selectivity, B+ trees need to read indexes from disk to obtain events that meet the conditions, resulting in expensive disk read overhead; on the other hand, after obtaining events that meet the conditions, these events need to be sorted (the matching engine requires input of an ordered set of events to generate matching results). When the amount of data is large, sorting will bring expensive computing overhead. Therefore, it is difficult for traditional tree-type indexes to bring good query acceleration effects to complex event queries in general complex event query scenarios. Summary of the invention
[0008] In order to overcome the problem that traditional tree indexes are difficult to quickly obtain matching-related events in general complex pattern query scenarios, the present invention designs an index-accelerated pattern matching method, which can effectively reduce the space overhead of the index and ensure that ordered events can be obtained without sorting during query. In addition, the method utilizes the time window constraints in the query pattern to avoid unnecessary disk access operations, thereby improving the efficiency of reading matching-related events from the disk and reducing the query delay of pattern matching.
[0009] In order to achieve the above invention object, the technical solution of the present invention is as follows:
[0010] An index acceleration pattern matching method for event query includes the following steps:
[0011] (1) Allocate a buffer pool, where each buffer is used to store different types of events. After an event is persisted to disk, it is inserted into a buffer with the same event type for indexing.
[0012] (2) When the buffer pool reaches the specified capacity, the data in the buffer is refreshed to the cluster of index blocks in sequence to ensure that the data timestamps in the cluster are ordered and stored continuously, and a regional map is constructed for the cluster to record the attribute range information in the cluster;
[0013] (3) When all clusters are constructed, compress the data of different categories in the index block, construct a range bitmap for the compressed index attributes, and then serialize the index block to disk and clear the buffer pool content;
[0014] (4) For a given pattern to be queried, estimate the overall selectivity of each variable in the pattern and sort them, select the variable with the smallest selectivity for query, obtain its query result, generate a set of matching intervals, and store it in the mapping dictionary;
[0015] (5) Select the variable with the lowest selection rate from the variables that have not been queried, update the matching interval set, and store the query result of each variable in the mapping dictionary;
[0016] (6) According to the query results stored in the mapping dictionary, relevant events are obtained from the disk, and they are transferred to the matching engine for matching to obtain the final query results.
[0017] Preferably, the index block is a basic unit for organizing the index structure into blocks, each index block contains multiple clusters, each cluster stores events of the same type, and its event type is stored only once. The attribute range information in the cluster includes: index block ID, time range, attribute value range, one or more items in the index block storage area, and a range bitmap (a bitmap structure that supports range queries) is used to index the attributes.
[0018] Preferably, the cluster of index blocks stores a timestamp list, a physical record identifier list, and a range bitmap of index attributes. Before serializing the index blocks, the timestamp list, the physical record identifier list, and the index attribute list are differentially compressed (i.e., the compressed value of the list element = the original value of the element - the minimum value of the list element) to reduce the storage space size of the index blocks. Differential compression can ensure that the index blocks can be decompressed without too much additional computing overhead.
[0019] Preferably, in the process of obtaining pattern-related events from disk, in order to skip irrelevant index block access as much as possible during query, a greedy strategy is used to select variables with the lowest selection rate for query in turn, and when querying, check whether the timestamp range of the variable corresponding to the content stored in each cluster overlaps with the matching interval set, and whether the attribute value range of each cluster storage overlaps with the query range. If they overlap, the content in the cluster is queried; otherwise, as long as one of these two conditions does not overlap, the events under the cluster cannot participate in the final matching, and the query of this cluster is skipped to avoid irrelevant index block access.
[0020] Preferably, the matching interval set is generated by the variable query event result with the lowest overall selection rate. Assuming that the event timestamp of a query is t and the query window is w, the matching interval Θ generated by this event is:
[0021]
[0022] Here, the first variable means that the event corresponding to the variable always appears in the first position of the matching result, and the tail variable means that the event corresponding to the variable always appears in the last position of the matching result.
[0023] Preferably, when it is detected that the timestamp range of the cluster corresponding to the variable overlaps with some elements of the matching interval set, the element is marked as having been accessed. After completing the query of a variable, the elements in the matching interval set that have not been accessed are deleted to narrow the range of the matching interval, reduce the number of disk accesses, and thereby reduce the disk reading overhead.
[0024] Preferably, after querying the relevant clusters on the disk corresponding to a single variable, query whether there are events satisfying the query conditions in the corresponding buffer. If so, append the query results to the variable query results to ensure the accuracy of the query results.
[0025] An index acceleration pattern matching processing device for event query, comprising:
[0026] The first building module is used to allocate a buffer pool, where each buffer is used to store different types of events. After the event is persisted to the disk, it is inserted into the buffer with the same event type for indexing.
[0027] The second construction module is used to refresh the data in the buffer to the cluster of index blocks in sequence when the buffer pool reaches the specified capacity, to ensure that the data timestamps in the cluster are ordered and stored continuously, and to construct regional map information for the cluster, recording the event time range stored in the cluster, the maximum and minimum value information of the attributes, and the location of the index block where the cluster is located;
[0028] The third building module is used to compress the data of different categories of index blocks after all clusters are constructed, construct range bitmaps for the compressed index attributes, and then serialize the index blocks to disk and clear the buffer pool content;
[0029] The first query module is used to estimate the overall selectivity of each variable in a given pattern to be queried and sort them, select the variable with the smallest selectivity for query, obtain its query result, generate a set of matching intervals, and store it in a mapping dictionary;
[0030] The second query module is used to select the variables with the lowest selection rate from the variables that have not been queried in turn for query, update the matching interval set, and store the query result of each variable in the mapping dictionary;
[0031] The third query module is used to obtain relevant events from the disk according to the query results stored in the mapping dictionary, transmit them to the matching engine for matching, and obtain the final query results.
[0032] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the index acceleration pattern matching method for event query as described above are implemented.
[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects: by dividing the index into blocks and organizing the events of the same type together, the event type under a cluster is stored only once, thereby reducing the storage redundancy of the event type; by compressing the attribute value, timestamp and physical record identifier list when serializing the index block to the disk, the index space size is reduced; by using the range bitmap index data, it is possible to ensure that the query results of a single variable in the query use a merge operation rather than a sorting operation, that is, to obtain query results with ordered timestamps. Since the processing of complex event queries is based on variables, during the query process, the query window constraints are additionally utilized, irrelevant index block access is avoided, and disk read operations are reduced, thereby improving the query rate. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 This is a flow chart of the index acceleration pattern matching method for event query of the present invention.
[0035] Figure 2 The following is a schematic diagram of the index construction process in the present invention.
[0036] Figure 3 The query plan generated for the query pattern in the present invention.
[0037] Figure 4 It is a schematic diagram comparing the average event insertion delay of the index of the present invention and the existing method.
[0038] Figure 5 The figure is a comparison diagram of the average query delay of statements between the index of the present invention and the existing method. DETAILED DESCRIPTION
[0039] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.
[0040] In the embodiment of the present invention, the smart grid monitoring application is taken as an example to exemplify the specific process of the index acceleration pattern matching method for event query. Considering the smart grid monitoring scenario, current sensors, temperature sensors and voltage sensors are deployed on the electric pile / electricity box. Each sensor regularly measures (the measurement cycle of each sensor can be different) the current sensor value, and sends the data measured by the sensor and the area where the sensor is located to the remote server. Assume that at 9:17 in the area numbered 8, the temperature data measured by the temperature sensor numbered 1001 is 89.9°C, and then the temperature sensor observes the event<T,1001,89.9,8,9:17> (Assume that the data transmission format is <sensor type, sensor number, measurement value, deployment location, measurement timestamp>) and sent to the remote server. After receiving the event transmitted by the temperature sensor, the remote server stores the event and builds relevant indexes for the two columns of measurement value and deployment location.
[0041] like Figure 1 As shown, this embodiment discloses a method for pattern matching in index-accelerated intelligent detection, which includes two stages: index building stage and pattern matching stage.
[0042] In this embodiment, the specific steps of the index construction phase are as follows: Figure 2 As shown: The event<T,1001,89.9,8,9:17> Persist on disk and insert it into a buffer with the same event type (here the sensor type is regarded as the event type) T in preparation for index building. Assuming that the buffer pool reaches the specified capacity at this time, the data in the buffer is refreshed to the cluster of the new index block (here is the index block with ID 2) in turn to ensure that the data timestamps in the cluster of the index block are ordered and stored continuously, and the regional map information is constructed for each cluster and inserted into the regional map. The regional map records the regional key information of each type of associated cluster, that is, the index block number, value range, time range, and storage area where the cluster is located. After all buffers are refreshed to the cluster of the index block, start compressing data of different categories, and then construct a range bitmap for the compressed measurement value list and deployment location list. After the range bitmap is built, the index block is serialized and stored on the disk, and the buffer pool content is cleared.
[0043] In this embodiment, it is assumed that a pattern matching query statement as shown below is given, which aims to monitor a pattern in which the current of the electric wire increases sharply and the temperature of the nearby electric poles continues to rise until it exceeds 80°C.
[0044]
[0045] The specific steps of this method are as follows Figure 3As shown in the figure: Since variable V3 has a Kleene Plus operator (+), and the attribute value of the last event corresponding to variable V3 is required to be greater than 80, in order to enhance the filtering performance, a virtual variable V4 is first created. The constraints corresponding to V4 are: event type is Z, and the attribute value is greater than 80; then, the original query pattern can be equivalent to "PATTERN(V1 N*V2 N*V3*V4)" (here N* indicates that any number of events can be spaced between two adjacent variables); next, evaluate the overall selection rate of variables V1~V4, and identify that variable V4 has the lowest selection rate; then query the query result of the constraint condition corresponding to variable V4①, and generate a matching interval set based on the result; next, query the query results corresponding to variables V1~V3. Considering that variables V1 and V2 have the same event type constraint, the two variables can be merged together for query to obtain query result②; after the query is completed, the unmarked elements of the matching interval set will be deleted, and then the matching interval set will be updated; next, according to the updated matching interval set and the constraint condition corresponding to variable V3, query the result corresponding to variable V3③; then merge query results② and ③, and obtain the filtered events from the disk based on the value of RID; finally, send the filtered events to the matching engine (such as a non-deterministic finite state machine) for matching to obtain the final matching result.
[0046] Furthermore, the performance of the pattern matching method in the index-accelerated intelligent monitoring described in the present invention is verified under three real data sets. JAVA is used here to implement the index structure and query method (FAST is used to refer to this method), and compared with the following three methods: (1) without building an index, completely traverse and scan the data on the disk and filter (FullScan is used to refer to this method); (2) use B+ tree to query equivalent predicates and range predicate constraints (NaiveIndex is used to refer to this method); and (3) use B+ tree to query partial equivalent predicates and range predicate constraints, and generate matching intervals based on the query results, and read the data in the matching intervals (IntervalScan is used to refer to this method). The performance of these three methods is compared. All programs are run on an Intel(R) Core(TM) i9-10900K CPU@
[0047] The results were measured on a Windows 11 machine with 3.70GHz, 32GB RAM, and 1TB SSD. The experimental results are as follows: (1) Figure 4 The comparison between the present invention and other methods in terms of average event insertion delay is shown. The results show that when building an index for data, the present method has the lowest insertion delay, which is an order of magnitude lower than the average insertion delay of NaiveIndex and IntervalScan; (2) Figure 5The latency comparison between the present invention and other methods when executing 500 queries on respective data sets is shown. Compared with the FullScan method, the present invention reduces the overall query latency by 8.2, 59 and 8.4 times on the Cluster, NASDAQ and Crimes data sets, respectively; compared with the IntervalScan method, the present invention reduces the overall query latency by 7.9, 42 and 10 times on the Cluster, NASDAQ and Crimes data sets, respectively; compared with the NaiveIndex method, the present invention reduces the overall query latency by 15.8, 163.7 and 52 times on the Cluster, NASDAQ and Crimes data sets, respectively.
[0048] In summary, this index-accelerated pattern matching method in intelligent detection can reduce the overhead of accessing data on the disk, avoid irrelevant disk access, reduce the overall query latency, and improve the query response speed.
[0049] The present invention also provides an index acceleration pattern matching processing device for event query, comprising:
[0050] The first building module is used to allocate a buffer pool, where each buffer is used to store different types of events. After the event is persisted to the disk, it is inserted into the buffer with the same event type for indexing.
[0051] The second construction module is used to refresh the data in the buffer to the cluster of index blocks in sequence when the buffer pool reaches the specified capacity, to ensure that the data timestamps in the cluster are ordered and stored continuously, and to construct regional map information for the cluster, recording the event time range stored in the cluster, the maximum and minimum value information of the attributes, and the location of the index block where the cluster is located;
[0052] The third building module is used to compress the data of different categories of index blocks after all clusters are constructed, construct range bitmaps for the compressed index attributes, and then serialize the index blocks to disk and clear the buffer pool content;
[0053] The first query module is used to estimate the overall selectivity of each variable in a given pattern to be queried and sort them, select the variable with the smallest selectivity for query, obtain its query result, generate a set of matching intervals, and store it in a mapping dictionary;
[0054] The second query module is used to select the variables with the lowest selection rate from the variables that have not been queried in turn for query, update the matching interval set, and store the query result of each variable in the mapping dictionary;
[0055] The third query module is used to obtain relevant events from the disk according to the query results stored in the mapping dictionary, transmit them to the matching engine for matching, and obtain the final query results.
[0056] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the index acceleration pattern matching method for event query as described above are implemented.
[0057] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, a person skilled in the art should understand that the specific implementation modes of the present invention can still be modified or replaced by equivalents, and any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be included in the scope of protection of the claims of the present invention.
Claims
1. An index acceleration pattern matching method for event query, characterized in that: The following steps are involved: (1) Allocate a buffer pool, where each buffer is used to store different types of events. After an event is persisted to disk, it is inserted into a buffer with the same event type for indexing. (2) When the buffer pool reaches the specified capacity, the data in the buffer is refreshed to the cluster of index blocks in sequence to ensure that the data timestamps in the cluster are ordered and stored continuously. A regional map is constructed for the cluster to record the event time range, attribute maximum and minimum value information, and the location of the index block where the cluster is located. (3) When all clusters are constructed, compress the data of different categories in the index block, construct a range bitmap for the compressed index attributes, and then serialize the index block to disk and clear the buffer pool content; (4) For a given pattern to be queried, estimate the overall selectivity of each variable in the pattern and sort them, select the variable with the smallest selectivity for query, obtain its query result, generate a set of matching intervals, and store it in the mapping dictionary; (5) Select the variable with the lowest selection rate from the variables that have not been queried, update the matching interval set, and store the query result of each variable in the mapping dictionary; (6) According to the query results stored in the mapping dictionary, relevant events are obtained from the disk, and they are transferred to the matching engine for matching to obtain the final query results.
2. The method according to claim 1, characterized in that The index block is the basic unit for organizing the index structure into blocks. Each index block contains multiple clusters. Each cluster stores events of the same type, and its event type is stored only once. The attribute range information in the cluster includes: index block ID, time range, attribute value range, one or more items in the index block storage area, and the attributes are indexed using a range bitmap.
3. The method according to claim 2, characterized in that The cluster of index blocks stores a timestamp list, a physical record identifier list, and a range bitmap of index attributes. Before serializing the index block, the timestamp list, the physical record identifier list, and the index attribute list are differentially compressed respectively.
4. The method according to claim 1, characterized in that: In the process of obtaining events related to the pattern from the disk, a greedy strategy is used to select the variable with the lowest selection rate for query in turn. When querying, check whether the timestamp range of the variable corresponding to the content stored in each cluster overlaps with the matching interval set, and whether the attribute value range stored in each cluster overlaps with the query range. If they overlap, query the content in the cluster; Otherwise, skip querying for that cluster.
5. The method according to claim 4, characterized in that The set of matching intervals is generated by the variable query event results with the lowest overall selection rate. Assuming that the event timestamp of a query is t and the query window is w, the matching interval Θ generated by this event is: The first variable means that the event corresponding to the variable always appears in the first position of the matching result, and the tail variable means that the event corresponding to the variable always appears in the last position of the matching result.
6. The method according to claim 4, characterized in that When it is detected that the timestamp range of the cluster corresponding to the variable overlaps with some elements of the matching interval set, the element is marked as visited. After completing the query of a variable, the elements in the matching interval set that have not been visited are deleted.
7. The method according to claim 1, characterized in that After querying the relevant clusters on the disk corresponding to a single variable, check whether there are events in the corresponding buffer that meet the query conditions. If so, append the query results to the variable query results.
8. An index acceleration pattern matching processing device for event query, characterized in that: include: The first building module is used to allocate a buffer pool, where each buffer is used to store different types of events. After the event is persisted to the disk, it is inserted into the buffer with the same event type for indexing. The second construction module is used to refresh the data in the buffer to the cluster of index blocks in sequence when the buffer pool reaches the specified capacity, to ensure that the data timestamps in the cluster are ordered and stored continuously, and to construct regional map information for the cluster, recording the event time range stored in the cluster, the maximum and minimum value information of the attributes, and the location of the index block where the cluster is located; The third building module is used to compress the data of different categories of index blocks after all clusters are constructed, construct range bitmaps for the compressed index attributes, and then serialize the index blocks to disk and clear the buffer pool content; The first query module is used to estimate the overall selectivity of each variable in a given pattern to be queried and sort them, select the variable with the smallest selectivity for query, obtain its query result, generate a set of matching intervals, and store it in a mapping dictionary; The second query module is used to select the variables with the lowest selection rate from the variables that have not been queried in turn for query, update the matching interval set, and store the query result of each variable in the mapping dictionary; The third query module is used to obtain relevant events from the disk according to the query results stored in the mapping dictionary, transmit them to the matching engine for matching, and obtain the final query results.
9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the index acceleration pattern matching method for event query as described in any one of claims 1 to 7 are implemented.