Method and system for query sharing of log data stream based on hierarchical multiplexing

A technology for sharing systems and data streams, applied in the field of query sharing of log data streams, can solve problems such as inappropriate data stream query processing, and achieve the effect of effective storage and computing resource sharing

CN107193898BActive Publication Date: 2019-12-03INST OF COMPUTING TECH CHINESE ACAD OF SCI
6 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Publication Date
2019-12-03

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention relates to a classification multiplexing-based log data flow query sharing method and system. The method comprises the steps of registering query operations containing a plurality of query statements to a system, and intercepting tuples in log data flow by setting a sliding window; preprocessing the query statements, extracting the query statements containing the same sub-query, and storing a query result as an intermediate result, wherein the intermediate result is shared among the query operations; and extracting the tuples with the same intermediate storage structure from the intercepted tuples, and storing the tuples by adopting a shared queue to be shared among the query operations. For log flow query storage sharing, the flexibility is provided for migration under a data sharing condition on the premise of performing effective storage and quick indexing on data of the tuples of the log flow; and for sub-query sharing, the purpose of computational resource sharing is achieved by extracting and sharing the same predicate query.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention relates to the field of query and storage of log data streams, in particular to a query sharing method and system based on hierarchically multiplexed log data streams. Background technique

[0002] With the continuous development and wide application of cloud computing and big data technologies, more and more attention is paid to the log data in the cluster. On the one hand, this simple log data in the cluster is the same as the traditional log data, and it needs to be stored accordingly during the operation of the system, so as to facilitate the abnormal operation and maintenance and recovery of the system. On the other hand, the log data in the cluster The data generation method is different from the traditional log form, and is generated in the system in the form of a data flow.

[0003] The processing of log data streams (hereinafter referred to as log streams) is different from traditional data streams. When processing traditional d...

Examples

Embodiment Construction

[0036] In order to make the above-mentioned features and effects of the present invention more clear and understandable, the implementation examples are listed below and described in detail in conjunction with the accompanying drawings.

[0037] The log stream query preprocessing object is based on query operators (predicates). Each tuple in the log stream entering the system must pass through all query operators before outputting query results. Thus, each tuple is stored in the corresponding query operator queue while it is waiting to be processed by the corresponding query operator. In some cases, the number of tuples in these queues is approximately the same. For example, the projection operator only forms a new tuple with the corresponding attributes of the tuple and outputs it to the waiting queue of the next query operator, and the number of tuples does not change. In the usual data flow query processing, these intermediate processing results with the same composition ...