Data flow processing method, electronic device, and computer program product

Abstract

Embodiments of the present disclosure provide a method, an electronic device, and a computer program product for data stream processing. The method includes: in response to a query request for an input data stream, determining a data label for an input data item in the input data stream, where the data label at least indicates a first matching relationship between the input data item and a first query condition of the query request; determining an output data stream corresponding to the input data stream, where the output data stream includes output data items corresponding to the input data items, and each output data item has a data label; and determining query information corresponding to the query request based on the output data stream. In this way, a real-time query request for a data stream can be responded to.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computers, and more particularly, to methods, electronic devices, and computer program products for data stream processing. Background Art

[0002] In the field of big data analysis, it has become a new trend to adopt data stream processing to replace traditional batch-based data processing. Some application scenarios require processing data streams with extremely low latency to obtain corresponding query results for specific continuous queries. However, currently available systems cannot respond to relatively complex continuous query requests. Summary of the Invention

[0003] Embodiments of the present disclosure provide a solution for data stream processing.

[0004] In a first aspect of the present disclosure, a method for data stream processing is provided, including: in response to a query request for an input data stream, determining a data label for an input data item in the input data stream, the data label at least indicating a first matching relationship between the input data item and a first query condition of the query request; determining an output data stream corresponding to the input data stream, the output data stream including output data items corresponding to the input data items, and each output data item having a data label; and based on the output data stream, determining query information corresponding to the query request.

[0005] In a second aspect of the present disclosure, an electronic device is provided, including a processor; and a memory coupled to the processor, the memory having instructions stored therein, the instructions causing the electronic device to perform operations when executed by the processor, the operations including: in response to a query request for an input data stream, determining a data label for an input data item in the input data stream, the data label at least indicating a first matching relationship between the input data item and a first query condition of the query request; determining an output data stream corresponding to the input data stream, the output data stream including output data items corresponding to the input data items, and each output data item having a data label; and based on the output data stream, determining query information corresponding to the query request.

[0006] In a third aspect of the present disclosure, a computer program product is provided, the computer program product being tangibly stored on a computer-readable medium and including machine-executable instructions that, when executed, cause the machine to perform any steps of the method according to the first aspect.

[0007] The Summary of the Invention section is provided to introduce, in simplified form, a selection of concepts that will be further described in the Detailed Description below. The Summary of the Invention section is not intended to identify key or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other objects, features, and advantages of the present disclosure will become more apparent from the following more detailed description of exemplary embodiments of the present disclosure when taken in conjunction with the accompanying drawings, in which like reference numerals generally represent like components in the exemplary embodiments of the present disclosure. In the drawings:

[0009] Figure 1 FIG. shows a schematic diagram of an exemplary environment in accordance with an embodiment of the present disclosure;

[0010] Figure 2 FIG. shows a flowchart of a process of data stream processing in accordance with an embodiment of the present disclosure;

[0011] Figure 3 FIG. shows a schematic diagram of an exemplary process of determining a data label in accordance with an embodiment of the present disclosure;

[0012] Figure 4 FIG. shows a schematic diagram of another exemplary process of determining a data label in accordance with an embodiment of the present disclosure;

[0013] Figure 5 FIG. shows a schematic diagram of a process of processing multiple input data streams in accordance with an embodiment of the present disclosure;

[0014] Figure 6 FIG. shows a schematic diagram of another process of processing an output data stream in accordance with an embodiment of the present disclosure;

[0015] Figure 7 FIG. shows a flowchart of a process of determining query information in accordance with an embodiment of the present disclosure;

[0016] Figure 8 FIG. shows a schematic diagram of a process of processing an output data stream to determine query information in accordance with an embodiment of the present disclosure; and

[0017] Figure 9 FIG. shows a block diagram of an exemplary device that may be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] The principles of the present disclosure will now be described with reference to several exemplary embodiments shown in the accompanying drawings.

[0019] As used herein, the term "including" and its variations mean open-ended inclusion, i.e., "including but not limited to". Unless otherwise specified, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "an example embodiment" and "an embodiment" mean "a set of example embodiments". The term "another embodiment" means "a set of additional embodiments". The terms "first", "second", etc. may refer to different or the same objects. There may be other explicit and implicit definitions hereinafter.

[0020] Continuous and infinite data provides a new storage object - data stream. The term "data stream" as used herein may refer to an ordered sequence of data items. This ordered sequence may be an infinite sequence of any number of data items. As an example, the data items may be events, and the data stream may be a persistent, resilient, append-only sequence of any number of events. Any number of fields may be included in the data items, and the fields include a field name and field data. For ease of understanding, in the following embodiments, the data items will be described as events, but the present disclosure is not limited thereto.

[0021] In a storage system, a data stream is divided into a set of data shards, which are typically referred to as data stream segments. The data stream segments act as containers for the data items in the data stream.

[0022] To facilitate querying of data items, a search engine in a storage system typically indexes each input data item and stores the index in a corresponding index shard based on the identification information of the data item. The index can be considered a persistent, resilient, and scalable container for the indexes of the data items in the storage system.

[0023] For example, it can be determined whether a specific term (e.g., cat, dog) appears in each of multiple data items (e.g., files) and where it appears, thereby generating a token and building an index accordingly. Storing the index of a data item in a corresponding index shard can be performed, for example, by a consistent hashing algorithm, and the identification information of the data item can form a key space, which is divided into multiple partitions corresponding to the index shards for responding to query requests for data items.

[0024] The index established in this way can support full-text queries. For example, assume there is event 1 "This is a dog" and event 2 "This is a cat". If the query condition of a full-text query is for "dog and cat", based on this query condition, tokens "dog" and "cat" will be generated. Then, based on the indexes pre-established for event 1 and event 2, it can be known that the tokens exist in the indexes of event 1 and event 2, and thus the query results for this query, namely event 1 and event 2, can be returned.

[0025] However, this query method requires pre-indexing data items and obtaining query results based on this index. Therefore, it cannot support real-time queries on the input data stream. This query method is sometimes also called "historical query".

[0026] Historical queries generally support two types of queries. One is "filtering", that is, judging whether a data item matches a specific condition. For each data item, the result is simply yes or no. For example, the following queries "Does the timestamp field indicate between 2015 and 2016?" and "Is the status field set to 'published'?" can both be considered filtering.

[0027] The other is "query", that is, judging how well this data item matches the query condition. In addition to determining whether the data item matches the query phrase, the matching search results can also be sorted according to the relevance score. The higher the score, the more relevant the data item indicates.

[0028] In historical queries, each time query results are obtained, it is in response to a specific query request, and the returned results are one of the data items that have been indexed. Therefore, historical queries cannot query new data items that continuously enter the system along with the data stream.

[0029] In some specific application scenarios, for example, filtering, monitoring, alarming, etc. of event logs, there is a need for real-time queries on such data items. "Continuous query" is a query that continuously progresses over time as data items of the data stream continuously enter the storage system. Traditional continuous queries generally only support filtering conditions and simple query conditions, and cannot provide complex functions like historical queries. For example, there is currently no corresponding solution for complex continuous queries such as join queries.

[0030] For example, consider the example given in Table 1 below. Table 1 shows events of web page access, which can be considered a data stream.

[0031] Table 1

[0032]

[0033] As an example, only 6 events in the data stream are listed in Table 1, but it will be understood that it may include any number of events. For each event, the example in Table 1 has 6 fields and corresponding recorded data, but it will be understood that it may include any number of fields.

[0034] Traditionally, if it is necessary to query malicious websites, website categories can be filtered. For example, for a real-time filtering query of "client IPs that have accessed virus web pages at the Beijing server", the query condition can be set as "server == Beijing" and "category == phishing" (where "== " can indicate whether the data of the specific field on the left side of the symbol matches the key phrase on the right side of the symbol), and a continuous query request is submitted to obtain the corresponding query result (for example, Event 4), and thus the corresponding query information (for example, 10.2.26.49) is obtained. For continuous queries, when the input data item in the input data stream meets the above query condition, the query information (for example, 10.2.26.49) can also be notified to a management personnel such as an IT administrator in real time.

[0035] As described above, traditional continuous query requests do not support complex combinations. For example, in order to block the client IP that has accessed the phishing web page the most times and has accessed www.abc.com a certain number of times, a combined query request can be set up to obtain such a client IP. However, since the query fields are different, the traditional continuous query method based on filtering and simple matching obviously cannot obtain the required query information.

[0036] For example, the traditional solution may need to query through the following steps: Step 1, through a query request, obtain the access counts of all client IPs that have accessed phishing web pages (for the example in Table 1, the result is IP 10.2.26.49, 1 time; IP 10.2.26.51, 1 time); Step 2, through another query request, obtain the access counts of all client IPs that have accessed www.abc.com (for the example in Table 1, the result is IP 10.2.26.49, 2 times; IP 10.2.26.112, 1 time); Step 3, combine the two access counts based on the client IP together; Step 4, sort the combined access counts to obtain the queried client IP, that is, 10.2.26.49.

[0037] To at least partially address the above drawbacks, embodiments of the present disclosure provide a data stream processing solution for implementing responses to complex continuous query requests. According to embodiments of the present disclosure, for a query request, when computing device 105 receives an input data item of an input data stream, it determines a data label corresponding to each input data item, where the data label indicates the matching relationship between the data in the input data item and the query condition; then, for each input data item, computing device 105 outputs a corresponding output data item with the data label, so that the number of data items included in the obtained output data stream corresponds to the number of data items included in the input data stream. Then, the labeled output data stream can be further processed to obtain query information.

[0038] Based on such a data stream processing mechanism, since what is output are multiple output data items with data indicating the matching relationship with the query condition, rather than specific query results, the output data stream can continue to be processed using data stream processing techniques (e.g., continuous query), and thus can support real-time query of complex query requests.

[0039] Figure 1 FIG. shows a schematic diagram of an exemplary environment 100 according to embodiments of the present disclosure, in which devices and / or methods according to embodiments of the present disclosure can be implemented. As Figure 1 shown, the method according to embodiments of the present disclosure is implemented at computing device 105. Computing device 105 can receive an input data stream 110, and the input data stream 110 includes input data items 111, 112, and 113. It will be understood that the input data stream can include any number of input data items. Computing device 105 can receive a query request 130 for the input data stream 110, and the query request 130 can include (multiple) query conditions 131.

[0040] Computing device 105 can perform a processing procedure 120 corresponding to the query request 130 on the input data stream 110 to obtain an output data stream 140. The output data stream 140 includes output data items 141, 142, and 143 corresponding to the input data items 111, 112, and 113, respectively.

[0041] It will be understood that the output data stream can also include any number of input data items, but in this example, the number of data items included in the output data stream is equal to the number of data items included in the input data stream. In some other examples, the number of data items included in the output data stream can be equal to the sum of the number of data items included in the (multiple) input data streams.

[0042] In some embodiments, the output data items 141, 142, and 143 each include a data tag that at least indicates a matching relationship between the corresponding input data items 111, 112, and 113 and the query condition 131 of the query request 130.

[0043] Based on the output data stream including the output data items 141, 142, and 143, the computing device 105 can perform further processing to obtain query information 160 corresponding to the query request 130.

[0044] The following will be combined with Figures 2 to 8 Describe in detail the process of controlling a storage system according to an embodiment of the present disclosure. For ease of understanding, the specific data mentioned in the following description is exemplary and is not used to limit the protection scope of the present disclosure.

[0045] Figure 2 A flowchart of a process 200 of a control storage system according to an embodiment of the present disclosure is shown. The process 200 can be implemented in the computing device 105 shown in Figure 1 It should be understood that the process 200 may further include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this regard.

[0046] In 202, the computing device 105 can, in response to a query request 130 for the input data stream 110, determine a data tag for the input data items in the input data stream 110, where the data tag at least indicates a first matching relationship between the input data item and the first query condition 131 of the query request 130.

[0047] Specifically, the following will be described with reference to the example given in Table 1. If the query request 130 is "Among the events of web page access, the client IP that accesses phishing web pages the most", then the query condition 131 can be "category == phishing". In some embodiments, the data tag can be set to be associated with the query condition 131. For example, for the query condition 131 "category == phishing", the data tag can correspondingly be "empty" (which indicates non - matching) or "phishing" (which indicates matching). In other embodiments, the data tag can be, for example, "N" or "Y" to identify the corresponding data item.

[0048] In some embodiments, the query request 130 may include multiple query conditions 131, such as a first query condition and a second query condition, and the computing device 105 can determine the data tag associated with the first query condition.

[0049] In some embodiments, the data tag associated with the first query condition can be determined as follows. First, the computing device 105 can determine a first query field corresponding to the query request 130. For example, for the data stream shown in Table 1 above and the query condition 131 "category == phishing", the corresponding query field can be determined as "category". Then, the computing device 105 can obtain first record data corresponding to the first query field in the input data item. For example, for the data stream shown in Table 1 above, the record data corresponding to "category" in event 1 can be obtained as "phishing". Next, the computing device 105 can determine a first matching relationship based on the first record data and the first query condition. For example, for event 1 in the data stream shown in Table 1 above, it can be determined that the matching relationship is a match, and thus the data tag is determined as "phishing"; while for event 2 in the data stream shown in Table 1 above, it can be determined that the matching relationship is a non-match, and thus the data tag is determined as "empty".

[0050] Additionally or alternatively, the computing device 105 can determine two data tags respectively associated with the first query condition and the second query condition. One of the two data tags indicates a first matching relationship between the input data item and the first query condition, and the other data tag indicates a second matching relationship between the input data item and the second query condition.

[0051] At 204, the computing device 105 can determine an output data stream 140 corresponding to the input data stream 110. The output data stream includes output data items corresponding to the input data items, and each output data item has the data tag.

[0052] In some embodiments, the computing device 105 can add the determined data tag to the input data item to generate an output data item. In some embodiments, the computing device 105 can associate the determined data tag with the record data in some fields of the input data item to generate an output data item. The computing device 105 can output the generated output data item as part of the output data stream 140. It can be understood that when the number of data items in the input data stream 110 is equal to the number of data items in the output data stream 140, the order of the input data item and the corresponding output data item corresponds.

[0053] In some embodiments, the computing device 105 can simultaneously process multiple query requests for multiple data streams and only output a merged data stream. It can be understood that in this case, although the number of data items in a single input data stream 110 is not equal to the number of data items in the output data stream, the order of the output data items included in the output data stream can correspond to the order of the input data items.

[0054] In some other examples, the number of data items included in the output data stream 140 may be equal to the total number of data items included in the input data stream(s) 110.

[0055] At 206, the computing device 105 may determine query information 160 corresponding to the query request 130 based on the output data stream 140.

[0056] Continuing with the example given in Table 1, for the query request 130 "In the event of web page access, the client IP that accessed the phishing web page the most", after obtaining the output data stream with or without the label "phishing", all output data items with the label "phishing" can be obtained through a conventional filtering operation, and then the filtered output data items can be sorted or counted conventionally to obtain the required query information 160 (i.e., the client IP).

[0057] As discussed above, traditional continuous query requests do not support the combination of complex query conditions because, for traditional continuous query requests, the computing device 105 only uses the query results that match the query conditions to respond, and the query results do not include information related to data items that do not match the query conditions. For example, for the continuous query request "The client IP that has accessed the phishing web page", in the traditional solution, the computing device 105 only returns the query results "10.2.26.49" and "10.2.26.51", and this query result may not include information such as the domain name associated with it, nor information related to other events such as event 2-3. This query result is not a data stream and thus is not suitable for various data stream processing.

[0058] According to an embodiment of the present disclosure, for each input data item, the computing device 105 determines a corresponding data label and includes the data label as part of the output data item in the output data stream 140. Therefore, for the input data items in the input data stream 110, corresponding output data items with data labels can be found in the output data stream 140, and the data label can indicate the matching relationship between the input data item and the query condition. Therefore, using the output data stream 140 composed of these output data items with data labels, various subsequent data stream processing (such as querying using combined query conditions, querying multiple input data streams 110, and traditional filtering, matching, etc.) can be conveniently performed to obtain the required query information 160 for the query request 130.

[0059] Therefore, based on such a data stream processing mechanism, since the output is multiple output data items with data indicating the matching relationship between the indication and the query condition, rather than specific query results, the output data stream can continue to be processed using data stream processing techniques (e.g., continuous query), and thus can support real-time query of complex query requests.

[0060] The following will be combined with Figure 3 the data stream shown in Table 1 to detail an exemplary implementation manner of determining data tags in 202. Figure 3 A schematic diagram showing an exemplary process of determining data tags according to an embodiment of the present disclosure is shown. The processing process 320 can be implemented in Figure 1 the computing device 105 shown in. It will be understood that although Figure 3 only the processing of the first three input data items in the data stream shown in Table 1 is schematically shown, similar processing can be applied to other input data items.

[0061] The computing device 105 can receive an input data stream 310, and the input data stream 310 includes multiple input data items 311, 312, and 313. In other words, the computing device 105 can continuously receive the input data items 311, 312, and 313. In response to a continuous query request 330 registered by a user such as an IT administrator, the computing device 105 can execute the processing process 320 on the input data stream 310 to obtain an output data stream 340, and the output data stream 340 includes output data items 341, 342, and 343.

[0062] Continuing to refer to the data stream shown in Table 1 above, the input data items 311, 312, and 313 can respectively correspond to Events 1, 2, and 3 in Table 1. The query request 310 can be, for example, "Among web page events, the client IP that accesses malicious web pages the most", so the query conditions included in the query request 310 can be determined as: the first query condition 331 "category == phishing" and the second query condition 332 "category == virus". Through the processing process 320, the computing device 105 can determine corresponding data tags. For example, for the input data item 312 (i.e., Event 2), through the processing process 320, the computing device 105 can determine the first data tag 3421 as "empty" and can determine the second data tag 3422 as "virus".

[0063] Specifically, the processing process 320 can include an operation for determining the first data tag. The operation for determining the first data tag here is similar to the operation of determining the first data tag associated with the first query condition described in Figure 2 202 and will not be elaborated here.

[0064] The processing procedure 320 further includes the following operations for determining the second data label. First, the computing device 105 determines a second query condition 332 (e.g., "category == virus") corresponding to the query request 330 and a second query field (e.g., category), where the second query condition is different from the first query condition. Then, the computing device 105 obtains second record data corresponding to the second query field in the input data items. For example, for event 1, the second record data is phishing. Next, the computing device 105 determines a second matching relationship between the input data item and the second query condition based on the second record data and the second query condition. For example, for event 1, the second matching relationship is non - matching. Finally, the computing device 105 determines a second data label indicating the second matching relationship. For example, for event 1, the second data label is "empty".

[0065] In some embodiments, the output data items 341, 342, and 343 may include the above - mentioned data labels and the record data in the input data items 311, 312, and 313, where the record data is associated with the query information to be queried. Taking the output data item 342 as an example, if what needs to be queried is the client IP, then in addition to the data labels 3421 and 3422, the output data item 342 may also include the record data 3420, that is, "10.2.26.112".

[0066] In some embodiments, the output data items 341, 342, and 343 may include the above - mentioned data labels and the input data items 311, 312, and 313 themselves. Taking the output data item 342 as an example, in addition to the data labels 3421 and 3422, the output data item 342 may also include the same data 3420 as the input data item 312. Thus, the output data items 341, 342, and 343 included in the obtained output data stream can be respectively represented as <event 1, "phishing", empty>, <event 2, empty, "virus">, <event 3, empty, "virus">.

[0067] It will be understood that additional data stream processing can be performed on the output data stream 340, such as the data stream processing described below with reference to Figures 6 - 8 for determining the corresponding query information. Therefore, the data stream processing method according to the embodiments of the present disclosure has better flexibility and scalability.

[0068] Figure 4 A schematic diagram showing another example process for determining a data label according to an embodiment of the present disclosure is shown. The process 400 can be implemented in the Figure 1 computing device 105 shown in Figure 4 The input data stream 410, input data items 411 - 413, query request 430, and first query condition 431 and second query condition 432 inFigure 3 The corresponding items described therein are similar and will not be elaborated here. The differences lie in the processing procedure 420 and the output data items 441, 442, and 443 included in the output data 440.

[0069] Specifically, the processing procedure 420 can determine the first matching relationship and the second matching relationship based on the procedures described with reference to Figure 3 Continuing to refer to the data flow shown in Table 1 above, for the input data item 411 (i.e., Event 1), the first matching relationship between this input data item and the first query condition 431 "category == phishing" is a match, and the second matching relationship between this input data item and the second query condition 432 "category == virus" is a non-match. For the input data item 413 (i.e., Event 3), the first matching relationship between this input data item and the first query condition 431 "category == phishing" is a non-match, and the second matching relationship between this input data item and the second query condition 432 "category == virus" is a match.

[0070] In some embodiments, since the corresponding query requests may not need to precisely distinguish the categories of malicious behaviors, a data label indicating both the first matching relationship and the second matching relationship can be adopted, thereby simplifying the subsequent processing of the output data flow. For example, for the input data item 411 (i.e., Event 1), the computing device 105 can determine the data label 4411 indicating the first matching relationship and the second matching relationship, that is, "malicious". Therefore, the output data item 441 can include the data label 4411 and the same data 4410 as the input data item 411. For the input data item 413 (i.e., Event 3), the computing device 105 can determine the data label 4431 indicating the first matching relationship and the second matching relationship, that is, "malicious". Therefore, the output data item 443 can include the data label 4413 and the same data 4430 as the input data item 413. Thus, the output data items 441, 442, and 443 included in the obtained output data flow can be respectively represented as <Event 1, "malicious">, <Event 2, "malicious">, <Event 3, "malicious">.

[0071] It will be understood that additional data flow processing can be performed on the output data flow 440, such as the data flow processing to be described below with reference to Figures 6 - 8 to be used for determining the corresponding query information. Therefore, the data flow processing method according to the embodiments of the present disclosure has better flexibility and scalability.

[0072] Figure 5 shows a schematic diagram of a process for processing multiple input data flows according to an embodiment of the present disclosure. The process 500 can be implemented in the Figure 1 computing device 105 shown in

[0073] As shown in the figure, computing device 105 can receive a first input data stream 510 from a first data source and a second input data stream 515 from a second data source. The first data source is different from the second data source, and the first input data stream 510 is different from the second input data stream 515. As an example, the first input data stream 510 can include a data stream of website access events as given in Table 1 above, and the second input data stream 515 can include a data stream of mail gateway events as given in Table 2 below. Of course, this is only an example, and according to specific application scenarios, the first input data stream 510 and the second input data stream 515 can include any appropriate data streams to be processed.

[0074] Table 2

[0075]

[0076] As an example, only 6 events in the data stream 515 are listed in Table 2, but it will be understood that it can include any number of events. For each event, the example in Table 1 has 6 fields and has corresponding recorded data, but it will be understood that it can include any number of fields. It will be understood that for ease of description, Figure 5 only the processing of the first three input data items 511 - 513 in the data stream shown in Table 1 and the processing of the first three input data items 516 - 518 in the data stream shown in Table 2 are schematically shown, but similar processing can be applied to other input data items.

[0077] Computing device 105 can execute processing procedure 520 and processing procedure 530 simultaneously. Processing procedure 520 is a procedure for determining data tags for input data items in input data stream 510 in response to query request 530. Figure 5 The input data stream 510 in is the same as the input data stream 410 described previously with reference to Figure 4 and the input data stream 510, input data items 511 - 513, query request 530, and corresponding query conditions are all similar to the corresponding items described in Figure 4 so the procedure for determining data tags is similar, and thus will not be elaborated here.

[0078] For example, for input data item 511 (Event 1), the corresponding output data item 542 can include data tag 5421 "malicious" and the same data 5420 as input data item 413. Thus, the first output data items 542, 543, and 546 included in the obtained first output data stream can be represented as <Event 1, "malicious">, <Event 2, "malicious">, <Event 3, "malicious"> respectively.

[0079] In response to a second query request 535 for a second input data stream 515, the computing device 105 may execute a processing procedure 525 to determine a third data label for second input data items 516 - 518 in the second input data stream 515, where the third data label indicates a third matching relationship between the second input data items 516 - 518 and a third query condition 536 of the second query request 535. For example, referring to Table 2 above, if the second query request is "In mail gateway events, the client IP that received the most malicious emails", then the third query condition (and optionally, the fourth query condition) may be "mail category == phishing" and "mail category == virus".

[0080] Then, the computing device 105 may determine a second output data stream corresponding to the second input data stream 515, where the second output data stream includes second output data items 541, 544, and 545 corresponding to the second input data items 511 - 513, and each second output data item has a third data label, where the third data label is associated with the data label for the first input data items 511 - 513. Continuing to refer to Table 2 above, for the input data item 516 (i.e., event 1'), a third data label of "malicious" may be determined through a process 525 similar to the processing procedure 420 described above. Thus, the second output data items 541, 544, and 545 included in the obtained second output data stream may be represented as <event 1', "malicious">, <event 2', "malicious">, <event 3', empty>, respectively. Figure 4 Next, the computing device 105 may incorporate the second output data stream into the output data stream for determining query information.

[0081]

[0082] ​In some embodiments, the computing device 105 may sequentially output corresponding output data items in the order in which the corresponding input data items in the plurality of input data streams are received. For example, the computing device 105 first receives the data item 516 in the second data stream 515 (e.g., event 2 in Table 2, 04 / 19 / 2019 09:39:12), and then receives the data item 511 in the first data stream 510 (e.g., event 1 in Table 1, 04 / 19 / 2019 09:40:12). Then, the computing device 105 may first perform the processing procedure 525 on the data item 516 to add the corresponding data label, thereby generating the data item 541, and then perform the processing procedure 520 on the data item 511 to add the corresponding data label, thereby generating the data item 542. Thus, the output data stream 540 may include the output data items 541-546 output in sequence, which may be respectively represented as <event 1', "malicious">, <event 1, "malicious">, <event 2, "malicious">, <event 2', "malicious">, <event 3', empty>, <event 3, "malicious">.

[0083] It will be understood that additional data stream processing may be performed on the output data stream 540, such as the data stream processing described below with reference to Figures 6 - 8 for determining the corresponding query information. Therefore, the data stream processing method according to the embodiments of the present disclosure has better flexibility and scalability.

[0084] Figure 6 FIG. shows a schematic diagram of another process for processing an output data stream according to an embodiment of the present disclosure. The process 600 may be implemented in the computing device 105 shown in Figure 1 . It will be understood that the output data stream 640 may be any one of the output data streams 340, 440, and 540 described with reference to Figures 3 to 5 , or an output data stream that has undergone other processing. For ease of understanding, the following describes an example in which the output data stream 640 is the output data stream 440 in Figure 4 .

[0085] Since the data processed by the processing procedure 420 in Figure 4 still maintains the form of a data stream, and the data items therein have a one-to-one correspondence with the data items in the (multiple) input data streams, therefore, in response to the third query request, the computing device 105 may perform the processing procedure 650 on the output data stream 640 to determine additional data labels.

[0086] Continuing to refer to the example given in Figure 1 , the output data items 641, 642, and 643 included in the output data stream 640 are related to Figure 4The output data items 441, 442, and 443 respectively correspond. Taking the output data item 642 as an example, it may include the data label 6421 "malicious" and the same data 6420 as the input data item. Therefore, the output data items 641, 642, and 643 can be respectively represented as <Event 1, "malicious">, <Event 2, "malicious">, and <Event 3, "malicious">.

[0087] If the third query request 635 is "In the event of web page access, the client IP that has accessed the dangerous website 'www.abc.com'", then the query condition 636 can be "domain name == www.abc.com". For example, for the output data item 642, through the processing procedure 650, the computing device 105 can determine that the matching relationship between the data item 642 and the query condition 636 is a match, and further determine that another data label 6622 for the output data item 642 is "dangerous website". Thus, the generated data item 662 can include the same data 6620 as the input data item, the data label 6621 "malicious", and another data label 6622 "dangerous website". Therefore, a data stream 660 including data items 661, 662, and 663 can be generated, and the data items 661, 662, and 663 can be respectively represented as <Event 1, "malicious", empty>, <Event 2, "malicious", "dangerous website">, and <Event 3, "malicious", "dangerous website">.

[0088] It will be understood that additional data stream processing can be performed on the data stream 660, such as the data stream processing to be described below with reference to Figures 7 - 8 for determining the corresponding query information.

[0089] The following will be combined with Figure 7 and Figure 8 to describe how to determine the query information corresponding to the query request. Figure 7 FIG. shows a flowchart of a process for determining query information according to an embodiment of the present disclosure. Figure 8 FIG. shows a schematic diagram of a process for processing an output data stream to determine query information according to an embodiment of the present disclosure. It can be understood that Figure 7 and Figure 8 The processes 700 and 800 shown can be implemented in the computing device 105 shown in Figure 1

[0090] Referring to Figure 7 , at 702, the computing device 105 can use data labels to filter the output data stream to determine target output data items.

[0091] Specifically, referring to the examples given in Table 1 and Figure 8 ​In the schematic diagram shown, if the query request 830 is "the client IP with the most accesses to malicious web pages", the method described with reference to Figures 2 to 6 can be used to determine data tags for each input data item to generate an output data stream 840.

[0092] For example, the output data stream 840 may include output data items 841 - 846, which are <Event 1, "Malicious">, <Event 2, "Malicious">, <Event 3, "Malicious">, <Event 4, "Malicious">, <Event 5, Empty>, <Event 6, "Malicious"> respectively. In this case, the computing device 105 can perform a filtering operation on the output data stream 840 through a processing procedure 850, for example, to determine the output data items with the "Malicious" data tag, that is, it can be determined that the target output data items are 841 - 844 and 846.

[0093] It can be understood that the output data items 841 - 846 may also have two or more tags. For example, having "Virus" and / or "Phishing" and / or "Dangerous Website", in this case, multiple filtering conditions or their logical combinations can be set to determine the target output data items.

[0094] Returning to the reference Figure 7 , at 704, the computing device 105 can determine query information corresponding to the query request based on the target output data items.

[0095] Continuing to refer to the example given in Table 1 and the Figure 8 schematic diagram shown, if the target output data items are 841 - 844 and 846, then based on these target output data items, through a processing procedure 850 including conventional sorting and statistical operations, the query information corresponding to the query request 830 can be determined, that is, 10.2.26.49.

[0096] It can be understood that the above processing procedure 850 can support processing an output data stream 540 obtained by processing input data items from multiple data sources as shown in Figure 5 , and processing an output data stream obtained by processing different query conditions of multiple fields as shown in Figure 6 . In this way, the embodiments of the present disclosure can support complex continuous queries such as join queries on data streams in a real - time manner.

[0097] Figure 9 shows a schematic block diagram of an example electronic device 900 that can be used to implement the embodiments of the present disclosure. For example, the electronic device 900 can be used to implement Figure 1The computing device 105 shown in [figure]. As shown in the figure, device 900 includes a central processing unit (CPU) 901, which can execute various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 902 or computer program instructions loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of device 900 can also be stored. The CPU 901, ROM 902, and RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0098] Multiple components in device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, optical disk, etc.; and a communication unit 909, such as a network card, modem, wireless communication transceiver, etc. The communication unit 909 allows device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0099] The processing unit 901 executes each of the methods and processes described above, such as any one of processes 200 to 800. For example, in some embodiments, any one of processes 200 to 800 can be implemented as a computer software program or a computer program product, which is tangibly contained in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the CPU 901, one or more steps of any of the processes 200 to 800 described above can be executed. Alternatively, in other embodiments, the CPU 901 can be configured to execute any one of processes 200 to 800 by any other appropriate means (e.g., by means of firmware).

[0100] This disclosure can be a method, apparatus, system, and / or computer program product. The computer program product can include a computer-readable storage medium having thereon computer-readable program instructions for performing various aspects of this disclosure.

[0101] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example—but not limited to—an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, any non-transitory storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device, such as a punched card or raised structures in grooves storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage medium used herein is not construed as being an instantaneous signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0102] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include a copper transmission cable, an optical fiber transmission, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0103] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer-readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present disclosure.

[0104] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0105] These computer-readable program instructions may be provided to a processing unit of a general purpose computer, a special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, when executed by the processing unit of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowchart illustrations and / or block diagrams. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises a manufacture including instructions for implementing various aspects of the functions / acts specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0106] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process such that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0107] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of code, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by special-purpose hardware-based systems that perform the specified functions or acts, or by combinations of special-purpose hardware and computer instructions.

[0108] The various embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable other ordinary skilled artisans in the art to understand the embodiments disclosed herein.

Claims

1. A method for processing a data stream, comprising: In response to receiving a registration of a continuous query request for an input data stream, when receiving an input data item in the input data stream, continuously determining a data label for the input data item, where the data label at least indicates a corresponding first matching relationship between the input data item and a first query condition of the continuous query request; When receiving the input data item in the input data stream, continuously generating an output data stream corresponding to the input data stream, where the output data stream includes an output data item corresponding to the input data item, and each output data item has the data label of the corresponding output data item; And When receiving the input data item in the input data stream, based on the output data stream, continuously generating query information in response to the continuous query request, the query information including a query result.

2. The method according to claim 1, wherein continuously determining the data label includes: Determining a first query field corresponding to the continuous query request; Obtaining first record data in the input data item corresponding to the first query field; And Determining the first matching relationship between the first record data and the first query condition.

3. The method according to claim 2, wherein continuously determining the data label further includes: Determining a second query condition and a second query field corresponding to the continuous query request, the second query condition being different from the first query condition; Obtaining second record data in the input data item corresponding to the second query field; Based on the second record data and the second query condition, determining a second matching relationship between the input data item and the second query condition; And Determining a data label indicating the first matching relationship and the second matching relationship.

4. The method according to claim 1, wherein continuously generating the query information includes: Using the data label to filter the output data stream to determine target output data items; Based on the target output data items, determining the query information corresponding to the query request.

5. The method according to claim 1, further comprising: In response to receiving an additional registration of an additional continuous query request for an additional input data stream, when receiving an additional input data item in the additional input data stream, continuously determining an additional data label for the additional input data item, where the additional data label at least indicates a corresponding third matching relationship between the additional input data item and an additional query condition of the additional continuous query request; When receiving the additional input data item in the additional input data stream, continuously generating an additional output data stream corresponding to the additional input data stream, where the additional output data stream includes an additional output data item corresponding to the additional input data item, and each additional output data item has the additional data label, where the additional data label is associated with the additional data label of the corresponding additional data label; And When receiving the additional input data item in the additional input data stream, continuously incorporate the additional output data stream into the output data stream for determining the query information.

6. The method according to claim 1, wherein the output data item further includes record data in the input data item, and the record data is associated with the query information.

7. The method according to claim 1, wherein the output data item further includes the input data item.

8. An electronic device, comprising: a processor; and a memory coupled to the processor, the memory having instructions stored therein, which when executed by the processor cause the electronic device to perform actions, the actions including: In response to registering for a continuous query request for an input data stream, when receiving an input data item in the input data stream, continuously determine a data label for the input data item, wherein the data label at least indicates a corresponding first matching relationship between the input data item and a first query condition of the continuous query request; When receiving the input data item in the input data stream, continuously generate an output data stream corresponding to the input data stream, wherein the output data stream includes an output data item corresponding to the input data item, and each output data item has the data label of the corresponding output data item; and When receiving the input data item in the input data stream, based on the output data stream, continuously generate query information in response to the continuous query request, the query information including a query result.

9. The electronic device according to claim 8, wherein continuously determining the data label includes: Determining a first query field corresponding to the continuous query request; Obtaining first record data in the input data item corresponding to the first query field; and Determining the first matching relationship between the first record data and the first query condition.

10. The electronic device according to claim 9, wherein continuously determining the data label further includes: Determining a second query condition and a second query field corresponding to the continuous query request, the second query condition being different from the first query condition; Obtaining second record data in the input data item corresponding to the second query field; Based on the second record data and the second query condition, determining a second matching relationship between the input data item and the second query condition; and Determining a data label indicating the first matching relationship and the second matching relationship.

11. The electronic device according to claim 8, wherein continuously generating and determining the query information includes: Using the data label to filter the output data stream to determine target output data items; Based on the target output data items, determining the query information corresponding to the query request.

12. The electronic device according to claim 8, wherein the actions further include: In response to receiving an additional registration for an additional consecutive query request for an additional input data stream, when receiving additional input data items in the additional input data stream, additional data tags for the additional input data items are continuously determined, where the additional data tags at least indicate a corresponding third matching relationship between the additional input data items and additional query conditions of the additional consecutive query request; when receiving the additional input data items in the additional input data stream, an additional output data stream corresponding to the additional input data stream is continuously generated, where the additional output data stream includes additional output data items corresponding to the additional input data items, and each additional output data item has the additional data tag, where the additional data tag is associated with the additional data tag of the corresponding additional data tag; and when receiving the additional input data items in the additional input data stream, the additional output data stream is continuously incorporated into the output data stream for determining the query information.

13. The electronic device according to claim 8, wherein the output data item further includes record data in the input data item, and the record data is associated with the query information.

14. The electronic device according to claim 8, wherein the output data item further includes the input data item.

15. A computer program product, the computer program product being stored on a non-transitory computer-readable medium and including machine-executable instructions that, when executed, cause the machine to perform the method according to any one of claims 1 to 7.

Images