Data filtering method, device, electronic device, medium and product

Abstract

The present invention provides a data filtering method, device, electronic device, medium and product, the method comprising: determining a target table filter object for processing a received flow log according to the received flow log; determining an index of a target data source according to a target table filter expression identifier corresponding to the target table filter object; wherein the target table filter expression identifier and the index of the target data source have a unique corresponding relationship; the target table filter expression identifier is determined according to the target data source identifier and the query data source portion of the target table filter expression; obtaining data from the target data source according to the index of the target data source, and transmitting the obtained data to the target table filter object so that the target table filter object filters the obtained data. The data filtering method provided by the present invention can accurately obtain the target data source corresponding to the flow log, realize accurate filtering of data, and enhance the user experience.

Description

Technical Field

[0001] The present invention relates to the field of distributed real-time streaming technology, and in particular to a data filtering method, device, electronic equipment, medium and product. Background Art

[0002] With the continuous advancement of data processing technology, businesses in real-time streaming big data often pre-define a large wide table under a data source (such as PostgreSQL) to filter data in real-time streams. Specifically, businesses define desired values for each field in this large wide table (similar to a large dimension table). For example, Kafka stream logs filter data based on pre-defined fields.

[0003] Currently, the maximum number of business rules in a high-concurrency Flink streaming job has reached 900. Each rule has corresponding operators such as filtering, joins, and window statistics. After optimization, the number has reached 12,245 subtasks. As business data volume continues to expand, the large and wide tables in the database are also increasing, and the number of rule expressions that require filtering is increasing.

[0004] In existing technologies, for example, a wide table with more than 100,000 records requires over 2,000 expressions to filter out the desired data. This requires optimizing the memory and threading of high-concurrency operators to improve business scalability. Otherwise, when Kafka traffic reaches 300,000 to 600,000 records per second (each record is approximately 5KB) and there are 700 business rules, and Flink's parallelism is limited to 15, some subtasks' stream states remain in the initialization state, preventing them from running. This results in a data backlog in the wide table, preventing data from being retrieved and processed properly, and resulting in a poor user experience. Summary of the Invention

[0005] The present invention provides a data filtering method, device, electronic device, medium and product to solve the technical problems in the prior art that too many large wide table expressions lead to data backlog, cannot be retrieved normally, and have a poor user experience, so as to achieve the purpose of improving the accuracy of data filtering and enhancing the user experience.

[0006] In a first aspect, the present invention provides a data filtering method, comprising:

[0007] Determining, based on the received flow log, a target table filter object for processing the flow log;

[0008] Determining the index of the target data source according to the target table filter expression identifier corresponding to the target table filter object; wherein there is a unique correspondence between the target table filter expression identifier and the index of the target data source; the target table filter expression identifier is determined according to the target data source identifier and the query data source portion of the target table filter expression;

[0009] According to the index of the target data source, data is obtained from the target data source, and the obtained data is transmitted to the target table filter object, so that the target table filter object filters the obtained data.

[0010] Furthermore, according to the data filtering method provided by the present invention, the method further includes:

[0011] Get pre-defined business rules;

[0012] Performing same-data-source determination on all data source expressions in the business rule according to a preset same-data-source determination condition, and setting a unique data source identifier for the data source expression determined to be from the same data source;

[0013] Extract the query data source part from all table filter expressions in the business rules respectively;

[0014] According to the query data source portion of the table filter expression and the data source identifier corresponding to the table filter expression, a table filter expression identifier is determined for each table filter expression in the business rule.

[0015] Furthermore, according to the data filtering method provided by the present invention, the method further includes:

[0016] Create a corresponding table filter object based on the table filter expression identifier;

[0017] The table filter object creates an index of a data source; wherein there is a unique corresponding relationship between the created index of the data source and the table filter expression identifier corresponding to the table filter object.

[0018] Furthermore, according to the data filtering method provided by the present invention, the method further includes:

[0019] Setting a global static object for storing the index of the data source, and storing the index of the existing data source in the global static object;

[0020] Accordingly, the table filter object creates an index of the data source, including:

[0021] According to the table filter expression identifier corresponding to the table filter object, searching the global static object for an index of a data source corresponding to the data source identifier and the query data source part in the table filter expression identifier;

[0022] If found, the index of the found data source is used as the index of the data source to be created, and a unique correspondence is established between the index and the table filter expression identifier corresponding to the table filter object;

[0023] If not found, create a data source index based on the data source identifier in the table filter expression identifier and the query data source part, and establish a unique correspondence between the created data source index and the table filter expression identifier corresponding to the table filter object.

[0024] Furthermore, according to the data filtering method provided by the present invention, the method further includes:

[0025] When a data source is updated, the index of the data source corresponding to the updated data source in the global static object is updated.

[0026] In a second aspect, the present invention further provides a data filtering device, comprising:

[0027] A first determining module, configured to determine a target table filter object for processing the flow log according to the received flow log;

[0028] a second determining module, configured to determine an index of a target data source according to a target table filter expression identifier corresponding to the target table filter object; wherein the target table filter expression identifier and the index of the target data source have a unique corresponding relationship; and the target table filter expression identifier is determined according to the target data source identifier and a query data source portion of the target table filter expression;

[0029] The filtering module is configured to obtain data from the target data source according to the index of the target data source, and transmit the obtained data to the target table filter object so that the target table filter object filters the obtained data.

[0030] In a third aspect, the present invention further provides an electronic device, comprising:

[0031] processor, memory, and bus, wherein

[0032] The processor and the memory communicate with each other via the bus;

[0033] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the steps of the data filtering method as described in any one of the above items.

[0034] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to execute the steps of the data filtering method described above.

[0035] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which implements the steps of the data filtering method as described in any one of the above items when executed by a processor.

[0036] The present invention provides a data filtering method, device, electronic device, medium and product, the method comprising: determining a target table filter object for processing a received flow log according to the received flow log; determining an index of a target data source according to a target table filter expression identifier corresponding to the target table filter object; wherein the target table filter expression identifier and the index of the target data source have a unique corresponding relationship; the target table filter expression identifier is determined according to the target data source identifier and the query data source portion of the target table filter expression; obtaining data from the target data source according to the index of the target data source, and transmitting the obtained data to the target table filter object so that the target table filter object filters the obtained data. The data filtering method provided by the present invention can accurately obtain the target data source corresponding to the flow log, realize accurate filtering of data, and enhance the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 It is a flow chart of a data filtering method provided by the present invention;

[0039] Figure 2 It is a schematic diagram of the corresponding relationship between the table filter expression and the data source provided by the present invention;

[0040] Figure 3 This is a schematic diagram of the overall process of a data filtering method provided by the present invention;

[0041] Figure 4 This is an example diagram in the prior art where no expression optimization processing has been performed;

[0042] Figure 5 This is an example diagram of the filtration optimization process provided by the present invention;

[0043] Figure 6 It is a structural schematic diagram of a data filtering device provided by the present invention;

[0044] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0045] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0046] Figure 1 The data filtering method provided by the present invention is as follows: Figure 1 As shown, the data filtering method provided by the present invention specifically includes the following steps:

[0047] Step 101: Determine a target table filter object for processing a received flow log according to the flow log.

[0048] In this embodiment, a target table filter object for processing the received flow log is determined based on the received flow log information. In this embodiment, the received flow log may be a Kafka flow log, and the target table filter object may be a large wide table, such as TableFilter. A specific expression may be TableFilter(external data source identifier, ip(sip) == data source wide table ("ip") and | or ....). It should be noted that the target table filter can be selected based on the user's actual needs and is not specifically limited here.

[0049] Step 102: Determine the index value of the target data source based on the target table filter expression identifier corresponding to the target table filter object; wherein, there is a unique correspondence between the target table filter expression identifier and the index of the target data source; the target table filter expression identifier is determined based on the target data source identifier and the query data source part of the target table filter expression.

[0050] In this embodiment, it is necessary to determine the index value of the corresponding target data source based on the target table filter expression identifier corresponding to the determined target table filter, wherein there is a unique correspondence between the target table filter expression identifier and the index value of the target data source, such as the target table filter expression identifier 1 corresponds to the index value 1 of the target data source, and the target table filter expression identifier 2 corresponds to the index value 2 of the target data source. After determining the target table filter expression identifier 2, the corresponding data source is retrieved according to the index value 2 of the corresponding target data source.

[0051] In this embodiment, the target table filter expression identifier is determined based on the target data source identifier and the query data source portion of the target table filter expression. In this embodiment, only the first judgment condition of the expression needs to be extracted. For example, if the target data source identifier ip(sip) == target table filter expression ("ip"), no matter how the subsequent and|or conditions change, they all correspond to the same data source identified above. The specific expression is as follows: tableFilter(external data source identifier, ip(sip) == data source wide table ("ip") and|or....). It should be noted that in other embodiments, the target table filter expression identifier can also be composed of other content, which is not specifically limited here.

[0052] Step 103: According to the index value of the target data source, data is obtained from the target data source, and the obtained data is transmitted to the target table filter object, so that the target table filter object filters the obtained data.

[0053] In this embodiment, it is necessary to obtain corresponding data from the target data source according to the index value of the target data source determined in the above step 102, and then transfer the obtained data to the target table filter object to filter the data to improve the accuracy of data processing.

[0054] According to the data filtering method provided by the present invention, the target table filter object for processing the flow log is determined based on the received flow log; the index value of the target data source is determined based on the target table filter expression identifier corresponding to the target table filter object; wherein, there is a unique correspondence between the target table filter expression identifier and the index of the target data source; the target table filter expression identifier is determined based on the target data source identifier and the query data source part of the target table filter expression; according to the index value of the target data source, data is obtained from the target data source, and the obtained data is transmitted to the target table filter object, so that the target table filter object filters the obtained data. The data filtering method provided by the present invention can accurately obtain the target data source corresponding to the flow log, realize accurate filtering of data, and improve user experience.

[0055] In another embodiment of the present invention, the method further comprises:

[0056] Get pre-defined business rules;

[0057] Performing same-data-source determination on all data source expressions in the business rule according to a preset same-data-source determination condition, and setting a unique data source identifier for the data source expression determined to be from the same data source;

[0058] Extract the query data source part from all table filter expressions in the business rules respectively;

[0059] According to the query data source portion of the table filter expression and the data source identifier corresponding to the table filter expression, a table filter expression identifier is determined for each table filter expression in the business rule.

[0060] In this embodiment, when a business rule is submitted, all data source expressions in the pre-defined business rules are judged to be the same data source based on the pre-set same data source judgment conditions. If both rules belong to the same data source, a unique data source identifier is set to determine the data source expression from the same data source. The same data source judgment condition is to confirm whether the SQL, query fields, field types, etc. corresponding to the data set of the large wide table are the same. If the above indicator information is the same in the two business rules, a unique data source identifier (ID) is determined for the business rule information.

[0061] In this embodiment, it is also necessary to determine the table filter expression identifier for each table filter expression in the business rule based on the query data source part of the table filter expression and the data source identifier corresponding to the table filter expression. In this embodiment, it is necessary to read the query data source part in the table filter (TableFilter) expression, such as IP (kafka stream log IP) == data source wide table ("data source IP"), determine it as part of the key expression of the index, and the external data source unique identifier corresponding to the table filter expression, merge the two parts together to form a new unique identifier to be determined as the table filter expression identifier.

[0062] According to the data filtering method provided by the present invention, by obtaining pre-defined business rules, and then judging the same data source for all data source expressions in the business rules according to pre-set same data source judgment conditions, a unique data source identifier is set for the data source expression determined to be the same data source, and the query data source part is extracted from all table filter expressions in the business rules respectively. According to the query data source part of the table filter expression and the data source identifier corresponding to the table filter expression, the table filter expression identifier is determined for each table filter expression in the business rule, which can improve the accuracy of data filtering and enhance the user experience.

[0063] In another embodiment of the present invention, the method further comprises:

[0064] Create a corresponding table filter object based on the table filter expression identifier;

[0065] The table filter object creates an index of a data source; wherein there is a unique corresponding relationship between the created index of the data source and the table filter expression identifier corresponding to the table filter object.

[0066] In this embodiment, a corresponding table filter object is created based on the table filter expression identifier, and the table filter object creates an index of the data source. Specifically, when the TableFilter object creates an external data source index, the index key content of the data source is determined, and then it is searched in the global static object. If it is determined that the table filter object already has an index object, the existing index object is directly used. Otherwise, an index object needs to be created for the table filter object. For detailed introduction, please refer to the following embodiment and will not be described in detail here.

[0067] It should be noted that in this embodiment, when the same table filter expression yields the same result set, a single copy of the table filter expression is reused throughout the entire virtual machine. Furthermore, a unique correspondence exists between the index of the created data source and the table filter expression identifier corresponding to the table filter object. This ensures streamlined processing of identical table filter expressions.

[0068] According to the data filtering method provided by the present invention, by creating a corresponding table filter object based on the table filter expression identifier, and the table filter object creating an index of the data source, it can ensure streamlined processing of the table filter expression and directly obtain the corresponding data source according to the index value, greatly optimizing thread overhead and memory usage, and improving user experience.

[0069] In another embodiment of the present invention, the method further comprises:

[0070] Setting a global static object for storing the index of the data source, and storing the index of the existing data source in the global static object;

[0071] Accordingly, the table filter object creates an index of the data source, including:

[0072] According to the table filter expression identifier corresponding to the table filter object, searching the global static object for an index of a data source corresponding to the data source identifier and the query data source part in the table filter expression identifier;

[0073] If found, the index of the found data source is used as the index of the data source to be created, and a unique correspondence is established between the index and the table filter expression identifier corresponding to the table filter object;

[0074] If not found, create a data source index based on the data source identifier in the table filter expression identifier and the query data source part, and establish a unique correspondence between the created data source index and the table filter expression identifier corresponding to the table filter object.

[0075] In this embodiment, when a distributed task is started, a global static object for storing the index of the data source is set during the initialization process, and the index of the existing data source is stored in the global static object. The distributed task is an enrichment task based on the operator of Apache Flink, where Apache Flink is a distributed processing engine and framework for stateful computing on bounded and unbounded data streams.

[0076] It should be noted that, in this embodiment, the table filter object creates an index of the data source, specifically including: according to the table filter expression identifier corresponding to the table filter object, searching the global static object for the index of the data source corresponding to the data source identifier in the table filter expression identifier and the query data source part, specifically: taking the unique identifier of the current data source instance, obtaining the TableFilter operator instances of all different expressions in the global static object, and then traversing all the obtained TableFilter instances, determining the unique index of the data source through the unique identifier of the TableFilter itself, and determining the corresponding data source according to the index.

[0077] It should be noted that if Figure 2As shown, a data source object should have n TabrelFilter expressions, that is, a large wide table result set can be divided into different result sets according to different field types or SQL statements, so that a data source object corresponds to multiple TabrelFilter expressions; a TableFilter expression corresponds to n TableFilter objects, that is, the result set of the large wide table A field, which can be referenced by multiple different business rules; in addition, a TableFilter expression corresponds to an external data source index. For example, the result set of the A field will only create a unique index, and there is no need to create multiple identical indexes because different businesses reference it.

[0078] It should be noted that, in this embodiment, when the corresponding index is found, the index of the found data source is used as the index of the data source to be created, and a unique correspondence is established with the table filter expression identifier corresponding to the table filter object; when the corresponding index is not found, the index of the data source is created based on the data source identifier in the table filter expression identifier and the query data source part, and a unique correspondence is established between the index of the created data source and the table filter expression identifier corresponding to the table filter object.

[0079] According to the data filtering method provided by the present invention, by searching the global static object for the data source identifier in the table filter expression identifier and the index of the data source corresponding to the query data source part based on the table filter expression identifier corresponding to the table filter object, it is possible to ensure streamlined processing of the table filter expression and directly obtain the corresponding data source according to the index value, greatly optimizing thread overhead and memory usage, and improving user experience.

[0080] In another embodiment of the present invention, the method further comprises:

[0081] When a data source is updated, the index of the data source corresponding to the updated data source in the global static object is updated.

[0082] In this embodiment, when a data source is updated, the index of the data source corresponding to the updated data source in the global static object needs to be updated simultaneously to ensure that the correspondence between the index value and the data source is updated in real time.

[0083] According to the data filtering method provided by the present invention, when a data source is updated, the index of the data source corresponding to the updated data source in the global static object is updated, which can ensure the real-time status between the data source and the index, improve the accuracy of data filtering processing, and greatly optimize thread overhead and memory usage.

[0084] In another embodiment of the present invention, the prior art is first introduced. Figure 3As shown, in the existing technology, each expression corresponds to a result set. When the business submits 700 rules, more than 2,000 table filter expressions will be generated. After the parallelism is increased to 15, the entire cluster occupies nearly 300G.

[0085] It should be noted that, in this embodiment, the specific processing flow is as follows Figure 4 As shown, when a rule is submitted, the SQL statement, query field, field type, etc. corresponding to the data set of the query large wide table are first compared. If the above indicator information is the same, a unique identifier is determined for the same table filter expression; then, each expression in the business rule is parsed, and it is necessary to extract the identifier of the external data source and the expression for querying the external data source in the expression. Only the first judgment condition of the expression needs to be extracted: for example, IP(sip)==data source wide table("IP"), and there is no need to manage how the subsequent and|or conditions change. They all correspond to the same data source corresponding to the above identifier. The expression can be specifically TableFilter(external data source identifier, IIP(sip)==data source wide table("IP") and|or....).

[0086] It should be noted that in this embodiment, the unique identifier of the data source and the extracted expression are combined into a single unique identifier, which is then stored in the instance corresponding to the TableFilter expression. In each JAVA virtual machine process, when referencing an external data source index, if the unique identifiers are identical, the same external data source is referenced. This allows over 2,000 expressions to be generated into approximately 200 data source indexes, significantly reducing memory usage and improving cluster performance.

[0087] In another embodiment of the present invention, for example, a specific example is: select * where (tableFilter(object(external data source id1), ip(kafka log ip) == data source wide table("data source ip") and "certain value" in data source wide table("data source field"), collect="none")) or (tableFilter(object("external data source id2"), ip(kafka log ip) == data source wide table("data source ip") and "certain value" in data source wide table("data source field"), collect="none")) or (tableFilter(object("external data source id3"), ip(kafka log ip) == data source wide table("data source ip") and "certain value" in data source wide table("data source field"), collect="none"))

[0088] Based on the above example, all data source expressions defined in the rule are first traversed. The SQL statements, field types, and connection methods in the data source expressions are compared. If they are identical, an external data source expression instance is used. The unique identifier of the table filter expression is also set. Then, all TableFilter operators in the rule are parsed. The external data source unique identifier is found through the referenced external data source ID and injected into the TableFilter instance. The IP address (Kafka log IP address) in the TableFilter expression is read as part of the index key expression. The data source identifier and the table filter expression are then merged to form a new unique identifier, which is then used as the index key. A global static object is stored in each Java virtual machine to store the external data source index. When a TableFilter object creates an external data source index, it searches the global static object based on the determined index key. If an existing index object is found, it is used directly. Otherwise, a new index object is created.

[0089] It should be noted that when a distributed task is started, data is injected into the index in the global static object during the initialization process. The logic of injecting data is: take the unique identifier of the current external data source instance, take all different expression TableFilter operator instances in the global static object, and then traverse all TableFilter instances to find the corresponding external data source index instance through the unique identifier of the TableFilter itself; when the external data source is updated, the index corresponding to the global static object needs to be updated directly; when Kafka's stream log enters the corresponding TableFilter operator, directly obtain data according to the index of the current external data source for filtering.

[0090] like Figure 5As shown, the data filtering method provided by the present invention significantly optimizes thread overhead and memory usage. In the prior art, each expression, regardless of whether it is identical, generates an instance object. This instance object also references its own index, regardless of whether the index corresponds to the same result set. In this embodiment, identical expressions are extracted, such as in (tableFilter(dataset object, ip(sip) == data source wide table ("ip") and|or....). Regardless of the conditions following the and or or, they all belong to the same type of expression, referencing the same result set. This reduces the number of identical expressions by more than three-quarters when submitting more than 2,000 expressions (for example, if a production line extracts only 27 unique expressions, only 27 result sets are generated). If an expression is 10 MB, the entire cluster only requires a little over 270 MB of memory, a memory reduction of over 19 GB after optimization. The most significant result is that a Flink job can fully launch 60 parallelisms, totaling 12,245 subtasks, in just 4 minutes, with stable cluster operation.

[0091] Figure 6 A data filtering device provided by an embodiment of the present invention, such as Figure 6 As shown, the data filtering device provided by the present invention specifically includes:

[0092] A first determining module 601 is configured to determine a target table filter object for processing a received flow log according to the received flow log;

[0093] The second determining module 602 is configured to determine the index of the target data source according to the target table filter expression identifier corresponding to the target table filter object; wherein the target table filter expression identifier and the index of the target data source have a unique corresponding relationship; and the target table filter expression identifier is determined according to the target data source identifier and the query data source portion of the target table filter expression;

[0094] The filtering module 603 is configured to obtain data from the target data source according to the index of the target data source, and transmit the obtained data to the target table filter object so that the target table filter object filters the obtained data.

[0095] According to the data filtering device provided by the present invention, the target table filter object for processing the flow log is determined based on the received flow log; the index of the target data source is determined based on the target table filter expression identifier corresponding to the target table filter object; wherein there is a unique correspondence between the target table filter expression identifier and the index of the target data source; the target table filter expression identifier is determined based on the target data source identifier and the query data source part of the target table filter expression; according to the index of the target data source, data is obtained from the target data source, and the obtained data is transmitted to the target table filter object, so that the target table filter object filters the obtained data. The data filtering device provided by the present invention can accurately obtain the target data source corresponding to the flow log, realize accurate filtering of data, and improve the user experience.

[0096] Furthermore, according to the data filtering method provided by the present invention, the data filtering device is further used to:

[0097] Get pre-defined business rules;

[0098] Performing same-data-source determination on all data source expressions in the business rule according to a preset same-data-source determination condition, and setting a unique data source identifier for the data source expression determined to be from the same data source;

[0099] Extract the query data source part from all table filter expressions in the business rules respectively;

[0100] According to the query data source portion of the table filter expression and the data source identifier corresponding to the table filter expression, a table filter expression identifier is determined for each table filter expression in the business rule.

[0101] According to the data filtering device provided by the present invention, by obtaining pre-defined business rules, and then judging the same data source for all data source expressions in the business rules according to pre-set same data source judgment conditions, a unique data source identifier is set for the data source expressions determined to be the same data source, and the query data source part is extracted from all table filter expressions in the business rules. According to the query data source part of the table filter expression and the data source identifier corresponding to the table filter expression, the table filter expression identifier is determined for each table filter expression in the business rules, which can improve the accuracy of data filtering and enhance the user experience.

[0102] Furthermore, according to the data filtering method provided by the present invention, the data filtering device is further used to:

[0103] Create a corresponding table filter object based on the table filter expression identifier;

[0104] The table filter object creates an index of a data source; wherein there is a unique corresponding relationship between the created index of the data source and the table filter expression identifier corresponding to the table filter object.

[0105] According to the data filtering device provided by the present invention, by creating a corresponding table filter object based on the table filter expression identifier, and the table filter object creates an index of the data source, it can ensure the streamlined processing of the table filter expression and directly obtain the corresponding data source according to the index value, greatly optimizing the thread overhead and memory usage, and improving the user experience.

[0106] Furthermore, according to the data filtering method provided by the present invention, the data filtering device is also used for:

[0107] Setting a global static object for storing the index of the data source, and storing the index of the existing data source in the global static object;

[0108] Accordingly, the table filter object creates an index of the data source, including:

[0109] According to the table filter expression identifier corresponding to the table filter object, searching the global static object for an index of a data source corresponding to the data source identifier and the query data source part in the table filter expression identifier;

[0110] If found, the index of the found data source is used as the index of the data source to be created, and a unique correspondence is established between the index and the table filter expression identifier corresponding to the table filter object;

[0111] If not found, create a data source index based on the data source identifier in the table filter expression identifier and the query data source part, and establish a unique correspondence between the created data source index and the table filter expression identifier corresponding to the table filter object.

[0112] According to the data filtering device provided by the present invention, by searching for the data source identifier in the table filter expression identifier and the index of the data source corresponding to the query data source part in the global static object according to the table filter expression identifier corresponding to the table filter object, it is possible to ensure streamlined processing of the table filter expression and directly obtain the corresponding data source according to the index value, greatly optimizing thread overhead and memory usage, and improving user experience.

[0113] Furthermore, according to the data filtering method provided by the present invention, the data filtering device is also used for:

[0114] When a data source is updated, the index of the data source corresponding to the updated data source in the global static object is updated.

[0115] According to the data filtering device provided by the present invention, when a data source is updated, the index of the data source corresponding to the updated data source in the global static object is updated, which can ensure the real-time status between the data source and the index, improve the accuracy of data filtering processing, and greatly optimize thread overhead and memory usage.

[0116] Since the principles of the apparatus described in the embodiment of the present invention are the same as those of the method described in the above embodiment, more detailed explanations are omitted here.

[0117] Figure 7 Schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as Figure 7 As shown, the present invention provides an electronic device, including: a processor (processor) 701, a memory (memory) 702 and a bus 703;

[0118] The processor 701 and the memory 702 communicate with each other via the bus 703.

[0119] The processor 701 is used to call the program instructions in the memory 702 to execute the methods provided in the above-mentioned method embodiments, for example, including: determining the target table filter object for processing the flow log based on the received flow log; determining the index of the target data source based on the target table filter expression identifier corresponding to the target table filter object; wherein there is a unique correspondence between the target table filter expression identifier and the index of the target data source; the target table filter expression identifier is determined based on the target data source identifier and the query data source part of the target table filter expression; according to the index of the target data source, obtaining data from the target data source, and transmitting the obtained data to the target table filter object, so that the target table filter object filters the obtained data.

[0120] In an embodiment of the present invention, a non-transitory computer-readable storage medium is provided, which stores computer instructions. The computer instructions enable the computer to execute the methods provided in the above-mentioned method embodiments, for example, including: determining a target table filter object for processing the flow log based on the received flow log; determining an index of a target data source based on a target table filter expression identifier corresponding to the target table filter object; wherein there is a unique correspondence between the target table filter expression identifier and the index of the target data source; the target table filter expression identifier is determined based on the target data source identifier and the query data source part of the target table filter expression; according to the index of the target data source, obtaining data from the target data source, and transmitting the obtained data to the target table filter object, so that the target table filter object filters the obtained data.

[0121] The present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above embodiments, the method including: determining a target table filter object for processing the flow log based on the received flow log; determining an index of a target data source based on a target table filter expression identifier corresponding to the target table filter object; wherein there is a unique correspondence between the target table filter expression identifier and the index of the target data source; the target table filter expression identifier is determined based on the target data source identifier and the query data source part of the target table filter expression; according to the index of the target data source, obtaining data from the target data source, and transmitting the obtained data to the target table filter object, so that the target table filter object filters the obtained data.

[0122] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data filtering method, characterized in that: include: Determine, according to the received flow log, a target table filter object for processing the flow log; wherein the target table filter object is a large wide table; Determining the index of the target data source according to the target table filter expression identifier corresponding to the target table filter object; wherein there is a unique correspondence between the target table filter expression identifier and the index of the target data source; the target table filter expression identifier is determined according to the target data source identifier and the query data source portion of the target table filter expression; According to the index of the target data source, data is obtained from the target data source, and the obtained data is transmitted to the target table filter object, so that the target table filter object filters the obtained data.

2. The data filtering method according to claim 1, characterized in that: The method also includes: Get pre-defined business rules; Performing same-data-source determination on all data source expressions in the business rule according to a preset same-data-source determination condition, and setting a unique data source identifier for the data source expression determined to be from the same data source; Extract the query data source part from all table filter expressions in the business rules respectively; According to the query data source portion of the table filter expression and the data source identifier corresponding to the table filter expression, a table filter expression identifier is determined for each table filter expression in the business rule.

3. The data filtering method according to claim 2, characterized in that: The method also includes: Create a corresponding table filter object based on the table filter expression identifier; The table filter object creates an index of a data source; wherein there is a unique corresponding relationship between the created index of the data source and the table filter expression identifier corresponding to the table filter object.

4. The data filtering method according to claim 3, characterized in that: The method also includes: Setting a global static object for storing the index of the data source, and storing the index of the existing data source in the global static object; Accordingly, the table filter object creates an index of the data source, including: According to the table filter expression identifier corresponding to the table filter object, searching the global static object for an index of a data source corresponding to the data source identifier and the query data source part in the table filter expression identifier; If found, the index of the found data source is used as the index of the data source to be created, and a unique correspondence is established between the index and the table filter expression identifier corresponding to the table filter object; If not found, create a data source index based on the data source identifier in the table filter expression identifier and the query data source part, and establish a unique correspondence between the created data source index and the table filter expression identifier corresponding to the table filter object.

5. The data filtering method according to claim 4, characterized in that: The method also includes: When a data source is updated, the index of the data source corresponding to the updated data source in the global static object is updated.

6. A data filtering device, characterized in that: include: A first determining module is configured to determine, based on the received flow log, a target table filter object for processing the flow log; wherein the target table filter object is a large wide table; a second determining module, configured to determine an index of a target data source according to a target table filter expression identifier corresponding to the target table filter object; wherein the target table filter expression identifier and the index of the target data source have a unique corresponding relationship; and the target table filter expression identifier is determined according to the target data source identifier and a query data source portion of the target table filter expression; The filtering module is configured to obtain data from the target data source according to the index of the target data source, and transmit the obtained data to the target table filter object so that the target table filter object filters the obtained data.

7. An electronic device, characterized in that: include: processor, memory, and bus, wherein, The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the steps of the data filtering method according to any one of claims 1 to 5 by calling the program instructions.

8. A computer program product comprising computer-executable instructions, characterized in that: When executed, the instructions are used to perform the steps of the data filtering method according to any one of claims 1 to 5.

9. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to execute the steps of the data filtering method according to any one of claims 1 to 3.

Images