Method, device, electronic device and storage medium for cardinality estimation

Abstract

The present application relates to the field of computer software technology, and discloses a method for cardinality estimation, which includes receiving a cardinality estimation rule composed of one or more initial rules; calling counters associated with each of the initial rules involved in the cardinality estimation rule, wherein the counters are generated based on the hyperlogarithmic counting HLLC algorithm, and the data corresponding to each counter is the data filtered based on the initial rule corresponding to it; generating a cardinality estimation result based on the cardinality estimation rule in combination with the called counters and executing the cardinality estimation result output. In a real-time, trillion-level or infinite cardinality estimation scenario, since the HLLC algorithm is used in advance to perform an initial cardinality estimation on the data that needs to be estimated, a plurality of counters are obtained. When a query condition input by a user is received and cardinality estimation needs to be performed, the pre-generated counters can be called according to the cardinality estimation rule for calculation, thereby meeting the counting quantity and real-time requirements.

Description

Technical Field

[0001] The present application relates to the field of computer software technology, and for example, to a method, device, electronic device and storage medium for cardinality estimation. Background Art

[0002] In cardinality estimation scenarios, it is often necessary to support real-time, that is, sub-second cardinality estimation. In such real-time cardinality estimation scenarios, all current methods generally use the HyperLogLog Counting (HLLC) algorithm as the estimation algorithm. However, the counter implemented based on the HLLC algorithm can achieve real-time estimation for a small amount of data. For cardinality estimation scenarios of billions or infinite amounts, it is difficult to complete cardinality estimation in a timely manner using a counter implemented based on the HLLC algorithm.

[0003] How to solve the cardinality estimation problem in real-time, trillion-level or infinite cardinality estimation scenarios has become an urgent problem to be solved. Summary of the invention

[0004] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0005] The embodiments of the present disclosure provide a method, an apparatus, an electronic device, and a storage medium for cardinality estimation, so as to solve the cardinality estimation problem in real-time, trillion-level, or infinite cardinality estimation scenarios.

[0006] The present disclosure provides a method for cardinality estimation, including:

[0007] Receive query conditions entered by the user;

[0008] Extracting a cardinality estimation rule from the query condition, the cardinality estimation rule being a calculation relationship between N initial rules, where N is a positive integer;

[0009] From the pre-generated W counters generated based on the hyperlogarithmic counting HLLC algorithm, call the counter corresponding to each initial rule, where W is a positive integer;

[0010] Generate a cardinality estimation result based on the cardinality estimation rule in combination with the called counter;

[0011] The cardinality estimation result is output as a query result.

[0012] In some implementations, there is an intersection relationship among the calculated relationships.

[0013] In some implementations, generating a cardinality estimation result based on the cardinality estimation rule in combination with the called counter includes:

[0014] Performing equivalent conversion on the cardinality estimation rule to convert it into an equivalent rule without intersection relationship in the calculation relationship;

[0015] The equivalent rule is used to perform cardinality estimation on the called counter to generate a cardinality estimation result.

[0016] In some implementations, the cardinality estimation rule is equivalently transformed in the following manner:

[0017]

[0018] Wherein, A is the initial rule, n is an integer greater than 1, and k is an integer greater than 1.

[0019] In some implementations, before outputting the cardinality estimation result as a query result, the method further includes:

[0020] Determine a first merge relation having the largest cardinality estimation result among the calculation relations of the equivalence rule;

[0021] Obtaining an error corresponding to the first merging relationship;

[0022] The error corresponding to the first merging relationship is used as the error of the current cardinality estimation rule, and the cardinality estimation result is subjected to error reduction.

[0023] In some implementations, the error err corresponding to the first merging relationship is determined based on the following formula:

[0024]

[0025] Where ∈ is the error of the HLLC algorithm.

[0026] In some implementations, W counters are generated based on the HLLC algorithm, including:

[0027] Acquire data, where the data involves M dimensions, where M is a positive integer;

[0028] Filter the data from each dimension based on W initial rules to obtain W filtered data combinations;

[0029] An initial cardinality estimation is performed on each of the data combinations based on the HLLC algorithm to generate a counter for each data combination.

[0030] In some implementations, each initial rule includes filtering rules for some or all of the M dimensions.

[0031] The present disclosure provides a device for cardinality estimation, including:

[0032] A receiving module, used for receiving query conditions input by a user;

[0033] An extraction module, used to extract a cardinality estimation rule from the query condition, wherein the cardinality estimation rule is a calculation relationship between N initial rules, where N is a positive integer;

[0034] A calling module, used to call the counters corresponding to each initial rule from the pre-generated W counters generated based on the hyperlogarithmic counting HLLC algorithm, where W is a positive integer;

[0035] A generating module, configured to generate a cardinality estimation result based on the cardinality estimation rule in combination with the called counter;

[0036] An output module is used to output the cardinality estimation result as a query result.

[0037] In some implementations, there is an intersection relationship among the calculated relationships.

[0038] In some implementations, the generating module generates a cardinality estimation result based on the cardinality estimation rule in combination with the called counter, for:

[0039] Performing equivalent conversion on the cardinality estimation rule to convert it into an equivalent rule without intersection relationship in the calculation relationship;

[0040] The equivalent rule is used to perform cardinality estimation on the called counter to generate a cardinality estimation result.

[0041] In some implementations, the generation module performs equivalent conversion on the cardinality estimation rule in the following manner:

[0042]

[0043] Wherein, A is the initial rule, n is an integer greater than 1, and k is an integer greater than 1.

[0044] In some implementations, before the output module outputs the cardinality estimation result as a query result, it is further configured to:

[0045] Determine a first merge relation having the largest cardinality estimation result among the calculation relations of the equivalence rule;

[0046] Obtaining an error corresponding to the first merging relationship;

[0047] The error corresponding to the first merging relationship is used as the error of the current cardinality estimation rule, and the cardinality estimation result is subjected to error reduction.

[0048] In some implementations, the error err corresponding to the first merging relationship is determined based on the following formula:

[0049]

[0050] Where ∈ is the error of the HLLC algorithm.

[0051] In some implementations, W counters are generated based on the HLLC algorithm, including:

[0052] Acquire data, where the data involves M dimensions, where M is a positive integer;

[0053] Filter the data from each dimension based on W initial rules to obtain W filtered data combinations;

[0054] An initial cardinality estimation is performed on each of the data combinations based on the HLLC algorithm to generate a counter for each data combination.

[0055] In some implementations, each initial rule includes filtering rules for some or all of the M dimensions.

[0056] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are executed by a processor to execute the method provided by the embodiment of the present disclosure.

[0057] An embodiment of the present disclosure further provides an electronic device, including a processor and a memory, wherein the memory stores computer instructions, and the processor is configured to execute the method provided by the embodiment of the present disclosure based on the computer instructions.

[0058] The method, device, storage medium and electronic device for cardinality estimation provided by the embodiments of the present disclosure can achieve the following technical effects: in real-time, trillion-level or infinite cardinality estimation scenarios, since the HLLC algorithm is used in advance to perform initial cardinality estimation on the data that needs cardinality estimation, multiple counters are obtained. When the query conditions input by the user are received and cardinality estimation is required, the pre-generated counters can be called for calculation according to the cardinality estimation rules, thereby meeting the counting quantity and real-time requirements.

[0059] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0061] Figure 1 is one of the flow charts of a method for cardinality estimation provided by an embodiment of the present disclosure;

[0062] Figure 2 This is a second flowchart of a method for cardinality estimation provided by an embodiment of the present disclosure;

[0063] Figure 3 This is a third flowchart of a method for cardinality estimation provided by an embodiment of the present disclosure;

[0064] Figure 4 This is a fourth flowchart of a method for cardinality estimation provided by an embodiment of the present disclosure;

[0065] Figure 5 is a structural schematic diagram for cardinality estimation provided by an embodiment of the present disclosure;

[0066] Figure 6 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0067] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0068] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0069] In the embodiments of the present disclosure, the terms "upper", "lower", "inside", "middle", "outside", "front", "back" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings. These terms are mainly intended to better describe the embodiments of the present disclosure and their embodiments, and are not intended to limit the indicated devices, elements or components to have a specific direction, or to be constructed and operated in a specific direction. Moreover, in addition to being used to indicate directions or positional relationships, some of the above terms may also be used to indicate other meanings. For example, the term "upper" may also be used to indicate a certain dependency or connection relationship in certain circumstances. For those of ordinary skill in the art, the specific meanings of these terms in the embodiments of the present disclosure can be understood according to specific circumstances.

[0070] In addition, the terms "disposed", "connected", and "fixed" should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral structure; it can be a mechanical connection, or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be an internal connection between two devices, elements, or components. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present disclosure can be understood according to specific circumstances.

[0071] Unless otherwise stated, the term "plurality" means two or more.

[0072] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0073] The embodiments of the present disclosure provide a method, an apparatus, an electronic device, and a storage medium for cardinality estimation, so as to solve the cardinality estimation problem in real-time, trillion-level, or infinite cardinality estimation scenarios.

[0074] like Figure 1 As shown, the embodiment of the present disclosure provides a method for cardinality estimation, including:

[0075] S101, receiving a query condition input by a user;

[0076] S102, extracting a cardinality estimation rule from the query condition, where the cardinality estimation rule is a calculation relationship between N initial rules, where N is a positive integer;

[0077] S103, calling the counter corresponding to each initial rule from the pre-generated W counters generated based on the hyperlogarithmic counting HLLC algorithm, where W is a positive integer;

[0078] S104, generating a cardinality estimation result based on the cardinality estimation rule and the called counter;

[0079] S105: Output the cardinality estimation result as a query result.

[0080] In real-time, trillion-level or infinite cardinality estimation scenarios, the HLLC algorithm is used in advance to perform initial cardinality estimation on the data that needs cardinality estimation, and multiple counters are obtained. When the query conditions entered by the user are received and cardinality estimation is required, the pre-generated counters can be called for calculation according to the cardinality estimation rules, thereby meeting the counting quantity and real-time requirements.

[0081] One use scenario of the disclosed embodiment is the field of advertising monitoring data. In the scenario of sub-second real-time estimation of the cardinality of device identification (Identity document, ID), due to the large amount of data logs and multiple dimensions, the original data cannot be put into the database. The HLLC algorithm can be used to pre-estimate the initial cardinality based on the ID cardinality of various dimensions, generate counters, and pre-store them in binary form for real-time query. After receiving the query instruction from the client, the system needs to use the query condition to filter each dimension to obtain a large number of counters, and merge the counters to obtain the final query condition value. However, in the cardinality estimation rule of the query condition conversion, when there is intersection or intersection, the result cannot be directly obtained, such as: the query requirement is to count the people who have seen brand advertisement A or the people who have seen brand advertisement B. Such statistical requirements can be achieved using pre-aggregated counters; but the query requirement is to count the people who have seen brand advertisement A and the people who have seen brand advertisement B, which cannot be achieved using pre-aggregated counters. In the scenario of cardinality estimation of massive data sets, in order to meet the demand for real-time and fast estimation, the present disclosure also proposes a method for transforming cardinality estimation rules, converting the intersection in the cardinality estimation rules into a merged form, thereby realizing real-time acquisition of cardinality estimation results when there is intersection or intersection in the cardinality estimation rules. Among them, if the initial rules in the calculation relationship of the cardinality estimation rules are in a merge relationship, the cardinality estimation can be directly performed by calling the counter of the corresponding initial rule, but if there is an intersection relationship in the calculation relationship of the cardinality estimation rules, the cardinality estimation rules need to be converted to achieve cardinality estimation.

[0082] In some implementations, when there is an intersection relationship in the calculation relationship of the cardinality estimation rule, such as Figure 2 As shown, S104 generates a cardinality estimation result based on the cardinality estimation rule in combination with the called counter, including:

[0083] S201, performing equivalent conversion on the cardinality estimation rule to convert it into an equivalent rule without intersection relationship in the calculation relationship;

[0084] S202: Use the equivalent rule to perform cardinality estimation on the called counter to generate a cardinality estimation result.

[0085] In practical applications, the cardinality estimation rule can be equivalently transformed in the following manner:

[0086]

[0087] Wherein, A is the initial rule, n is an integer greater than 1, and k is an integer greater than 1.

[0088] In some embodiments, Figure 3 As shown, before S105 outputs the cardinality estimation result as the query result, it also includes:

[0089] S301, determining a first merging relationship having the largest cardinality estimation result among the calculation relationships of the equivalent rule;

[0090] S302, obtaining an error corresponding to the first merging relationship;

[0091] S303: Taking the error corresponding to the first merging relationship as the error of the current cardinality estimation rule, and performing error reduction on the cardinality estimation result.

[0092] For the error reduction method, those skilled in the art may refer to the conventional error reduction method of the HLLC algorithm, and the present disclosure is not limited thereto.

[0093] For other forms of cardinality estimation rules, those skilled in the art may set the error of error reduction according to needs, and the present disclosure is not limited thereto.

[0094] In some implementations, the error err corresponding to the first merging relationship is determined based on the following formula:

[0095]

[0096] Where ∈ is the error of the HLLC algorithm.

[0097] In some embodiments, Figure 4 As shown, W counters are generated based on the HLLC algorithm, including:

[0098] S401, obtaining data, where the data involves M dimensions, where M is a positive integer;

[0099] S402, filtering the data from each dimension based on W initial rules to obtain W filtered data combinations;

[0100] S403: Perform initial cardinality estimation on each of the data combinations based on the HLLC algorithm to generate a counter for each data combination.

[0101] In practical applications, a data table T can be pre-built to store the acquired data, and the table schema can be expressed as: counter, dimension 1, dimension 2, ..., dimension M. The dimensions can be provinces, cities, exposures, clicks, or other dimensions set by technicians in this field according to actual needs.

[0102] Then based on Figure 4 In the manner shown, W counters are generated and stored in binary form for subsequent use after receiving the user's query conditions.

[0103] In some implementations, each initial rule includes filtering rules for some or all of the M dimensions.

[0104] In the actual application, among the W initial rules, the first initial rule can be expressed as:

[0105] A={r1,r2,...,r u},

[0106] Among them, r1, r2, ..., r u represents the filtering rules of some dimensions in the M dimensions, and u is a positive integer less than or equal to M. Each specific rule r u It is used to filter the data under a certain dimension with one constraint, such as equality, inequality (including not equal to, greater than or less than), inclusion, or multiple constraints.

[0107] like Figure 5 As shown, the embodiment of the present disclosure also provides a device for cardinality estimation, including:

[0108] Receiving module 501, used to receive query conditions input by a user;

[0109] An extraction module 502, configured to extract a cardinality estimation rule from the query condition, wherein the cardinality estimation rule is a calculation relationship between N initial rules, where N is a positive integer;

[0110] The calling module 503 is used to call the counter corresponding to each initial rule from the pre-generated W counters generated based on the hyperlogarithmic counting HLLC algorithm, where W is a positive integer;

[0111] A generating module 504, configured to generate a cardinality estimation result based on the cardinality estimation rule in combination with the called counter;

[0112] The output module 505 is used to output the cardinality estimation result as a query result.

[0113] In some implementations, there is an intersection relationship among the calculated relationships.

[0114] In some implementations, the generating module 504 generates a cardinality estimation result based on the cardinality estimation rule in combination with the called counter, for:

[0115] Performing equivalent conversion on the cardinality estimation rule to convert it into an equivalent rule without intersection relationship in the calculation relationship;

[0116] The equivalent rule is used to perform cardinality estimation on the called counter to generate a cardinality estimation result.

[0117] In some implementations, the generating module 504 performs equivalent conversion on the cardinality estimation rule in the following manner:

[0118]

[0119] Wherein, A is the initial rule, n is an integer greater than 1, and k is an integer greater than 1.

[0120] In some implementations, before the output module 505 outputs the cardinality estimation result as a query result, it is further configured to:

[0121] Determine a first merge relation having the largest cardinality estimation result among the calculation relations of the equivalence rule;

[0122] Obtaining an error corresponding to the first merging relationship;

[0123] The error corresponding to the first merging relationship is used as the error of the current cardinality estimation rule, and the cardinality estimation result is subjected to error reduction.

[0124] In some implementations, the error err corresponding to the first merging relationship is determined based on the following formula:

[0125]

[0126] Where ∈ is the error of the HLLC algorithm.

[0127] In some implementations, the apparatus provided by the embodiments of the present disclosure further includes the following modules to implement generation of W counters based on the HLLC algorithm, including:

[0128] An acquisition module 506 is used to acquire data, where the data involves M dimensions, where M is a positive integer;

[0129] A filtering module 507, configured to filter the data from each dimension based on W initial rules to obtain W filtered data combinations;

[0130] The estimation module 508 is used to perform initial cardinality estimation on each of the data combinations based on the HLLC algorithm to generate a counter for each data combination.

[0131] In some implementations, each initial rule includes filtering rules for some or all of the M dimensions.

[0132] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are executed by a processor to execute the method provided by the embodiment of the present disclosure.

[0133] like Figure 6 The embodiment of the present disclosure further provides an electronic device, including a processor 601 and a memory 602, wherein the memory 602 stores computer instructions, and the processor 601 is configured to execute the method provided by the embodiment of the present disclosure based on the computer instructions.

[0134] The method, device, storage medium and electronic device for cardinality estimation provided by the embodiments of the present disclosure can achieve the following technical effects: in real-time, trillion-level or infinite cardinality estimation scenarios, since the HLLC algorithm is used in advance to perform initial cardinality estimation on the data that needs cardinality estimation, multiple counters are obtained. When the query conditions input by the user are received and cardinality estimation is required, the pre-generated counters can be called for calculation according to the cardinality estimation rules, thereby meeting the counting quantity and real-time requirements.

[0135] The above description and the accompanying drawings sufficiently illustrate the embodiments of the present disclosure to enable those skilled in the art to practice them. Other embodiments may include structural and other changes. The embodiments represent only possible variations. Unless explicitly required, individual components and functions are optional, and the order of operations may vary. Portions and features of some embodiments may be included in or replace portions and features of other embodiments. The embodiments of the present disclosure are not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for cardinality estimation, characterized in that include: Receive query conditions entered by the user; Extracting a cardinality estimation rule from the query condition, the cardinality estimation rule being a calculation relationship between N initial rules, where N is a positive integer; From the pre-generated W counters generated based on the hyperlogarithmic counting HLLC algorithm, call the counter corresponding to each initial rule, where W is a positive integer; Generate a cardinality estimation result based on the cardinality estimation rule in combination with the called counter; Outputting the cardinality estimation result as a query result; There is an intersection relationship in the calculation relationship; Generate a cardinality estimation result based on the cardinality estimation rule in combination with the called counter, including: Performing equivalent conversion on the cardinality estimation rule to convert it into an equivalent rule without intersection relationship in the calculation relationship; Using the equivalent rule to perform cardinality estimation on the called counter to generate a cardinality estimation result; The cardinality estimation rule is equivalently transformed as follows: Where A is the initial rule, n is an integer greater than 1, and k is an integer greater than 1; A1, A2, ... A n are multiple initial rules before conversion, A i1 , A i2 ……A ik are multiple equivalent rules with no intersection relations after conversion; Generate W counters based on the HLLC algorithm, including: Acquire data, where the data involves M dimensions, where M is a positive integer; Filter the data from each dimension based on W initial rules to obtain W filtered data combinations; Performing an initial cardinality estimation on each of the data combinations based on the HLLC algorithm to generate a counter for each data combination; Each initial rule includes filtering rules for some or all of the M dimensions.

2. The method according to claim 1, characterized in that Before outputting the cardinality estimation result as the query result, the method further includes: Determine a first merge relation having the largest cardinality estimation result among the calculation relations of the equivalence rule; Obtaining an error corresponding to the first merging relationship; The error corresponding to the first merging relationship is used as the error of the current cardinality estimation rule, and the cardinality estimation result is subjected to error reduction.

3. The method according to claim 2, characterized in that The error err corresponding to the first merging relationship is determined based on the following formula: Where ∈ is the error of the HLLC algorithm.

4. The method according to claim 1, characterized in that Among the W initial rules, the first initial rule is expressed as: A＝{r1、r2……r u}， Among them, r1, r2, ... r u represents the filtering rules of some dimensions in the M dimensions, u is a positive integer less than or equal to M; each specific rule r u It is used to filter data under a certain dimension based on one or more constraints, such as equality, inequality, or inclusion.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are executed by a processor to perform the method according to any one of claims 1 to 3.

6. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer instructions, and the processor is configured to execute the method according to any one of claims 1 to 3 based on the computer instructions.

Images