Data stream oriented summary addressing optimization method, device and equipment and storage medium

Abstract

The application relates to a data stream summary addressing optimization method, device and equipment and a storage medium. The method comprises the following steps: constructing a data summary structure; the physical storage structure of the data summary structure comprises a plurality of buckets, each bucket comprises three domain counters, the data summary structure supports real-time estimation of an aggregated variance according to the values of the three domain counters and returns an aggregated estimation result; performing online addressing according to the aggregated estimation result, selecting a bucket meeting a user aggregated variance threshold or expanding a summary space and returning an empty bucket; performing a data insertion operation according to the bucket index returned by the addressing and the key value of the data and recording the addressing index; and performing data query according to the data key to be queried and the addressing index and returning a query result. The method always guarantees that the aggregated variance of the data stream elements meets a user-defined aggregated variance threshold, the threshold of the aggregated variance can be freely adjusted according to the actual scene needs of the user, and the quality of data stream aggregation is guaranteed.

Description

Technical Field

[0001] This application relates to the field of network communication technology, and in particular to a method, apparatus, device and storage medium for optimizing data flow summary addressing. Background Technology

[0002] Data stream summarization is an important method in the field of big data, providing online approximate queries of data streams using a constant space. Its main idea is to pre-construct a contiguously distributed array in memory, organized into buckets, for dynamically arriving key-value data.<key,value> It addresses one or more buckets from an array and aggregates (usually by summing) the data values ​​into the selected buckets. During a query, the same addressing process is used to select the corresponding buckets, and the aggregated result of the buckets is used as an approximate value of the data.

[0003] The insertion and query of data streams depend on the addressing method to achieve fast aggregation of any number of data without changing the data storage space. The addressing method of the data stream summary determines the location of the summary buckets. The mainstream addressing techniques include key-based or value-based methods: (1) Key-based method, which uses a pre-determined hash function to determine the hash value of the data key and determines the bucket in the array through modulo operation. This method is simple to calculate and is independent of the data value, so it is the most popular. However, its disadvantage is that the hash calculation process is expensive and the hash function is not aware of the data distribution, which may lead to large differences in the values ​​of the same data being addressed, resulting in a higher deviation of the aggregation calculation result from the actual data. (2) Value-based method, which pre-classifies the values ​​of the data stream, aggregates key-value pairs of similar data streams into the same bucket, and uses the average value of the bucket as the aggregation result, so that the aggregation mean is as close as possible to the actual value. This method minimizes the aggregation error caused by the difference in data values. However, its disadvantage is that it lacks a unified addressing index, the aggregation result lacks accuracy guarantee, and the data is classified before insertion, which affects the data processing flow and may not be feasible in many real-time data analysis scenarios. Summary of the Invention

[0004] Therefore, it is necessary to provide a data flow summarization addressing optimization method, apparatus, device, and storage medium to address the aforementioned technical problems. This method can solve the problem that existing summarization techniques lack awareness of aggregation errors.

[0005] A data flow summary addressing optimization method, the method comprising: Construct a data summary structure; the physical storage structure of the data summary structure includes multiple buckets, each bucket including three domain counters, and the data summary structure supports real-time estimation of aggregate variance based on the values ​​of the three domain counters and returns the aggregate estimation results.

[0006] Based on the aggregation estimation results, perform online addressing, select a bucket that meets the user's aggregation variance threshold, or expand the summary space and return an empty bucket.

[0007] The data insertion operation is performed based on the bucket index returned by the address and the key value of the data, and the address index is recorded.

[0008] The data is queried based on the data key to be queried and the addressing index, and the query results are returned.

[0009] In one embodiment, a data summary structure is constructed, wherein the physical storage structure of the data summary structure includes three domain counters: a sum domain counter, a symbol sum domain counter, and a count domain counter.

[0010] The sum and field counter is used to accumulate the value of the data.

[0011] The sign and field counters are used to accumulate the sign weight multiplier of the sum; the sign weight is a hash value calculated using the key of the input data.

[0012] The counter in the counting field is used to accumulate the total number of data keys.

[0013] In one embodiment, aggregate variance is calculated and estimated in real time based on the values ​​of the three domain counters, and the aggregate estimation result is returned, including: The ratio of the sum-counter value to the counter value is used as the mean.

[0014] The absolute value of the squared difference between the values ​​of the sign and domain counters and the mean is used as the aggregate variance.

[0015] Return the mean, the aggregate variance, and the values ​​of the bucket sum field counter, the sign sum field counter, and the count field counter.

[0016] In one embodiment, online addressing is performed based on the aggregation estimation results to select a bucket that meets the user's aggregation variance threshold, or the summary space is expanded to return an empty bucket, including: Iterate through the current array, record the set of non-empty buckets in the array, and set the index of the initialized empty bucket to the position of the first empty bucket found.

[0017] Calculate the optimal aggregation variance of the non-empty bucket set: ,in Representing the i Aggregate variance of individual buckets.

[0018] If the optimal aggregation variance is not greater than the user aggregation variance threshold, the bucket corresponding to the optimal aggregation variance is returned; if the non-empty bucket set is not empty, the index of the first bucket is returned.

[0019] If the optimal aggregation variance is greater than the user aggregation variance threshold, then the summary array is expanded, and the first empty bucket of the expanded array is returned; the specific steps of the summary array expansion include: allocating new memory... m The array is initialized with zero fields for each empty bucket; the empty buckets are appended to the end of the current array, and the size of the array is updated to twice the number of empty buckets.

[0020] In one embodiment, a data insertion operation is performed based on the bucket index returned by the address and the key value of the data, and the address index is recorded, including: The bucket is updated based on the key-value pairs of the data. The update method is as follows:

[0021]

[0022] When the key of the data is appearing for the first time:

[0023] in, For the updated bucket, SUM For the value of the field counter, SSUM For the values ​​of the symbol and field counters, CNT The value of the counter in the counting field, < key , value > represents key-value pairs of data.

[0024] Determine the key of the first occurrence of data based on the index of the dictionary or approximate data record bucket, and insert the index dictionary based on the key of the data.

[0025] If the key for the data exists, return all corresponding bucket indices; otherwise, return False.

[0026] In one embodiment, the key of the first occurrence of data is determined based on the index of a dictionary or approximate data record bucket, and index dictionary insertion is performed based on the key of the data, including: Determine the key of the first occurrence of data based on the index of the record bucket in the dictionary or an approximate data structure.

[0027] If the key of the first occurrence of data does not exist in the dictionary, then <key, >Append to the bucket index.

[0028] When the key of the first occurrence of data exists in the dictionary, and the bucket index... If not in the record, add it. <key, >To bucket index.

[0029] When the key of the first occurrence of data exists in the dictionary, and the bucket index is used... If the record is in progress, the dictionary will not be updated.

[0030] In one embodiment, a data query is performed based on the key of the data to be queried and the addressing index, and the query results are returned, including: The bucket index dictionary is queried based on the key of the data to be queried and the addressing index, and the set of bucket indexes corresponding to the key of the data to be queried is returned.

[0031] The sum of the averages of all buckets is calculated based on the set of bucket indices, and the sum of the averages is returned.

[0032] A data flow summary addressing optimization apparatus, the apparatus comprising: A data summary structure construction module is used to construct a data summary structure. The physical storage structure of the data summary structure includes multiple buckets, each bucket including three domain counters. The data summary structure supports real-time estimation of aggregate variance based on the values ​​of the three domain counters and returns the aggregate estimation results.

[0033] The online addressing module is used to perform online addressing based on the aggregation estimation results, select buckets that meet the user's aggregation variance threshold, or expand the summary space and return an empty bucket.

[0034] The data insertion module is used to perform data insertion operations based on the bucket index returned by the address and the key value of the data, and to record the address index.

[0035] The data query module is used to perform data queries based on the data key to be queried and the addressing index, and return the query results.

[0036] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement any of the methods described above.

[0037] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods described above.

[0038] The aforementioned data stream summary addressing optimization method, apparatus, device, and storage medium include the following steps: constructing a data summary structure; the physical storage structure of the data summary structure includes multiple buckets, each bucket including three domain counters; the data summary structure supports real-time estimation of aggregation variance based on the values ​​of the three domain counters and returns the aggregation estimation result; performing online addressing based on the aggregation estimation result, selecting a bucket that meets the user's aggregation variance threshold, or expanding the summary space and returning an empty bucket; performing data insertion operations based on the bucket index returned by the addressing and the data key value, recording the addressing index; and performing data querying based on the data key to be queried and the addressing index, returning the query result. This method consistently ensures that the aggregation variance of data stream elements meets the user-defined aggregation variance threshold, and the aggregation variance threshold can be freely adjusted according to the user's actual scenario needs, ensuring the quality of data stream aggregation. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating a data flow-oriented summary addressing optimization method in one embodiment; Figure 2 This is the architecture of the data summary structure in another embodiment; Figure 3 This is the flow chart of a data flow-oriented summary addressing optimization method in another embodiment; Figure 4 This is a block diagram of a data flow-oriented summary addressing optimization device in one embodiment; Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0041] In one embodiment, such as Figure 1 As shown, a data flow-oriented summary addressing optimization method is provided, which includes the following steps: Step 100: Construct the data summary structure; the physical storage structure of the data summary structure includes multiple buckets, each bucket includes three domain counters. The data summary structure supports real-time estimation of aggregate variance based on the values ​​of the three domain counters and returns the aggregate estimation results.

[0042] Specifically, it constructs data summaries, provides data aggregation calculations, and supports real-time estimation of the aggregation variance of elements in the bucket.

[0043] Specifically, Figure 2This is the system architecture of the data summary structure. It inserts data key-value pairs <0x45, 20>, sets the user-defined aggregation variance threshold to 100, and presents a data summary after some elements have been inserted. The addressing process calculates the aggregation variance estimate and selects the third bucket as the optimal bucket position. The insertion process updates the buckets and their indexes. The query process selects the return position of the bucket index dictionary and obtains the aggregation calculation result. It can be seen that this invention ensures that the aggregation variance of the buckets meets the user-defined threshold.

[0044] Step 102: Perform online addressing based on the aggregation estimation results, select a bucket that meets the user's aggregation variance threshold, or expand the summary space and return an empty bucket; Specifically, online addressing selects a bucket that meets the user-defined aggregation variance threshold, or expands the summary space and returns an empty bucket.

[0045] Step 104: Perform the data insertion operation based on the bucket index returned by the address and the key value of the data, and record the address index; Step 106: Perform a data query based on the data key and addressing index, and return the query results.

[0046] The aforementioned optimization method for data stream summary addressing includes: constructing a data summary structure; the physical storage structure of the data summary structure includes multiple buckets, each bucket including three domain counters; the data summary structure supports real-time estimation of aggregation variance based on the values ​​of the three domain counters and returns the aggregation estimation result; performing online addressing based on the aggregation estimation result, selecting a bucket that meets the user's aggregation variance threshold, or expanding the summary space and returning an empty bucket; performing data insertion operations based on the bucket index returned by the addressing and the data key value, and recording the addressing index; performing data query based on the data key to be queried and the addressing index, and returning the query result. This method always ensures that the aggregation variance of data stream elements meets the user-defined aggregation variance threshold, and the aggregation variance threshold can be freely adjusted according to the user's actual scenario needs to ensure the quality of data stream aggregation.

[0047] In one embodiment, step 100 includes: constructing a data summary structure, wherein the physical storage structure of the data summary structure includes three domain counters: a sum domain counter, a sign sum domain counter, and a count domain counter; the sum domain counter is used to accumulate the value of the data; the sign sum domain counter is used to accumulate the multiplication of the sign weight of the sum; the sign weight is a hash value calculated using the key of the input data; and the count domain counter is used to accumulate the total number of data keys.

[0048] Specifically, construct a data summary. Figure 2 It is the system architecture of the data summary structure.

[0049] (1.1) Composition of physical storage structure: In summary, it consists of... mIt consists of 128 (system parameter, default) buckets. Each bucket includes three field counters, and each field of the bucket is initialized to all zeros: (1.1.1) and the field (SUM): the value of the accumulated data; (1.1.2) Symbol and Field (SSUM): The symbol weighted (+1 / -1) multiplication of the accumulated data is actually calculated through a hash function, that is, the hash value h(key) is calculated using the key of the input data, which is in the range of +1 / -1, and then the hash value is multiplied by the value of the data as the input of the symbol sum; (1.1.3) Count field (CNT): The total number of data keys is accumulated, so the same key is counted only once; (1.2) Calculation of aggregate variance: (1.2.1) Mean (denoted as mu): Returns the ratio of the sum domain to the count domain, i.e., mu = SUM / CNT. The mean represents the average unbiased estimate of the aggregation result. (1.2.2) Aggregate variance: Returns the absolute value of the squared difference between the sign, domain, and mean, i.e., delta = |SSUM 2 -mu 2 Aggregate variance represents the degree of deviation of the inserted elements; (1.3) Aggregation result calculation: Return the mean, aggregation variance, and bucket count CNT, SUM, symbol and SSUM calculated in step (1.2).

[0050] In one embodiment, the aggregate variance is calculated in real time based on the values ​​of the three domain counters, and the aggregate estimation result is returned, including: taking the ratio of the sum domain counter value to the count domain counter value as the mean; taking the absolute value of the squared difference between the sign and domain counter values ​​and the mean as the aggregate variance; and returning the mean, aggregate variance, and the values ​​of the sum domain counter, sign and domain counter, and count domain counter of the bucket.

[0051] In one embodiment, step 102 includes: traversing the current array, recording the set of non-empty buckets in the array, and setting the index of the initialized empty bucket to the position of the first empty bucket found; calculating the optimal aggregation variance of the set of non-empty buckets: ,in Representing the i The aggregation variance of each bucket is calculated; if the optimal aggregation variance is not greater than the user's aggregation variance threshold, the bucket corresponding to the optimal aggregation variance is returned; if the set of non-empty buckets is not empty, the index of the first bucket is returned; if the optimal aggregation variance is greater than the user's aggregation variance threshold, the summary array is expanded, and the first empty bucket of the expanded array is returned; the specific steps of the summary array expansion include: allocating new memory... mThe array is initialized with zero fields for each empty bucket; the empty buckets are appended to the end of the current array, and the size of the array is updated to twice the number of empty buckets.

[0052] Specifically, online addressing uses a user-defined aggregation variance threshold tau (default 100) as the upper bound of the aggregation variance of buckets, selecting only the optimal bucket that meets the criteria.

[0053] (2.1) Estimating the bucket by aggregate variance optimally: (2.1.1) Initialize the index of the empty bucket, empty_bkt (default is -1); (2.1.2) Traverse the current array and record the set of non-empty buckets of the array (denoted as ). non_empty_bkt_set ), and will empty_bkt Set the position of the first empty bucket found; (2.1.3) Calculate the optimal aggregation variance of the non-empty bucket set: ,in This represents the aggregation variance of the i-th bucket obtained through step (1.2.2); (2.1.4) If Not greater than tau If the aggregation variance threshold is met, then return to the bucket. ; (2.1.5) If not an empty bucket set non_empty_bkt_set If not empty, return the index of the first bucket; (2.1.6) Otherwise, go to (2.2) to expand the array and return the first empty bucket of the expanded array; (2.2) Summary Structure Expansion: (2.2.1) Doubling the array structure: Allocating new memory m The array is initialized with empty buckets, their fields are set to zero, the empty buckets are appended to the end of the current array, and the array size is updated. m :=2* m .

[0054] In one embodiment, step 104 includes: updating the bucket based on the key-value pairs of the data, wherein the update method is as follows:

[0055]

[0056] When the key of the data is appearing for the first time:

[0057] in, For the updated bucket, SUM For the value of the field counter, SSUM For the values ​​of the symbol and field counters, CNTThe value of the counter in the counting field, < key , value > represents key-value pairs of data.

[0058] Determine the key of the first occurrence of data based on the index of the dictionary or approximate data record bucket. Insert the index dictionary based on the key of the data. If the key of the data exists, return all corresponding bucket indices; otherwise, return False.

[0059] Specifically, data insertion: targeting key-value pairs of data (<key,value> The addressing procedure is called to return the bucket index (denoted as ). ), update bucket (denoted as ).

[0060] (3.1) Bucket update: (3.1.1) ; (3.1.2) ; (3.1.3) If the key of the data appears for the first time, ; (3.2) Bucket Index Dictionary Update: The index of the bucket is recorded through a dictionary or approximate data structure, and the key of the first occurrence of the data is recorded. It should be noted that different bucket indexes are not directly related to addressing optimization. Only the addressing result is recorded to facilitate the determination of whether the key of the data is the first occurrence (involving update (3.1.2)).

[0061] (3.2.1) Bucket index dictionary insertion: If the key does not exist in the dictionary, then <key, >Append to the bucket index; If the key exists in the dictionary, and the bucket index is used... If not in the record, add it. <key, >To the bucket index; If the key exists in the dictionary and is indexed by a bucket. If the record is in progress, the dictionary will not be updated.

[0062] (3.2.2) Bucket index dictionary query: If the key of the data exists, return all corresponding bucket indexes; otherwise, return False.

[0063] In one embodiment, the key of the first occurrence of data is determined based on the index of a dictionary or an approximate data record bucket. Index dictionary insertion is then performed based on the data key, including: determining the key of the first occurrence of data based on the index of a dictionary or an approximate data structure record bucket; if the key of the first occurrence of data does not exist in the dictionary, then... <key, >Append to the bucket index; when the key of the first occurrence of data exists in the dictionary, and the bucket index is not present. If not in the record, add it. <key, >To the bucket index; when the key of the first occurrence of data exists in the dictionary, and the bucket index If the record is in progress, the dictionary will not be updated.

[0064] In one embodiment, step 106 includes: querying the bucket index dictionary based on the key and addressing index of the data to be queried, and returning the bucket index set corresponding to the key of the data to be queried; calculating the mean sum of each bucket based on the bucket index set, and returning the mean sum.

[0065] Specifically, the data query steps include: (4.1) Query the bucket index dictionary: Returns the set of bucket indexes corresponding to the keys of the data (denoted as set); (4.2) Aggregation result calculation: Returns the sum of the averages of all buckets in the bucket index set, i.e.: ,in Calculated according to (1.2.1).

[0066] In one embodiment, such as Figure 3 As shown, a data flow summary addressing optimization method is provided, with the following steps: First, initialize the data summary structure. Second, based on the user aggregation variance, call the addressing procedure to select a bucket that meets the user aggregation variance. Third, aggregate the data into the bucket returned by the addressing. Fourth, based on the data key to be queried, return the estimated result of the data summary.

[0067] It should be understood that, although Figure 1 , Figure 3 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 , Figure 3 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0068] In one embodiment, such as Figure 4 As shown, a data flow summary addressing optimization device is provided, including a data summary structure construction module, an online addressing module, a data insertion module, and a data query module, wherein: The data summary structure building module is used to build the data summary structure. The physical storage structure of the data summary structure includes multiple buckets, each bucket includes three domain counters. The data summary structure supports real-time estimation of the aggregate variance based on the values ​​of the three domain counters and returns the aggregate estimation results. The online addressing module is used to perform online addressing based on the aggregation estimation results, select buckets that meet the user's aggregation variance threshold, or expand the summary space and return empty buckets; The data insertion module is used to perform data insertion operations based on the bucket index returned by the address and the key value of the data, and to record the address index; The data query module is used to perform data queries based on the data key and address index to be queried, and return the query results.

[0069] In one embodiment, the data summary structure building module is also used to build a data summary structure. The physical storage structure of the data summary structure includes three domain counters: a sum domain counter, a sign sum domain counter, and a count domain counter. The sum domain counter is used to accumulate the value of the data. The sign sum domain counter is used to accumulate the multiplication of the sign weight of the sum. The sign weight is a hash value calculated using the key of the input data. The count domain counter is used to accumulate the total number of data keys.

[0070] In one embodiment, the data summary structure building module is further configured to use the ratio of the sum field counter value to the count field counter value as the mean; use the absolute value of the squared difference between the sign sum field counter value and the mean as the aggregate variance; and return the mean, aggregate variance, and the values ​​of the sum field counter, sign sum field counter, and count field counter of the bucket.

[0071] In one embodiment, the online addressing module is further configured to traverse the current array, record the set of non-empty buckets in the array, and set the index of the initialized empty bucket to the position of the first empty bucket found; calculate the optimal aggregation variance of the set of non-empty buckets: ,in Representing the i The aggregation variance of each bucket is calculated; if the optimal aggregation variance is not greater than the user's aggregation variance threshold, the bucket corresponding to the optimal aggregation variance is returned; if the set of non-empty buckets is not empty, the index of the first bucket is returned; if the optimal aggregation variance is greater than the user's aggregation variance threshold, the summary array is expanded, and the first empty bucket of the expanded array is returned; the specific steps of the summary array expansion include: allocating new memory... m The array is initialized with zero fields for each empty bucket; the empty buckets are appended to the end of the current array, and the size of the array is updated to twice the number of empty buckets.

[0072] In one embodiment, the data insertion module is further configured to update the bucket based on the key-value pairs of the data, wherein the update method is as follows:

[0073]

[0074] When the key of the data is appearing for the first time:

[0075] in, For the updated bucket, SUM For the value of the field counter, SSUM For the values ​​of the symbol and field counters, CNT The value of the counter in the counting field, < key , value > These are key-value pairs for the data; Determine the key of the first occurrence of data based on the index of the dictionary or approximate data record bucket. Insert the index dictionary based on the key of the data. If the key of the data exists, return all corresponding bucket indices; otherwise, return False.

[0076] In one embodiment, the data insertion module is further configured to: determine the key of the first occurrence of data based on the index of a dictionary or an approximate data structure record bucket; if the key of the first occurrence of data does not exist in the dictionary, then... <key, >Append to the bucket index; when the key of the first occurrence of data exists in the dictionary, and the bucket index is not present. If not in the record, add it. <key, >To the bucket index; when the key of the first occurrence of data exists in the dictionary, and the bucket index If the record is in progress, the dictionary will not be updated.

[0077] In one embodiment, the data query module is further configured to query the bucket index dictionary based on the key and addressing index of the data to be queried, and return the bucket index set corresponding to the key of the data to be queried; calculate the mean sum of each bucket based on the bucket index set, and return the mean sum.

[0078] Specific limitations regarding the data flow-oriented summary addressing optimization apparatus can be found in the limitations of the data flow-oriented summary addressing optimization method described above, and will not be repeated here. Each module in the aforementioned data flow-oriented summary addressing optimization apparatus can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0079] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a data flow-oriented summary addressing optimization method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0080] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0081] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiment.

[0082] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0083] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0084] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data flow-oriented summary addressing optimization method, characterized in that, The method includes: Construct a data summary structure; the physical storage structure of the data summary structure includes multiple buckets, each bucket including three domain counters; wherein the three domain counters include: a domain counter for accumulating the value of the data; a sign and domain counter for accumulating the product of the sign weights of the data values, the sign weights being hash values ​​calculated using the keys of the input data; and a count domain counter for accumulating the total number of data keys; The data summary structure supports real-time estimation of aggregate variance based on the values ​​of three domain counters and returns the aggregate estimation result; specifically, it includes: using the ratio of the sum domain counter value to the count domain counter value as the mean; using the absolute value of the squared difference between the sign and domain counter values ​​and the mean as the aggregate variance; and returning the mean, the aggregate variance, and the values ​​of the sum domain counter, sign and domain counter, and count domain counter for the bucket. Based on the aggregation estimation results, online addressing is performed to select buckets that meet the user's aggregation variance threshold, or the summary space is expanded, and an empty bucket is returned. Specifically, this includes: traversing the current array, recording the set of non-empty buckets in the array, and setting the index of the initialized empty bucket to the position of the first empty bucket found; calculating the optimal aggregation variance of the set of non-empty buckets; if the optimal aggregation variance is not greater than the user's aggregation variance threshold, the bucket corresponding to the optimal aggregation variance is returned; if the set of non-empty buckets is not empty, the index of the first bucket is returned; if the optimal aggregation variance is greater than the user's aggregation variance threshold, and the set of non-empty buckets is empty, the summary array is expanded, and the first empty bucket in the expanded array is returned. Perform the data insertion operation based on the bucket index returned by the address and the key value of the data, and record the address index; The data is queried based on the data key to be queried and the addressing index, and the query results are returned.

2. The method according to claim 1, characterized in that, Based on the aggregation estimation results, online addressing is performed. The optimal aggregation variance of the non-empty bucket set is: in, Represents the aggregate variance of the i-th bucket; The specific steps for expanding a summary array include: allocating new memory... m There are several empty buckets; the fields of the empty buckets are initialized to zero; the empty buckets are appended to the end of the current array, and the size of the array is updated to twice the number of empty buckets. m It is a positive integer.

3. The method according to claim 1, characterized in that, The data insertion operation is performed based on the bucket index returned by the address and the key value of the data. The address index is recorded, including: The bucket is updated based on the key-value pairs of the data. The update method is as follows: When the key of the data is appearing for the first time: in, For the updated bucket, SUM For the value of the field counter, SSUM For the values ​​of the symbol and field counters, CNT The value of the counter in the counting field, < key , value > These are key-value pairs for the data; Determine the key of the first occurrence of data based on the index of the dictionary or approximate data record bucket, and insert the index dictionary based on the key of the data; If the key for the data exists, return all corresponding bucket indices; otherwise, return False.

4. The method according to claim 3, characterized in that, Based on the index of the dictionary or approximate data record bucket, determine the key of the first occurrence of the data, and perform index dictionary insertion based on the data key, including: Determine the key of the first occurrence of data based on the index of the record bucket in the dictionary or an approximate data structure; If the key of the first occurrence of data does not exist in the dictionary, then <key, >Append to the bucket index; When the key of the first occurrence of data exists in the dictionary, and the bucket index... If not in the record, add it. <key, >To the bucket index; When the key of the first occurrence of data exists in the dictionary, and the bucket index is used... If the record is in progress, the dictionary will not be updated.

5. The method according to claim 1, characterized in that, The data is queried based on the key of the data to be queried and the addressing index, and the query results are returned, including: Based on the key of the data to be queried and the addressing index, the bucket index dictionary is queried, and the set of bucket indexes corresponding to the key of the data to be queried is returned. The sum of the averages of all buckets is calculated based on the set of bucket indices, and the sum of the averages is returned.

6. A data flow-oriented summary addressing optimization apparatus, characterized in that, The device includes: A data summary structure construction module is used to construct a data summary structure. The physical storage structure of the data summary structure includes multiple buckets, each bucket including three domain counters. These three domain counters include: a sum domain counter, used to accumulate data values; a sign sum domain counter, used to accumulate the product of the sign weights of the data values, where the sign weights are hash values ​​calculated using the keys of the input data; and a count domain counter, used to accumulate the total number of data keys. The data summary structure supports real-time estimation of the aggregation variance based on the values ​​of the three domain counters and returns the aggregation estimation result. Specifically, this includes: using the ratio of the sum domain counter value to the count domain counter value as the mean; using the absolute value of the squared difference between the sign sum domain counter value and the mean as the aggregation variance; and returning the mean, the aggregation variance, and the values ​​of the sum domain counter, sign sum domain counter, and count domain counter for each bucket. The online addressing module is used to perform online addressing based on the aggregation estimation results, select buckets that meet the user's aggregation variance threshold, or expand the summary space and return empty buckets. Specifically, it includes: traversing the current array, recording the set of non-empty buckets in the array, and setting the index of the initialized empty bucket to the position of the first empty bucket found; calculating the optimal aggregation variance of the set of non-empty buckets; if the optimal aggregation variance is not greater than the user's aggregation variance threshold, returning the bucket corresponding to the optimal aggregation variance; if the set of non-empty buckets is not empty, returning the index of the first bucket; if the optimal aggregation variance is greater than the user's aggregation variance threshold and the set of non-empty buckets is empty, expanding the summary array and returning the first empty bucket in the expanded array. The data insertion module is used to perform data insertion operations based on the bucket index returned by the address and the key value of the data, and to record the address index; The data query module is used to perform data queries based on the data key to be queried and the addressing index, and return the query results.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images