A frequency histogram merging method, apparatus and data processing system

Abstract

A frequency histogram merging method is provided. A sliding window slides left and right on a frequency histogram, and height values of columns in the sliding window are obtained. According to the height values of the columns in the sliding window, a distribution uniformity rate of the heights of the columns in the sliding window is determined. In a case where the distribution uniformity rate of the heights of the columns in the sliding window is not less than a set threshold, the heights of the columns in the sliding window are basically the same, and the columns in the sliding window are merged together, so that columns with a relatively large height difference are avoided from being merged together, and loss of statistical information is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a frequency histogram merging method and device and a data processing system. BACKGROUND

[0002] A database is a data processing system developed to meet the needs of data processing. When querying data in the database using a structured query language (SQL) statement, the database needs to generate a query plan according to the SQL statement. The query plan is generated depending on the size of the selection rate of the filter predicate. The size of the selection rate is estimated by the statistical information of the database. The statistical information of the database is the parameters such as distinct value, most common value (MCV), and frequency histogram collected by the database at idle or scheduled time. Among them, the frequency histogram is mainly used in the selection rate estimation of range queries.

[0003] Generally, the calculation of the selection rate does not need to be accurate, and the time consumed by the calculation is as short as possible. Therefore, the database does not accurately record the frequency histogram of the data distribution on a column, but uses some algorithms to obtain a relatively rough frequency histogram. In order to simplify the frequency histogram, several adjacent columns in the frequency histogram can be merged together to obtain a rough frequency histogram. However, during the process of merging the columns of the frequency histogram, several columns with a large difference in height are merged together, which will cause a large amount of loss of the statistical information of the database. SUMMARY

[0004] In order to solve the above problems, the embodiments of the present application provide a frequency histogram merging method, device and data processing system. During the process of merging the columns of the frequency histogram, the data processing system slides a sliding window on the frequency histogram. The data processing system determines whether the columns in the sliding window are merged together by using the concept of cross entropy. When the height difference of the columns in the sliding window is relatively large, the data processing system calculates that the distribution uniformity rate in the sliding window is relatively low. The data processing system removes one column with a large height difference from the adjacent two columns in the sliding window, and then merges the columns in the sliding window together. The data processing system merges the columns with a small height difference in the sliding window together, thereby reducing the loss of statistical information of the simplified frequency histogram.

[0005] To this end, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, the embodiments of the present application provide a frequency histogram merging method, comprising: making a sliding window slide left and right on a frequency histogram, the frequency histogram comprising a plurality of columns; obtaining height values of each column within the sliding window, determining a distribution uniformity rate of the height of each column within the sliding window according to the height values of each column within the sliding window; and merging each column within the sliding window into one bucket in a case where the distribution uniformity rate of the height of each column within the sliding window is not less than a set threshold.

[0007] In this embodiment, the sliding window is used to slide on the frequency histogram, and each column within the sliding window is obtained. According to the height of each column within the sliding window, the distribution uniformity rate of the height of each column within the sliding window is calculated. When the distribution uniformity rate of the height of each column within the sliding window is not less than the set threshold, the heights of each column within the sliding window are basically the same. Then, the columns within the sliding window are merged together, which can avoid the columns with large height difference from being merged together and reduce the loss of statistical information.

[0008] In an embodiment, the determining of the distribution uniformity rate of the height of each column within the sliding window according to the height values of each column within the sliding window comprises: determining a cross-entropy of the uniform distribution of each column within the sliding window according to the total height values of each column within the sliding window; and determining the distribution uniformity rate of the height of each column within the sliding window according to the cross-entropy of the uniform distribution of each column within the sliding window.

[0009] In an embodiment, the cross-entropy of the uniform distribution of each column within the sliding window is calculated in the following manner:

[0010]

[0011] wherein i represents the order of each column in the frequency histogram, N represents the total number of columns of the frequency histogram, ni represents the height of the i th column, S N represents the total height of each column within the sliding window.

[0012] In an embodiment, the distribution uniformity rate of the height of each column within the sliding window is calculated in the following manner:

[0013]

[0014] wherein N represents the total number of columns of the frequency histogram, C N represents the cross-entropy of each column within the sliding window.

[0015] In an embodiment, in the case that the uniformity of the distribution of the heights of the columns in the sliding window is not less than a set threshold, the columns in the sliding window are merged into one bucket, comprising: in the case that the uniformity of the distribution of the heights of the columns in the sliding window is not less than a set threshold, a column is added in the sliding window; the height values of the columns in the sliding window in which the column is added are obtained, and the uniformity of the distribution of the heights of the columns in the sliding window in which the column is added is determined; in the case that the uniformity of the distribution of the heights of the columns in the sliding window in which the column is added is less than a set threshold, the columns in the sliding window before the column is added are merged into one bucket.

[0016] In an embodiment, the sliding window further comprises a left end side and a right end side, the left end side being a side of the sliding window close to a coordinate origin of the frequency histogram, and the right end side being a side of the sliding window away from the coordinate origin of the frequency histogram; the method further comprises: in the case that the uniformity of the distribution of the heights of the columns in the sliding window is less than a set threshold, the heights of the columns in the sliding window except for a column close to the right end side are merged to obtain one bucket.

[0017] In an embodiment, the method further comprises: moving the left end side of the sliding window to the column close to the right end side in the sliding window.

[0018] In an embodiment, before the step of determining the uniformity of the distribution of the heights of the columns in the sliding window according to the height values of the columns in the sliding window, the method further comprises: in the case that the number of the columns in the sliding window is 1, determining that the uniformity of the distribution of the heights of the columns in the sliding window is zero.

[0019] In an embodiment, the method further comprises: in the case that the number of the columns of the frequency histogram is less than a set threshold, merging the columns of the frequency histogram into one bucket.

[0020] In a second aspect, an embodiment of the present application provides a frequency histogram merging apparatus, comprising: an execution unit, a transceiver unit, and a processing unit, the processing unit being in communication connection with the transceiver unit and the execution unit, comprising: the execution unit being configured to make a sliding window slide left and right on a frequency histogram, the frequency histogram comprising a plurality of columns; the transceiver unit being configured to obtain height values of the columns in the sliding window; and the processing unit being configured to determine a uniformity of the distribution of the heights of the columns in the sliding window according to the height values of the columns in the sliding window, and in the case that the uniformity of the distribution of the heights of the columns in the sliding window is not less than a set threshold, merge the columns in the sliding window into one bucket.

[0021] In an embodiment, the processing unit is specifically configured to determine a cross-entropy of uniform distribution of each column in the sliding window according to the total height value of each column in the sliding window; and determine a distribution uniformity rate of the height of each column in the sliding window according to the cross-entropy of uniform distribution of each column in the sliding window.

[0022] In an embodiment, the processing unit is specifically configured to, in a case where the distribution uniformity rate of the height of each column in the sliding window is not less than a set threshold, increase one column in the sliding window; the transceiver unit is specifically configured to obtain the height value of the column in the sliding window with the increased column; and the processing unit is further configured to determine the distribution uniformity rate of the height of each column in the sliding window with the increased column, and in a case where the distribution uniformity rate of the height of each column in the sliding window with the increased column is less than the set threshold, combine each column in the sliding window before the increased column into one bucket.

[0023] In an embodiment, the sliding window further includes a left end side and a right end side, the left end side being a side of the sliding window close to the origin of the frequency histogram coordinate, and the right end side being a side of the sliding window away from the origin of the frequency histogram coordinate; and the processing unit is further configured to, in a case where the distribution uniformity rate of the height of each column in the sliding window is less than a set threshold, combine the height of the columns other than one column close to the right end side in the sliding window to obtain one bucket.

[0024] In an embodiment, the execution unit is further configured to move the left end side of the sliding window to the one column close to the right end side in the sliding window.

[0025] In an embodiment, the processing unit is further configured to, in a case where the number of columns in the sliding window is 1, determine the distribution uniformity rate of the height of the column in the sliding window as zero.

[0026] In an embodiment, the processing unit is further configured to, in a case where the number of columns of the frequency histogram is less than a set threshold, combine each column of the frequency histogram into one bucket.

[0027] In a third aspect, an embodiment of the present application provides a data processing system, including: at least one processor, the processor being configured to execute instructions stored in a memory to enable an electronic device to perform the method in each possible implementation of the first aspect.

[0028] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, having stored thereon a computer program, which, when executed in a computer, causes the computer to perform the method according to any possible implementation of the method of the first aspect.

[0029] In a fifth aspect, an embodiment of the present application provides a computer program product, characterized in that the computer program product stores instructions, which, when executed in a computer, cause the computer to perform the method according to any possible implementation of the method of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0030] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application.

[0031] Figure 1 A flowchart of a process for a data processing system to estimate a frequency histogram of data of a database;

[0032] Figure 2 A structural diagram of an electronic device provided in an embodiment of the present application;

[0033] Figure 3 A flowchart of a frequency histogram merging method provided in an embodiment of the present application;

[0034] Figure 4 A height diagram of each element on a frequency histogram;

[0035] Figure 5 A diagram of a frequency histogram merging device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0037] The term "and / or" in the present document is used to describe an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The symbol " / " in the present document represents an or relationship of associated objects, for example, A / B represents A or B.

[0038] The terms "first" and "second" and the like in the specification and claims of the present document are used to distinguish different objects, and are not used to describe a specific order of the objects. For example, the first response message and the second response message are used to distinguish different response messages, and are not used to describe a specific order of the response messages.

[0039] In the embodiments of the present application, the words "in an embodiment", "for example", and the like are used to indicate an example, an illustration, or an illustration. Any embodiment or design scheme described as "in an embodiment" or "for example" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of the words "in an embodiment" or "for example" and the like is intended to present the relevant concept in a specific manner.

[0040] In the description of the embodiments of the present application, unless otherwise specified, "a plurality of" means two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0041] A database refers to a collection of a large amount of data stored in a computer for a long time, organized, shareable, and uniformly managed. The storage space of a database is large, and can store millions of data, tens of millions of data, or hundreds of millions of data. Instead of storing data randomly, a database stores data according to certain rules to avoid reducing the efficiency of querying data. Therefore, a database is also a computer software system for storing and managing data according to a data structure.

[0042] Statistical information refers to information that uses statistical methods to process statistical data or information that has an impact on human activities. The statistical information of a database refers to information recording the number of rows, size, distribution characteristics, and the like of a database object (table, index, and the like). When a data processing system queries data, statistical information can be used to query data in a database to improve the efficiency of the data processing system querying data.

[0043] A frequency histogram, also known as a frequency distribution frequency histogram, represents a graph of a frequency distribution. In a frequency histogram, the horizontal axis of the frequency histogram represents each element, and the vertical axis of the frequency histogram represents the frequency of each element. The height of the column of each element of the frequency histogram is the frequency of each element.

[0044] A bucket refers to a column in which several adjacent columns of a frequency histogram are combined together.

[0045] A simplified frequency histogram refers to combining several adjacent columns of a frequency histogram according to a set combination rule. The frequency histogram including several buckets after combination is referred to as a simplified frequency histogram.

[0046] Figure 1 The flowchart for estimating a frequency histogram for data processing system data of a database. As shown in FIG. 1, the data processing system first queries the database to obtain statistical information of the database, and then estimates a frequency histogram of the database according to the statistical information of the database. Figure 1As shown, the database stores a large amount of data. The data in the database exists in the form of "pages." When the data processing system estimates the frequency histogram, it randomly selects a portion of pages from the database as target pages. After obtaining the target pages, the data processing system extracts each element from the target pages and calculates the frequency of each element. Using each element as the parameter of the horizontal axis of the frequency histogram and the frequency of each element as the parameter of the vertical axis, the data processing system can calculate the frequency histogram of the database. When simplifying the frequency histogram, the data processing system can merge the bars of the frequency histogram according to a set merging rule. The merged bars are grouped together as a bucket, resulting in a simplified frequency histogram.

[0047] In existing technologies, data processing systems simplify frequency histograms by traversing the bars from left to right. Whenever the total height of a bar reaches a predetermined number, the system merges the current bars into a bucket, resulting in multiple buckets of equal height. However, existing systems only consider the total height of the bars within a bucket when merging buckets, neglecting the individual heights of the bars within that bucket. If the height differences between the bars within a bucket of the simplified frequency histogram are significant, the simplified frequency histogram will lose a substantial amount of statistical information.

[0048] To address the shortcomings of existing data processing systems that lose statistical information when simplifying frequency histograms, embodiments of this application provide a data processing system, a frequency histogram merging method, and a frequency histogram merging apparatus.

[0049] Figure 2 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application. Figure 2 As shown, the data processing system 200 includes a memory 201, a transceiver 202, and a processor 203. The memory 201 and the processor 203 can establish a communication connection through the transceiver 202. The data processing system 200 can be a server, computer, laptop, solid-state drive, smartphone, tablet, or other electronic device.

[0050] The memory 201 can include a volatile memory, such as a random-access memory (RAM) or the like. The memory 201 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), a solid-state drive (SSD), or the like. The memory 201 can be a cloud server, and can include a combination of the above-described various memories. In embodiments of the present application, the memory 201 can serve as a database for storing a large amount of data. The memory 201 is used to store an application program for execution by the processor 203, so that the processor 203 can execute the frequency histogram merging method. The memory 201 can serve as a cache data for storing frequency histograms, buckets, statistical information, and the like. The memory 201 can have other functions, which are not limited herein.

[0051] The transceiver 202 is used to establish a communication connection between the memory 201 and the processor 203, so as to realize data transmission between the memory 201 and the processor 203. The transceiver 202 can be a wired interface, such as a gold wire, a pin, a printed circuit board (PCB), or other types of wired interfaces. The transceiver 202 can be a wireless interface, such as a Bluetooth module, a near field communication (NFC) module, or other wireless communication modules.

[0052] The processor 203 can be a central processing unit (CPU), a cloud server, or the like, and is used to receive data stored in the memory 201 through the transceiver 202, and execute the frequency histogram merging method.

[0053] Figure 3 A flowchart of a frequency histogram merging method provided in embodiments of the present application is shown in FIG. 3. As shown in FIG. 3, the method is executed by the processor 203, and the specific implementation process is as follows: Figure 3

[0054] In step S301, the processor 203 controls a sliding window to slide on a frequency histogram, and determines a column inside an initial sliding window.

[0055] In embodiments of the present application, the horizontal axis of the frequency histogram represents each element, and the vertical axis of the frequency histogram represents the frequency of each element. The height of the column of each element of the frequency histogram is the frequency of each element, and the height of the i-th column is denoted as n i . ​

[0056] The sliding window is a window with variable size, which slides forward in the same direction at both ends. The sliding window has two parameters, slide and size. The slide is the step length of the sliding window each time, and the size is the width of the sliding window. In the embodiment of the present application, the sliding window slides one step length each time, and the step length is the distance between the center positions of two adjacent columns. The width of the sliding window is an integer multiple of the step length.

[0057] The sliding direction of the sliding window on the frequency histogram is the left-to-right direction. The left-to-right direction refers to the direction in which the positive half of the horizontal axis of the frequency histogram points. The left end of the sliding window is defined as low, and the right end is defined as high. In other embodiments, the sliding direction of the sliding window on the frequency histogram is the right-to-left direction, which is not limited in the present application.

[0058] In the embodiment of the present application, when the sliding window starts to slide on the frequency histogram, the left end low of the sliding window is at the left side of the first column. The right end high of the sliding window is between the first column and the second column. The width of the sliding window is one step length.

[0059] In step S302, the processor 203 judges whether the column number close to the right end side in the sliding window is less than the total number of columns of the frequency histogram. If the right end high of the sliding window is not less than the total number of columns of the frequency histogram, step S309 is performed. If the right end high of the sliding window is less than the total number of columns of the frequency histogram, step S303 is performed.

[0060] The processor 203 acquires the columns in the sliding window and determines the column numbers in the sliding window. The processor 203 judges whether the column number close to the right end side in the sliding window is less than the total number N of columns of the frequency histogram. In one embodiment, when the column number close to the right end side in the sliding window is not less than the total number N of columns of the frequency histogram, all the columns of the frequency histogram are in the sliding window. At this time, it can be understood that the number of columns of the frequency histogram is relatively small, and the processor 203 can combine all the columns of the frequency histogram together. In one embodiment, when the column number close to the right end side in the sliding window is less than the total number N of columns of the frequency histogram, part of the columns of the frequency histogram are in the sliding window. At this time, the processor 203 can detect the uniformity rate of the distribution of the columns in the sliding window, and then combine part of the columns of the frequency histogram together.

[0061] In step S303, the processor 203 judges whether the width of the sliding window is greater than one step length. If the width of the sliding window is greater than one step length, step S304 is performed. If the width of the sliding window is not greater than one step length, step S306 is performed.

[0062] The processor 203 obtains the width of the sliding window and determines the number of pillars within the sliding window based on the ratio of the sliding window width to the step size. The processor 203 determines whether the number of pillars within the sliding window is greater than 1. In one embodiment, when the number of pillars within the sliding window is greater than 1, the processor 203 can calculate the distribution uniformity SC of the pillars within the sliding window based on the height of each pillar. In one embodiment, when the number of pillars within the sliding window is not greater than 1, there is only one pillar within the sliding window, and there is no distribution uniformity SC; therefore, SC = 0.

[0063] In step S304, processor 203 calculates the uniformity of the height of each column within the sliding window.

[0064] Processor 203 retrieves the bars within the sliding window and calculates the merging index S of the bars based on their heights. N Specifically:

[0065]

[0066] Where i represents the order of each bar in the frequency histogram, and N represents the total number of bars in the frequency histogram.

[0067] Processor 203 uses the mathematical concept of cross-entropy to measure the degree of deviation between the frequency histogram and a uniform distribution. Processor 203 utilizes the combined index S of the individual bars. N Calculate the cross-entropy C between the distribution represented by each bar within the sliding window and the uniform distribution. N Specifically:

[0068]

[0069] Among them, C N The minimum value is log(N).

[0070] Processor 203 calculates the cross-entropy C based on the uniform distribution of each bar within the sliding window. N Calculate the uniformity of the height distribution SC of each column within the sliding window. N Specifically:

[0071]

[0072] Among them, the uniform distribution rate SC N Between 0 and 1. Uniform distribution rate SC N The larger the value, the closer the distribution of the N pillars is to a uniform distribution, and the more likely it is that the N pillars will be combined. Uniformity distribution rate SC N The smaller the value, the less uniform the distribution of the N pillars is, and the less likely it is to merge the N pillars.

[0073] The processor 203 can calculate the cross entropy C of the uniform distribution of the heights of the columns in the sliding window according to the following formula: N The cross entropy C can be calculated according to the following formula: N+1 Specifically, the cross entropy C is calculated according to the following formula:

[0074]

[0075] Let The cross entropy C can be simplified according to the following formula: N+1 The cross entropy C can be simplified according to the following formula:

[0076] C N+1 = αC N + β

[0077] In step S305, the processor 203 determines whether the uniform distribution rate of the heights of the columns in the sliding window is less than the set threshold t. If the uniform distribution rate of the heights of the columns in the sliding window is not less than the set threshold t, step S306 is performed. If the uniform distribution rate of the heights of the columns in the sliding window is less than the set threshold t, step S307 is performed.

[0078] After the processor 203 calculates the uniform distribution rate SC of the heights of the columns in the sliding window, the processor 203 determines whether the uniform distribution rate SC of the heights of the columns in the sliding window is less than the set threshold t. In an embodiment, if the uniform distribution rate SC of the columns in the sliding window is less than the set threshold t, it indicates that the columns in the sliding window are not uniformly distributed, and the width of the sliding window needs to be reduced to improve the uniform distribution rate of the heights of the columns in the sliding window. In an embodiment, if the uniform distribution rate SC of the columns in the sliding window is not less than the set threshold t, it indicates that the heights of the columns in the sliding window are uniformly distributed.

[0079] In step S306, the processor 203 moves the right end side high of the sliding window to increase the width of the sliding window.

[0080] When the number of columns in the sliding window is 1, the uniform distribution rate of the columns in the sliding window cannot be calculated. The processor 203 can increase the width of the sliding window to add a column in the sliding window. In an embodiment, the processor 203 can move the right end side high of the sliding window to the right by one step to add a column.

[0081] The processor 203 calculates the uniform distribution rate SC of the columns in the sliding window after adding a column in the sliding window. When the processor 203 calculates that the uniform distribution rate SC of the columns in the sliding window after adding a column is less than the threshold t, the columns in the sliding window are not uniformly distributed, and the width of the sliding window needs to be reduced. The processor 203 takes one column in the sliding window as a separate bucket. When the processor 203 calculates that the uniform distribution rate SC of the columns in the sliding window after adding a column is not less than the threshold t, the columns in the sliding window are uniformly distributed. The processor 203 can add another column in the sliding window.

[0082] When the processor 203 determines that the columns in the sliding window are uniformly distributed, the width of the sliding window can be increased, and a column can be added in the sliding window. In an embodiment, the processor 203 can move the right end side high of the sliding window to the right by one step, and add a column.

[0083] The processor 203 calculates the uniform distribution rate SC of the columns in the sliding window after adding a column in the sliding window. The processor 203 adds a column again after the columns in the existing sliding window are uniformly distributed. It can be understood that the processor 203 recalculates the uniform distribution rate SC of the heights of the columns in the sliding window after the sliding window adds a column each time, and the serial number of the column close to the right end side in the sliding window is less than the total number N of columns of the frequency histogram. If the calculated uniform distribution rate SC is still not less than the threshold t, the processor 203 can move the right end side high of the sliding window to the right by one step, and add a column. In this way, the sliding window can accommodate more columns, reduce the number of buckets after the frequency histogram is merged, and improve the concentration of the frequency histogram. In step S307, the processor 203 merges the heights of the columns in the sliding window except for the column close to the right end side high, and obtains a bucket.

[0084] When the number of columns in the initial sliding window is 1, the processor 203 cannot calculate the uniform distribution rate SC of the columns in the sliding window. When the number of columns in the sliding window is 2, the sliding window is based on the initial sliding window adding a column. When the processor 203 calculates that the uniform distribution rate SC of the two columns in the sliding window is less than the threshold t, the newly added column of the sliding window is removed, and the column in the initial sliding window is taken as a bucket.

[0085] Similarly, when the number of bars in the sliding window is greater than 2, the sliding window adds one bar based on the previous sliding window. When the processor 203 calculates that the uniform distribution rate SC of each bar in the current sliding window is less than a set threshold t, it removes the newly added bar in the sliding window and merges the heights of all bars in the current sliding window except for one bar near the right end (high) to obtain a bucket.

[0086] In step S308, processor 203 moves the left side of the sliding window to a pillar inside the sliding window near the right side.

[0087] Processor 203 merges the heights of all bars within the current sliding window, except for the one bar closest to the right edge (high), into a single bucket. Then, it moves the left edge of the sliding window to the bar closest to the right edge within the sliding window. At this point, the entire sliding window has moved to the right side of the bar whose frequency histogram has been merged into the previous sliding window's right edge. Processor 203 then slides the sliding window onto other bars to create the next bucket.

[0088] In step S309, processor 203 merges the height of the columns in the sliding window.

[0089] The right side of the sliding window, high, moves to the right of the total number of bars N in the frequency histogram, and the sliding window has merged the last bar of the frequency histogram. After determining that the uniform distribution rate SC of each bar in the sliding window is not less than a set threshold t, the processor 203 merges the bars of the sliding window together to obtain a bucket.

[0090] In this embodiment, the processor 203 uses a sliding window to slide across the frequency histogram during the merging process. The processor 203 uses the concept of cross-entropy to determine whether the bars within the sliding window should be merged. When the height differences between the bars within the sliding window are significant, the uniformity of distribution within the sliding window is calculated to be low. The processor 203 removes the bar with the larger height difference between two adjacent bars within the sliding window and then merges the bars within the sliding window together. The processor 203 merges bars with smaller height differences within the sliding window together, reducing the loss of statistical information in the simplified frequency histogram.

[0091] by Figure 4 Taking the frequency histogram shown as an example, the frequency histogram has 9 elements, and the heights of each element from left to right are 5, 5, 6, 4, 9, 1, 5, 5 and 5 respectively.

[0092] For existing merging methods based on the number of items within each bucket, the number of items in each bucket is set to 10. The frequency histogram generated by the existing technical solution is as follows:

[0093] [5, 5], [6, 4], [9, 1], [5, 5], [5]

[0094] For the frequency histogram merging method of the present application, the merging threshold t = 0.98 is set. The frequency histogram merged by the frequency histogram merging scheme of the present application is:

[0095] [5, 5, 6, 4], [9], [1], [5, 5, 5]

[0096] Comparing the two schemes, the heights of the 7th column, the 8th column and the 9th column are the same. The prior art scheme merges the 7th column and the 8th column together and separately stores the 9th column. In the frequency histogram merging scheme of the present application, the heights of the 7th column, the 8th column and the 9th column are identified, the 7th column, the 8th column and the 9th column are merged together and then stored, so that the storage efficiency of the statistical information is higher.

[0097] The height difference between the 5th column and the 6th column is relatively large and the change degree is violent, so the 5th column and the 6th column should not be merged together. The prior art scheme merges the 5th column and the 6th column together, which causes relatively serious loss of statistical information. In the frequency histogram merging scheme of the present application, the heights of the 5th column and the 6th column are identified, and the 5th column and the 6th column are stored separately, thereby reducing the loss of statistical information.

[0098] In the embodiment of the present application, the greater the value of the threshold t is, the more stringent the merging requirement for each column of the frequency histogram is, and the more the number of the merged buckets is. The smaller the value of the threshold t is, the looser the merging requirement for each column of the frequency histogram is, and the fewer the number of the merged buckets is.

[0099] In the embodiment of the present application, when the processor 203 obtains the set number of the merged buckets without setting the threshold t, the threshold t can be obtained by using the dichotomy and dynamic programming method. In one embodiment, let C[m, n] represent the cross entropy C calculated by the mth column to the nth column with the uniform distribution, and then the recursive formula is:

[0100] C[m, n+1] = αC[m, n] + β

[0101] Processor 203 calculates all possible N×N arrays of C[m,n] according to a recursive formula. N represents the number of bars in the frequency histogram. Processor 203 performs a binary search on a set threshold t. t∈[0,1]. When t=1, no merging occurs in the frequency histogram. At this time, the number of buckets in the frequency histogram is the maximum, and no information loss occurs. If the number of buckets in the frequency histogram is greater than the set number, the set threshold t is reduced. Processor 203 uses the binary search method to set t=0.5 to reduce the number of buckets in the frequency histogram, and so on, until the optimal set threshold t is found.

[0102] It should be noted that when processor 203 calculates the number of merged buckets based on the set threshold t, it can use a pre-stored N×N array, a process with a complexity of O(N). However, the complexity of processor 203 building the N×N array is O(N^2), and the complexity of finding the set threshold t using binary search is logarithmic. Therefore, the total complexity is O(N^2).

[0103] Figure 5 This is a schematic diagram of a frequency histogram merging device provided in an embodiment of this application. Figure 5 As shown, the frequency histogram merging device 500 includes an execution unit 501, a transceiver unit 502, and a processing unit 503. The processing unit 503 is communicatively connected to the transceiver unit 502 and the execution unit 501, and its specific operation is as follows:

[0104] The execution unit 501 is used to make the sliding window slide left and right on the frequency histogram, which includes multiple bars; the transceiver unit 502 is used to obtain the height value of each bar in the sliding window; the processing unit 503 is used to determine the uniformity of the height distribution of each bar in the sliding window based on the height value of each bar in the sliding window; and if the uniformity of the height distribution of each bar in the sliding window is not less than a set threshold, the bars in the sliding window are merged into a bucket.

[0105] In one embodiment, the processing unit 503 is specifically used to determine the cross-entropy of the uniform distribution of each column in the sliding window based on the total height value of each column in the sliding window; and to determine the uniformity of the height distribution of each column in the sliding window based on the cross-entropy of the uniform distribution of each column in the sliding window.

[0106] In an embodiment, the processing unit 503 is specifically configured to increase a column in the sliding window if a uniformity of distribution of the heights of the columns in the sliding window is not less than a set threshold value; the execution unit 501 is configured to obtain the height values of the columns in the sliding window in which the column is increased; and the processing unit 503 is further configured to determine the uniformity of distribution of the heights of the columns in the sliding window in which the column is increased, and combine the columns in the sliding window before the column is increased into one bucket if the uniformity of distribution of the heights of the columns in the sliding window in which the column is increased is less than the set threshold value.

[0107] In an embodiment, the sliding window further comprises a left end side and a right end side, the left end side being a side of the sliding window close to the coordinate origin of the frequency histogram, and the right end side being a side of the sliding window away from the coordinate origin of the frequency histogram; and the processing unit 503 is further configured to combine the heights of the columns in the sliding window except for one column close to the right end side to obtain one bucket if the uniformity of distribution of the heights of the columns in the sliding window is less than the set threshold value.

[0108] In an embodiment, the execution unit 501 is further configured to move the left end side of the sliding window to the one column close to the right end side in the sliding window.

[0109] In an embodiment, the processing unit 503 is further configured to determine that the uniformity of distribution of the heights of the columns in the sliding window is zero if the number of the columns in the sliding window is 1.

[0110] In an embodiment, the processing unit 503 is further configured to combine the columns of the frequency histogram into one bucket if the number of the columns of the frequency histogram is less than a set threshold value.

[0111] The embodiments of the present application further provide a computer readable storage medium, which has stored thereon a computer program, and the computer program causes a computer to execute any one of the methods described in the embodiments of the present application and the corresponding description. Figures 3-5

[0112] The embodiments of the present application further provide a computer program product, which has stored thereon instructions, and the instructions cause a computer to execute any one of the methods described in the embodiments of the present application and the corresponding description. Figures 3-5

[0113] ​​Those skilled in the art can appreciate that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present application.

[0114] In addition, various aspects or features of embodiments of the present application can be realized as methods, apparatuses, or articles of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" as used in this application is intended to encompass a computer program accessible from any computer-readable device, carrier, or media. For example, computer-readable media can include but are not limited to magnetic storage devices (e.g., hard disk, floppy disk, magnetic strips, etc.), optical disks (e.g., CD-ROM, DVD, etc.), smart cards, and flash memory devices (e.g., EPROM, card, stick, or key drive, etc.). In addition, the various storage media described herein can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" can include but is not limited to wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0115] In the above embodiments, Figure 5 The frequency histogram merging device 500 in the above embodiments can be realized in whole or in part by software, hardware, firmware, or any combination thereof. When realized by software, it can be realized in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (e.g., floppy disk, hard disk, magnetic tape), optical media (e.g., DVD), or semiconductor media (e.g., SSD), etc.

[0116] It should be understood that the magnitude of the serial number of the processes described above does not mean the order of execution, the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0118] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0119] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0120] The functions, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application essentially or say the part that contributes to the prior art or part of the technical solutions can be embodied in the form of software product, and the computer software product stored in a storage medium includes a plurality of instructions for making a computer device (which can be a personal computer, a server, or an access network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk and various program code storage media.

[0121] The above is only a specific implementation of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the embodiments of the present application, which should be covered in the protection scope of the embodiments of the present application.

Claims

1. A frequency histogram merging method, wherein the method is executed by a processor, characterized in that, include: During the database statistical information processing, a frequency histogram collected from the database is obtained. The frequency histogram represents the frequency state of each element in the database. The frequency histogram includes multiple bars, and each bar represents the frequency of an element. This causes the sliding window to slide on the frequency histogram; Obtain the height value of each column within the sliding window; Based on the height values ​​of each column within the sliding window, determine the uniformity of the height distribution of each column within the sliding window; If the uniformity of the height distribution of each column within the sliding window is not less than a set threshold, then add a column within the sliding window. Obtain the height value of the column within the sliding window for adding columns, and determine the uniformity of the height distribution of each column within the sliding window for adding columns; If the uniformity of the height distribution of each column within the sliding window where the column is added is less than a set threshold, the columns within the sliding window before the column was added are merged into a bucket. If the uniformity of the height distribution of each column within the sliding window where the column is added is not less than a set threshold, then another column is added within the sliding window, and the uniformity of the height distribution of each column within the sliding window where the column is added is determined.

2. The method according to claim 1, characterized in that, Determining the uniformity of the height distribution of each column within the sliding window based on the height values ​​of each column within the sliding window includes: Based on the total height of each column within the sliding window, determine the cross-entropy of the uniform distribution of each column within the sliding window; The uniformity of the height distribution of each column within the sliding window is determined based on the cross-entropy of the uniform distribution of each column within the sliding window.

3. The method according to claim 2, characterized in that, The cross-entropy of the uniformly distributed columns within the sliding window is calculated as follows: Where i represents the order of each bar in the frequency histogram, N represents the total number of bars in the frequency histogram, and n i S represents the height of the i-th pillar. N This represents the total height of all the pillars within the sliding window.

4. The method according to claim 3, characterized in that, The uniformity of the height distribution of each column within the sliding window is calculated as follows: Where N represents the total number of bars in the frequency histogram, C N This represents the cross-entropy of each column within the sliding window.

5. The method according to any one of claims 1-4, characterized in that, The sliding window further includes a left end and a right end, wherein the left end is the side of the sliding window closer to the origin of the frequency histogram coordinates, and the right end is the side of the sliding window farther from the origin of the frequency histogram coordinates; the method further includes: If the uniformity of the height distribution of each column within the sliding window is less than a set threshold, the heights of all columns within the sliding window except for one column near the right end are combined to obtain a bucket.

6. The method according to claim 5, characterized in that, Also includes: Move the left side of the sliding window to a pillar inside the sliding window near the right side.

7. The method according to any one of claims 1-4, characterized in that, Before determining the uniformity of the height distribution of each column within the sliding window based on the height values ​​of each column within the sliding window, the method further includes: When the number of columns within the sliding window is 1, the uniformity of the height distribution of the columns within the sliding window is determined to be zero.

8. The method according to any one of claims 1-4, characterized in that, Also includes: When the number of bars in the frequency histogram is less than a set threshold, the bars of the frequency histogram are merged into a bucket.

9. A frequency histogram merging device, comprising: An execution unit, a transceiver unit, and a processing unit, wherein the processing unit is communicatively connected to the transceiver unit and the execution unit, characterized in that it includes: The transceiver unit is used to obtain the frequency histogram collected by the database during the database statistical information processing process. The frequency histogram represents the frequency state of each element in the database. The frequency histogram includes multiple bars, and each bar represents the frequency of an element. The execution unit is used to make the sliding window slide on the frequency histogram to obtain the height value of each bar in the sliding window; The processing unit is used to determine the uniformity of the height distribution of each column in the sliding window based on the height value of each column in the sliding window. If the uniformity of the height distribution of each column within the sliding window is not less than a set threshold, then add a column within the sliding window. Obtain the height value of the column within the sliding window for adding columns, and determine the uniformity of the height distribution of each column within the sliding window for adding columns; If the uniformity of the height distribution of each column within the sliding window where the column is added is less than a set threshold, the columns within the sliding window before the column was added are merged into a bucket. If the uniformity of the height distribution of each column within the sliding window where the column is added is not less than a set threshold, then another column is added within the sliding window, and the uniformity of the height distribution of each column within the sliding window where the column is added is determined.

10. The apparatus according to claim 9, characterized in that, The processing unit is specifically used for: Based on the total height of each column within the sliding window, determine the cross-entropy of the uniform distribution of each column within the sliding window; The uniformity of the height distribution of each column within the sliding window is determined based on the cross-entropy of the uniform distribution of each column within the sliding window.

11. The apparatus according to any one of claims 9-10, characterized in that, The sliding window further includes a left end and a right end, the left end being the side of the sliding window closer to the origin of the frequency histogram coordinates, and the right end being the side of the sliding window farther from the origin of the frequency histogram coordinates; the processing unit is further configured to: If the uniformity of the height distribution of each column within the sliding window is less than a set threshold, the heights of all columns within the sliding window except for one column near the right end are combined to obtain a bucket.

12. The apparatus according to claim 11, characterized in that, The execution unit is also used for Move the left side of the sliding window to a pillar inside the sliding window near the right side.

13. The apparatus according to any one of claims 9-10, characterized in that, The processing unit is also used for When the number of columns within the sliding window is 1, the uniformity of the height distribution of the columns within the sliding window is determined to be zero.

14. The apparatus according to any one of claims 9-10, characterized in that, The processing unit is also used for When the number of bars in the frequency histogram is less than a set threshold, the bars of the frequency histogram are merged into a bucket.

15. A data processing system, characterized in that, include: At least one processor, the processor being configured to execute instructions stored in a memory to cause an electronic device to perform the method as described in any one of claims 1-8.

Images