Histogram generation method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a histogram generation method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a local histogram corresponding to each partial node; acquiring local significant change points in each local histogram, and determining global significant change points according to the local significant change points; dividing global data buckets according to the global significant change points; according to the global data bucket, determining a target data bucket from local data buckets corresponding to the local histogram; acquiring a local frequency corresponding to the target data bucket, and performing weighted calculation on the local frequency to generate a target frequency; and generating a global histogram according to the global data bucket and the target frequency. By adopting the method, the accuracy of a data processing result can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a histogram generation method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Technology

[0002] In distributed database systems, obtaining accurate and efficient global statistics is crucial for optimizing query performance and improving the accuracy of data analysis.

[0003] Related technologies utilize traditional histograms to statistically analyze global data in distributed databases. However, for distributed databases in general, it is necessary to merge the traditional histograms of each node to achieve statistical analysis of the global data distribution.

[0004] However, since the data in a distributed database is distributed across multiple nodes, the traditional histograms of each node are merged to generate statistics for the distributed database data, which can easily lead to distorted results. Summary of the Invention

[0005] Therefore, it is necessary to provide a histogram generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of data processing results in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a histogram generation method, including:

[0007] Obtain the local histogram corresponding to each sub-node;

[0008] Obtain the local significant change points in each local histogram, and determine the global significant change points based on the local significant change points;

[0009] Divide the global data into buckets based on points of significant global change;

[0010] Based on the global data bucket, determine the target data bucket from the local data bucket corresponding to the local histogram;

[0011] Obtain the local frequencies corresponding to the target data bucket, and perform weighted calculations on the local frequencies to generate the target frequency;

[0012] Generate a global histogram based on the global data buckets and the target frequency.

[0013] In one embodiment, before determining the global significant change point based on the local significant change points, the histogram generation method further includes: performing a sorting operation on the local significant change points to obtain a significant change sequence; determining the difference value between each local significant change point in the significant change sequence and its adjacent local significant change points; and determining the global significant change point based on the local significant change points if the difference value meets a preset condition.

[0014] In one embodiment, the difference value includes a distance difference value and a frequency difference value. The preset condition includes at least one of the distance difference value being less than or equal to a preset distance threshold and the frequency difference value being less than or equal to a preset frequency threshold. When the difference value meets the preset condition, the global significant change point is determined based on the local significant change point, including: when the distance difference value is less than or equal to the preset distance threshold and the frequency difference value is less than or equal to the preset frequency threshold, the mathematical statistical value of the local significant change point and the adjacent local significant change points is determined, and the global significant change point is determined based on the mathematical statistical value.

[0015] In one embodiment, after determining the difference between each local significant change point in the significant change sequence and its neighboring local significant change points, the histogram generation method further includes: when the distance difference value is greater than a preset distance threshold, or when the frequency difference value is greater than a preset frequency threshold, taking the local significant change point and its neighboring local significant change point as global significant change points.

[0016] In an optional embodiment, determining the target data bucket from the local data buckets corresponding to the local histogram based on the global data bucket includes: if there is an intersection between the global data bucket and the local data bucket, then the local data bucket that intersects with the global data bucket is the target data bucket.

[0017] In an exemplary embodiment, the local data bucket that intersects with the global data bucket is the target data bucket, which includes: when there are at least two local data buckets corresponding to the intersection, obtaining the intersection size of each local data bucket with the global data bucket; and selecting the local data bucket with the largest intersection with the global data bucket as the target data bucket based on the intersection size.

[0018] Secondly, this application also provides a histogram generation apparatus, comprising:

[0019] The histogram acquisition module is used to acquire the local histogram corresponding to each sub-node;

[0020] The change point determination module is used to obtain local significant change points in each local histogram and determine global significant change points based on the local significant change points;

[0021] The partitioning module is used to divide the global data into buckets based on globally significant change points.

[0022] The data bucket determination module is used to determine the target data bucket from the local data buckets corresponding to the local histogram based on the global data buckets;

[0023] The target frequency generation module is used to obtain the local frequencies corresponding to the target data bucket, perform weighted calculations on the local frequencies, and generate the target frequency.

[0024] The histogram generation module is used to generate a global histogram based on the global data buckets and the target frequency.

[0025] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method embodiments.

[0026] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method embodiments.

[0027] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method embodiments.

[0028] The above-mentioned histogram generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product, in which the global significant change points are determined based on the local significant change points in the local histogram, and the global data buckets are divided based on the generated global significant change points, and the target frequency corresponding to the global data buckets is further determined, can reduce the problem of data inaccuracy caused by directly merging local histograms. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating a histogram generation method in one embodiment;

[0031] Figure 2 This is a flowchart illustrating the histogram generation method in another embodiment;

[0032] Figure 3 This is a flowchart illustrating the histogram generation method in another embodiment;

[0033] Figure 4This is a structural block diagram of a histogram generation device in one embodiment;

[0034] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0036] In one embodiment, such as Figure 1 As shown, a histogram generation method is provided. This embodiment illustrates the method's application to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0037] Step 102: Obtain the local histogram corresponding to each sub-node.

[0038] In one embodiment, each sub-node can be a node of a distributed database.

[0039] Optionally, each sub-node includes a local histogram calculation module for generating local histograms.

[0040] Optionally, each sub-node includes a change point determination module, used to obtain local significant change points in each local histogram and determine global significant change points based on the local significant change points.

[0041] Optionally, each sub-node may also include a target frequency generation module, which is used to obtain the local frequency corresponding to the target data bucket, perform weighted calculation on the local frequency, and generate the target frequency.

[0042] In one embodiment, each sub-node can be deployed on its respective server, and the distributed database can include a server group consisting of servers corresponding to each sub-node.

[0043] Histograms are used to display the frequency distribution of continuous or discrete data. Optionally, a histogram can be a MaxDiff histogram.

[0044] The local histogram is used to display the frequency distribution of continuous or discrete data within each node. Optionally, the local histogram can be a local MaxDiff histogram.

[0045] The global histogram is used to display the frequency distribution of continuous or discrete data in a distributed database. Optionally, the local histogram can be a global MaxDiff histogram.

[0046] Optionally, the global histogram can be generated from the local histogram.

[0047] In one embodiment, the global histogram can be generated by a coordinating node.

[0048] In one alternative embodiment, the coordinating node can be one of the sub-nodes.

[0049] Optionally, if one of the sub-nodes acts as a coordinating node, the coordinating node establishes communication connections with the other sub-nodes so that the coordinating node can obtain local histograms from the other sub-nodes.

[0050] In one embodiment, each sub-node can pre-generate a local histogram based on its own local data.

[0051] In an optional embodiment, each sub-node communicates with the coordinating node, and the coordinating node receives local histograms sent by each sub-node and can generate a global histogram based on the received local histograms.

[0052] In one embodiment, the generation of a local histogram may include: after each node obtains its own local dataset, iterating through the dataset, determining the frequency of each data point in the dataset, determining local significant change points based on the frequency of each data point, determining local data buckets based on the local significant change points, and generating a local histogram based on the local data buckets and the frequency of data in the local data buckets.

[0053] Local data buckets are used to divide the local dataset into local data intervals.

[0054] Optionally, the local data bucket includes a left boundary and a right boundary to separate local data intervals. For example, the local data bucket is [a, b].

[0055] Among them, the local significant change points are used to characterize the data points in the dataset corresponding to the sub-nodes where the data frequency changes significantly.

[0056] Optionally, local significant change points are used to define the left and right boundaries of local data buckets.

[0057] Step 104: Obtain the local significant change points in each local histogram, and determine the global significant change points based on the local significant change points.

[0058] Among them, globally significant change points are used to define the boundaries of the global data buckets in the global histogram corresponding to the distributed database.

[0059] The global data bucket is used to divide the global data in the distributed database into global data intervals.

[0060] In one embodiment, after receiving the local histograms transmitted by each sub-node, the coordination node reads the local data buckets corresponding to each local histogram and can obtain the local significant change points of each local data bucket.

[0061] In an optional embodiment, after obtaining local significant change points, the coordination node performs mathematical processing on the local significant change points to obtain global significant change points.

[0062] In an exemplary embodiment, a sorting operation is performed on the local significant change points to obtain a significant change sequence. The difference value between each local significant change point in the significant change sequence and its adjacent local significant change points is determined. If the difference value meets a preset condition, the global significant change point is determined based on the local significant change points.

[0063] Optionally, sorting the points of significant local change can be performed by ranking them from smallest to largest to obtain a sequence of significant changes. Alternatively, the points of significant local change can be ranked from largest to smallest to obtain a sequence of significant changes.

[0064] In one embodiment, after obtaining the significant change sequence, the local significant change points included in the significant change sequence and their respective adjacent local significant change points can be determined, and the global significant change points can be determined based on the local significant change points and their adjacent local significant change points.

[0065] In one embodiment, the difference between a local significant change point and its adjacent local significant change points can be determined, and the global significant change point can be determined based on the difference value. For example, if the difference value meets a preset condition, the global significant change point can be determined based on the difference value.

[0066] In one embodiment, the difference value includes a distance difference value and a frequency difference value, and the preset condition includes at least one of the distance difference value being less than or equal to a preset distance threshold and the frequency difference value being less than or equal to a preset frequency threshold.

[0067] In an exemplary embodiment, when the difference value meets a preset condition, determining the global significant change point based on the local significant change point may include: when the distance difference value is less than or equal to a preset distance threshold and the frequency difference value is less than or equal to a preset frequency threshold, determining the mathematical statistical value of the local significant change point and its adjacent local significant change points, and determining the global significant change point based on the mathematical statistical value.

[0068] Optionally, the mathematical statistical value can be the average between a local significant change point and its adjacent local significant change points.

[0069] In one embodiment, the preset distance threshold and the preset frequency threshold can be preset. For example, a preset distance threshold and preset frequency threshold generation control can be provided, allowing the user to set the preset distance and preset frequency values.

[0070] In an exemplary embodiment, when the difference value meets a preset condition, determining the global significant change point based on the local significant change point may include: when the distance difference value is greater than a preset distance threshold, or when the frequency difference value is greater than a preset frequency threshold, taking the local significant change point and its adjacent local significant change points as the global significant change point.

[0071] For example, suppose the distributed database includes three nodes: a first node, a second node, and a third node, and the data range of each node is 0 to 100.

[0072] For example, suppose the distance threshold can be 5 and the frequency difference threshold can be 10.

[0073] Optionally, the local significant change points of the first node can be 10, 30, 50, and 70. The local data buckets generated based on the above local significant change points and the corresponding frequencies of each local data bucket can include: data bucket [0, 10], frequency 5; data bucket [10, 30], frequency 12; data bucket [30, 50], frequency 20; data bucket [50, 70], frequency 8; and data bucket [70, 100], frequency 10.

[0074] Optionally, the local significant change points of the second node can be 20, 40, 55, and 85. The local data buckets generated based on the above local significant change points and the corresponding frequencies of each local data bucket can include: data bucket [0, 20], frequency 7; data bucket [20, 40], frequency 15; data bucket [40, 55], frequency 18; data bucket [55, 85], frequency 7; and data bucket [85, 100], frequency 5.

[0075] Optionally, the local significant change points of the third node can be 15, 35, 50, or 75. The local data buckets generated based on the above local significant change points and the corresponding frequencies of each local data bucket can include: data bucket [0, 15], frequency 10; data bucket [15, 35], frequency 14; data bucket [35, 50], frequency 17; data bucket [50, 75], frequency 6; and data bucket [75, 100], frequency 9.

[0076] In one embodiment, the coordinating node obtains the local significant change points, local data buckets, and frequencies corresponding to the first, second, and third nodes, respectively.

[0077] For example, the local significant change points may include 10, 15, 20, 30, 35, 40, 50, 55, 70, 75 and 85. The obtained local significant change points can be sorted, for example, sorted from smallest to largest, to obtain the significant change sequence [10, 15, 20, 30, 35, 40, 50, 55, 70, 75, 85].

[0078] In one embodiment, a local significant change point 10 is first obtained, and its adjacent local significant change point is 15.

[0079] The distance difference between the local significant change point 10 and the adjacent local significant change point 15 is determined to be |10-15|=5. Using the local data frequency 5 corresponding to the local data bucket ([0,10]) where the local significant change point 10 is located, and the frequency 10 of the local data bucket ([0,15]) where the local significant change point 10 is located, the frequency difference is determined to be |5-10|=5.

[0080] Optionally, the distance difference value is compared with a preset distance threshold to determine whether the distance difference value is less than or equal to the preset distance threshold. For example, determining whether the distance difference value is less than or equal to the preset distance threshold can be that if the distance difference value is 5 and the preset distance threshold is also 5, then the distance difference value can be considered to meet the condition of the preset distance threshold.

[0081] Optionally, the frequency difference value is compared with a preset frequency threshold to determine whether the frequency difference value is less than or equal to the preset frequency threshold. For example, it is determined whether the frequency difference value is less than or equal to a preset distance threshold. For example, it is determined whether the frequency difference value is less than or equal to the preset frequency threshold, where the frequency difference value is 5 and the preset frequency threshold is 10. Since 5 < 10, the frequency difference value can be considered to meet the preset frequency threshold condition.

[0082] In one embodiment, when the distance difference value meets the condition of a preset distance threshold and the frequency difference value also meets the condition of a preset frequency threshold, the mathematical statistical values ​​of the local significant change point and the adjacent local significant change point are determined.

[0083] For example, a local significant change point is 10, and an adjacent local significant change point is 15. A mathematical statistical value is determined based on the local significant change point 10 and the adjacent local significant change point 12.

[0084] In an alternative embodiment, the mathematical statistic can be the average of the local significant change point 10 and the adjacent local significant change point 12, for example, (10+15) / 2=12.5.

[0085] In one embodiment, the calculated average value can be used to determine the globally significant change points. For example, the calculated average value is used to replace adjacent local significant points, and the resulting significant change sequence is [12.5, 20, 30, 35, 40, 50, 55, 70, 75, 85].

[0086] In one embodiment, a local significant change point 12.5 and an adjacent local significant change point 20 are further obtained.

[0087] The distance difference between the local significant change point 12.5 and its adjacent local significant change point 20 is determined to be |12.5-20|=7.5. Since the distance difference value of 7 is greater than the preset distance threshold of 5, it is considered that the distance difference value does not meet the preset condition. Therefore, for the local significant change point 12.5 and its adjacent local significant change point 20, the determination of mathematical statistical values ​​and frequency difference values ​​will no longer be performed, thereby reducing the amount of data processing and improving processing efficiency.

[0088] Alternatively, the frequency difference value can be determined as |12-15|=3 by using the local data frequency 12 corresponding to the local data bucket ([10, 30]) where the local significant change point 10 is located, and the frequency 15 of the local data bucket ([20, 45]) where the local significant change point 20 is located.

[0089] Optionally, the frequency difference value is compared with a preset difference threshold. If 3 < 10, the frequency difference value can be considered to meet the preset condition. Further, the distance difference value between the local significant change point 12.5 and the adjacent local significant change point 20 is determined to be |12.5-20|=7.5. Since the distance difference value 7 is greater than the preset distance threshold 5, the distance difference value is considered not to meet the preset condition.

[0090] Optionally, if the frequency difference value meets the preset conditions, but the distance difference value does not meet the preset conditions, then the mathematical statistical values ​​of the local significant change points and adjacent local significant change points are not determined, nor are the local significant change points in the significant change sequence replaced by mathematical statistical values. Instead, the local significant change points and adjacent local significant change points are treated as global significant change points, which remain [12.5, 20, 30, 35, 40, 50, 55, 70, 75, 85].

[0091] In an optional embodiment, local significant change points and adjacent local significant change points are obtained from the significant change sequence. The process involves determining the distance difference value and frequency difference value between the local significant change point and its adjacent counterpart, comparing the distance difference value with a preset distance threshold, and comparing the frequency difference value with a preset frequency threshold. Based on the comparison results, it is determined whether to replace the local significant change point in the significant change sequence, until the significant change sequence has been traversed. Optionally, the methods for determining the distance difference value, frequency difference value, comparison between the distance difference value and the preset distance threshold, and comparison between the frequency difference value and the preset frequency threshold are as described above and will not be repeated here.

[0092] Optionally, after traversing the significant change sequence to complete the processing of each local significant change point, the processed significant change sequence is obtained as [12.5, 20, 32.5, 40, 50, 52.5, 70, 72.5, 85].

[0093] Optionally, the distance difference value is compared with a preset distance threshold to determine whether the distance difference value is less than or equal to the preset distance threshold. For example, determining whether the distance difference value is less than or equal to the preset distance threshold can be that if the distance difference value is 5 and the preset distance threshold is also 5, then the distance difference value can be considered to meet the condition of the preset distance threshold.

[0094] Optionally, the frequency difference value is compared with a preset frequency threshold to determine whether the frequency difference value is less than or equal to the preset frequency threshold. For example, it is determined whether the frequency difference value is less than or equal to a preset distance threshold. For example, it is determined whether the frequency difference value is less than or equal to the preset frequency threshold, where the frequency difference value is 5 and the preset frequency threshold is 10. Since 5 < 10, the frequency difference value can be considered to meet the preset frequency threshold condition.

[0095] In one embodiment, when the distance difference value meets the condition of a preset distance threshold and the frequency difference value also meets the condition of a preset frequency threshold, the mathematical statistical values ​​of the local significant change point and the adjacent local significant change point are determined.

[0096] For example, a local significant change point is 10, and an adjacent local significant change point is 15. A mathematical statistical value is determined based on the local significant change point 10 and the adjacent local significant change point 12.

[0097] In an alternative embodiment, the mathematical statistic can be the average of the local significant change point 10 and the adjacent local significant change point 12, for example, (10+15) / 2=12.5.

[0098] In one embodiment, the calculated average value can be used to determine the globally significant change points. For example, the calculated average value is used to replace adjacent local significant points, resulting in a significant change sequence of [12.5, 20, 30, 35, 40, 50, 55, 70, 75, 85], where all data included in the significant change sequence are globally significant change points.

[0099] Step 106: Divide the global data into buckets based on the points of significant global change.

[0100] In one embodiment, global data buckets are obtained by using globally significant change points in a significant change sequence and adjacent globally significant change points.

[0101] For example, suppose the significant change sequence is [12.5, 20, 30, 35, 40, 50, 55, 70, 75, 85], where the globally significant change points are 12.5, 20, 30, 35, 40, 50, 55, 70, 75, 85.

[0102] The global significant change point is 12.5, and the adjacent global significant change point is 20. At this point, the global data bucket [12.5, 20] is obtained.

[0103] Optionally, further division can yield global data buckets [0, 12.5], [12.5, 20], [20, 30], [30, 35], [35, 40], [40, 50], [50, 55], [55, 70], [70, 75], and [75, 80].

[0104] Step 108: Based on the global data bucket, determine the target data bucket from the local data bucket corresponding to the local histogram.

[0105] In one embodiment, it is determined whether there is an intersection between the global data bucket and the local data bucket, so as to determine the target data bucket from the local data bucket.

[0106] In one embodiment, if there is an intersection between the global data bucket and the local data bucket, the local data bucket that intersects with the global data bucket is the target data bucket.

[0107] Alternatively, the global data bucket can be compared with the local data buckets corresponding to each node to determine whether there is any overlap between the global data bucket and the local data bucket.

[0108] For example, the global data bucket [0, 12.5] is obtained, and the obtained global data bucket is compared with the local data bucket corresponding to the first node. If it is determined that there is an intersection with the local data bucket [0, 10], then the data bucket [0, 10] is the target data bucket.

[0109] Optionally, the obtained global data bucket [0, 12.5] is compared with the local data bucket corresponding to the second node. If it is determined that there is an intersection with the local data bucket [0, 20], then the data bucket [0, 20] is also the target data bucket.

[0110] Optionally, the obtained global data bucket [0, 12.5] is compared with the local data bucket corresponding to the third node. If it is determined that there is an intersection with the local data bucket [0, 15], then the data bucket [0, 15] is also the target data bucket.

[0111] In one embodiment, when there are at least two local data buckets corresponding to the intersection, the size of the intersection between each local data bucket and the global data bucket is obtained; based on the intersection size, the local data bucket with the largest intersection with the global data bucket is selected as the target data bucket.

[0112] In one embodiment, if there are at least two local data buckets in the same node that intersect with the global data bucket, the target data bucket can be determined based on the size of the intersection.

[0113] For example, if the global data bucket [12.5, 20] is obtained, the obtained global data bucket is compared with the local data bucket corresponding to the first node. If it is determined that there is an intersection with the local data bucket [10, 30], then the data bucket [10, 30] is the target data bucket.

[0114] Optionally, the obtained global data bucket [12.5, 20] is compared with the local data bucket corresponding to the second node. If it is determined that there is an intersection with the local data bucket [0, 20], then the data bucket [0, 20] is also the target data bucket.

[0115] Optionally, the obtained global data bucket [12.5, 20] is compared with the local data bucket corresponding to the third node to determine if there is an intersection with the local data buckets [0, 15] and [15, 35]. Since there are two local data buckets that intersect with the global data bucket, the target data bucket can be determined based on the size of the intersection between the global data bucket [12.5, 20] and the local data bucket [0, 15], and the comparison result of the size of the intersection between the global data bucket [12.5, 20] and the local data bucket [15, 35]. For example, the local data bucket with the larger intersection is selected as the target data bucket.

[0116] For example, if the intersection size between the global data bucket [12.5, 20] and the local data bucket [0, 15] is less than the intersection size between the global data bucket [12.5, 20] and the local data bucket [15, 35], then the local data bucket [15, 35] is selected as the target data bucket.

[0117] Step 110: Obtain the local frequency corresponding to the target data bucket, and perform weighted calculation on the local frequency to generate the target frequency.

[0118] Step 112: Generate a global histogram based on the global data buckets and the target frequency.

[0119] The target frequency is used to characterize the frequency corresponding to the global data bucket.

[0120] In an optional embodiment, after determining the target data bucket, the target frequency can be calculated based on the local frequency corresponding to the target data bucket, and after obtaining the target frequency, a global histogram can be generated by combining it with the corresponding global data bucket.

[0121] For example, assuming a global data bucket [12.5, 20], the corresponding target data buckets include a local data bucket [10, 30] corresponding to the first node, a local data bucket [0, 20] corresponding to the second node, and a local data bucket [15, 35] corresponding to the third node.

[0122] In one embodiment, after identifying the target data bucket, the local frequency corresponding to each target data bucket is determined. For example, the local frequency of local data bucket [10, 30] is 12, the local frequency of local data bucket [0, 20] is 7, and the local frequency of local data bucket [15, 35] is 14.

[0123] In an optional embodiment, after obtaining the local frequencies corresponding to each target data bucket, the target frequency is calculated by weighted average.

[0124] Optionally, the average value of the local frequencies can be calculated as the target frequency. For example, if the local frequencies are 12, 7, and 14, the target frequency can be calculated as (12+7+14) / 3=11, and the target frequency corresponding to the global data bucket [12.5, 20] can be 11.

[0125] In one embodiment, a target frequency is determined for each global data bucket.

[0126] For example, assuming a global data bucket [0, 12.5], the corresponding target data buckets include the local data buckets [0, 10] corresponding to the first node, the local data buckets [0, 20] corresponding to the second node, and the local data buckets [0, 15] corresponding to the third node.

[0127] In one embodiment, after identifying the target data bucket, the local frequency corresponding to each target data bucket is determined. For example, the local frequency of local data bucket [0, 10] is 5, the local frequency of local data bucket [0, 20] is 7, and the local frequency of local data bucket [0, 15] is 10.

[0128] In an optional embodiment, after obtaining the local frequencies corresponding to each target data bucket, the target frequency is calculated by weighted average.

[0129] Optionally, the average value of the local frequencies can be calculated as the target frequency. For example, if the local frequencies are 12, 7, and 14, the calculated target frequency can be (5+7+10) / 3=7.33, and the target frequency corresponding to the global data bucket [12.5, 20] can be 11.

[0130] In one embodiment, the same method is used to determine the corresponding target frequency for other global data buckets.

[0131] In one embodiment, after the target frequencies corresponding to each global data bucket are determined, a global histogram is generated based on the global data buckets and their corresponding target frequencies.

[0132] In the aforementioned histogram generation method, global significant change points are determined based on local significant change points in the local histograms. Global data buckets are then partitioned based on these generated global significant change points, and the target frequencies corresponding to the global data buckets are further determined. This reduces the inaccuracy caused by directly merging local histograms. Furthermore, in determining the target frequencies, a weighted local evaluation method is used to determine the frequencies corresponding to the global data buckets. This comprehensively considers the overall data image of each node in the distributed database, thereby improving the accuracy of frequency determination and ultimately enhancing the accuracy of the generated histogram, reducing distortion. Moreover, in determining global significant change points using local significant change points, distance difference values ​​and frequency difference values ​​are set. This reduces data distribution discrepancies caused by the generation of global significant change points from local significant change points, improving the accuracy of the data processing results.

[0133] In one embodiment, such as Figure 2 As shown, a histogram generation method is provided. This embodiment illustrates the method's application to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0134] Step 202: Obtain the local histogram corresponding to each sub-node.

[0135] Step 204: Obtain the local significant change points in each local histogram, and perform a sorting operation on the local significant change points to obtain the significant change sequence.

[0136] Step 206: Determine the difference between each local significant change point in the significant change sequence and its adjacent local significant change points.

[0137] The difference values ​​include distance difference values ​​and frequency difference values.

[0138] Step 208: Determine whether the distance difference value is less than or equal to the preset distance threshold and whether the frequency difference value is less than or equal to the preset frequency threshold. If yes, proceed to step 210; otherwise, proceed to step 212.

[0139] Step 210: Determine the mathematical statistical values ​​of the local significant change points and their adjacent local significant change points, determine the global significant change points based on the mathematical statistical values, and continue to step 214.

[0140] Step 212: The local significant change points and their adjacent local significant change points are taken as global significant change points.

[0141] Step 214: Divide the global data into buckets based on the points of significant global change.

[0142] Step 216: Based on the global data bucket, determine the target data bucket from the local data buckets corresponding to the local histogram.

[0143] Step 218: Obtain the local frequencies corresponding to the target data bucket, and perform weighted calculations on the local frequencies to generate the target frequency.

[0144] Step 220: Generate a global histogram based on the global data buckets and the target frequency.

[0145] In the aforementioned histogram generation method, global significant change points are determined based on local significant change points in the local histograms. Global data buckets are then partitioned based on these generated global significant change points, and the target frequencies corresponding to the global data buckets are further determined. This reduces the inaccuracy caused by directly merging local histograms. Furthermore, in determining the target frequencies, a weighted local evaluation method is used to determine the frequencies corresponding to the global data buckets. This comprehensively considers the overall data image of each node in the distributed database, thereby improving the accuracy of frequency determination and ultimately enhancing the accuracy of the generated histogram, reducing distortion.

[0146] In one embodiment, such as Figure 3 As shown, a histogram method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0147] Step 302: Obtain the local histogram corresponding to each sub-node.

[0148] Step 304: Obtain the local significant change points in each local histogram, and determine the global significant change points based on the local significant change points.

[0149] Step 306: Divide the global data into buckets based on the points of significant global change.

[0150] Step 308: Determine the number of local data buckets that intersect with the global data bucket.

[0151] Step 310: Determine whether the number of local data buckets that intersect with the global data bucket is greater than or equal to two. If yes, proceed to step 312; otherwise, proceed to step 316.

[0152] Step 312: Obtain the intersection size of each local data bucket and the global data bucket.

[0153] Step 314: Based on the intersection size, select the local data bucket with the largest intersection with the global data bucket as the target data bucket.

[0154] Step 316: Determine the local data bucket that intersects with the global data bucket as the target data bucket.

[0155] Step 318: Obtain the local frequency corresponding to the target data bucket, and perform weighted calculation on the local frequency to generate the target frequency.

[0156] Step 320: Generate a global histogram based on the global data buckets and the target frequency.

[0157] In the aforementioned histogram generation method, global significant change points are determined based on local significant change points in the local histograms. Global data buckets are then partitioned based on these generated global significant change points, and the target frequencies corresponding to the global data buckets are further determined. This reduces the data inaccuracies caused by directly merging local histograms. Furthermore, in this histogram generation method, distance difference values ​​and frequency difference values ​​are set when using local significant change points to determine global significant change points. This approach reduces data distribution discrepancies caused by the generation of global significant change points from local significant change points, improving the accuracy of the data processing results.

[0158] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0159] Based on the same inventive concept, this application also provides a histogram generation apparatus for implementing the histogram generation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more histogram generation apparatus embodiments provided below can be found in the limitations of the histogram generation method described above, and will not be repeated here.

[0160] In one exemplary embodiment, such as Figure 4 As shown, a histogram generation device 400 is provided, including: a histogram acquisition module 402, a change point determination module 404, a partitioning module 406, a data bucket determination module 408, a target frequency generation module 410, and a histogram generation module 412, wherein:

[0161] The histogram acquisition module 402 is used to acquire the local histogram corresponding to each sub-node.

[0162] The change point determination module 404 is used to obtain local significant change points in each local histogram and determine global significant change points based on the local significant change points.

[0163] The partitioning module 406 is used to partition the global data into buckets based on globally significant change points.

[0164] The data bucket determination module 408 is used to determine the target data bucket from the local data buckets corresponding to the local histogram based on the global data bucket.

[0165] The target frequency generation module 410 is used to obtain the local frequency corresponding to the target data bucket, perform weighted calculation on the local frequency, and generate the target frequency.

[0166] The histogram generation module 412 is used to generate a global histogram based on the global data bucket and the target frequency.

[0167] In one embodiment, the change point determination module 404 is further configured to perform a sorting operation on the local significant change points to obtain a significant change sequence before determining the global significant change point based on the local significant change points; determine the difference value between each local significant change point in the significant change sequence and its adjacent local significant change points; and determine the global significant change point based on the local significant change points if the difference value meets a preset condition.

[0168] In an optional embodiment, the difference value includes a distance difference value and a frequency difference value, and the preset conditions include at least one of the distance difference value being less than or equal to a preset distance threshold and the frequency difference value being less than or equal to a preset frequency threshold; the change point determination module 404 is further configured to determine the mathematical statistical value of a local significant change point and its adjacent local significant change points when the distance difference value is less than or equal to the preset distance threshold and the frequency difference value is less than or equal to the preset frequency threshold, and determine a global significant change point based on the mathematical statistical value.

[0169] In one embodiment, the change point determination module 404 is further configured to, when the distance difference value is greater than a preset distance threshold, or the frequency difference value is greater than a preset frequency threshold, use the local significant change point and the adjacent local significant change point as the global significant change point.

[0170] In one embodiment, the data bucket determination module 408 is further configured to, if there is an intersection between the global data bucket and the local data bucket, then the local data bucket that intersects with the global data bucket is the target data bucket.

[0171] In one embodiment, the data bucket determination module 408 is further configured to, when the number of local data buckets corresponding to the intersection is at least two, obtain the intersection size of each local data bucket with the global data bucket; and, based on the intersection size, select the local data bucket with the largest intersection with the global data bucket as the target data bucket.

[0172] Each module in the aforementioned histogram generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0173] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a histogram generation method.

[0174] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0175] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method embodiments.

[0176] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method embodiments.

[0177] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the above-described method embodiments.

[0178] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0179] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0180] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0181] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A histogram generation method characterized by, The method comprises: obtaining local histograms corresponding to each subnode; obtaining local significant change points in each local histogram, and determining global significant change points according to the local significant change points; dividing global data buckets according to the global significant change points; determining target data buckets from local data buckets corresponding to the local histograms according to the global data buckets; obtaining local frequencies corresponding to the target data buckets, and performing weighted calculation on the local frequencies to generate target frequencies; generating a global histogram according to the global data buckets and the target frequencies.

2. The method of claim 1, wherein, Before the global significant change points are determined according to the local significant change points, the method further comprises: performing a sorting operation on the local significant change points to obtain a significant change sequence; determining difference values between each local significant change point and an adjacent local significant change point in the significant change sequence; in a case where the difference values meet preset conditions, determining the global significant change points according to the local significant change points.

3. The method of claim 2, wherein, The difference values include distance difference values and frequency difference values, and the preset conditions include at least one of a distance difference value being less than or equal to a preset distance threshold value and a frequency difference value being less than or equal to a preset frequency threshold value; in the case where the difference values meet the preset conditions, determining the global significant change points according to the local significant change points, comprises: when the distance difference value is less than or equal to the preset distance threshold value and the frequency difference value is less than or equal to the preset frequency threshold value, determining mathematical statistics values of the local significant change point and the adjacent local significant change point, and determining the global significant change points based on the mathematical statistics values.

4. The method of claim 3, wherein, After the difference values between each local significant change point and an adjacent local significant change point in the significant change sequence are determined, the method further comprises: when the distance difference value is greater than the preset distance threshold value or the frequency difference value is greater than the preset frequency threshold value, regarding the local significant change point and the adjacent local significant change point as the global significant change points.

5. The method of claim 1, wherein, The target data buckets are determined from the local data buckets corresponding to the local histograms according to the global data buckets, comprising: if there is an intersection between the global data buckets and the local data buckets, the local data buckets having the intersection with the global data buckets are target data buckets.

6. The method of claim 5, wherein, The local data buckets having the intersection with the global data buckets are target data buckets, comprising: when the number of the local data buckets corresponding to the intersection is at least two, obtaining intersection sizes of each local data bucket and the global data buckets; based on the intersection sizes, selecting the local data bucket having the largest intersection with the global data buckets as the target data bucket.

7. A histogram generation apparatus characterized by comprising: The device comprises: a histogram obtaining module configured to obtain local histograms corresponding to each subnode; a change point determining module configured to obtain local significant change points in each local histogram, and determine global significant change points according to the local significant change points; a dividing module configured to divide global data buckets according to the global significant change points; a data bucket determination module, configured to determine a target data bucket from the local data bucket corresponding to the local histogram according to the global data bucket; a target frequency generation module, configured to obtain a local frequency corresponding to the target data bucket, and perform weighted calculation on the local frequency to generate a target frequency; a histogram generation module, configured to generate a global histogram according to the global data bucket and the target frequency.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

Images