A statistical information processing method and device, a storage medium and an electronic device

Abstract

The application discloses a statistical information processing method and device, a storage medium and an electronic device, and relates to the technical field of computers. The method comprises the following steps: obtaining overall data change information corresponding to a plurality of data partitions in a data table; in response to the overall data change information meeting a data change condition, determining at least one target partition with data change from the plurality of data partitions; associating a statistical block corresponding to the data table with time information to obtain a target statistical block; and updating statistical information based on the target statistical block and the at least one target partition to obtain updated target statistical information. Compared with the prior art, the application can reduce system resource consumption and comprehensively grasp the dynamic change of the data table by determining the target partition in response to the data change condition; and the accuracy of the updated statistical information can be ensured, and the rationality and stability of the query execution plan can be improved by associating the statistical block with the time information and updating the statistical information in combination with the target partition.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a method, apparatus, storage medium and electronic device for processing statistical information. Background Technology

[0002] In database systems, statistical information is the core basis for query optimizers to generate efficient execution plans. Its accuracy and timeliness directly affect database query performance. Especially in big data and complex query scenarios, the processing effect of statistical information is crucial to the overall system performance.

[0003] Currently, the existing method for processing statistical information is to use periodic full table sampling or fixed data change thresholds to trigger updates, generating static snapshots containing statistical data. The query optimizer only uses the latest static snapshot to estimate query costs and select the optimal execution path.

[0004] However, static snapshots generated using this method lack temporal semantic association and cannot reflect the evolution of data over time through the processing, resulting in large cost estimation deviations in time-sensitive query scenarios, which in turn reduces the accuracy and stability of execution plan selection. Summary of the Invention

[0005] In view of this, this application provides a method, apparatus, storage medium and electronic device for processing statistical information. The main purpose is to improve the technical problem that static snapshots generated by the prior art lack temporal semantic association, cannot reflect the evolution characteristics of data over time through the processing process, resulting in large deviations in cost estimation in time-sensitive query scenarios, and thus reducing the accuracy and stability of execution plan selection.

[0006] Firstly, this application provides a method for processing statistical information, including: Retrieve overall data change information corresponding to multiple data partitions in a data table; In response to the overall data change information meeting the data change conditions, at least one target partition with data change is determined from the plurality of data partitions; The statistical blocks corresponding to the data table are associated with time information to obtain the target statistical block; The statistical information is updated based on the target statistical block and the at least one target partition to obtain updated target statistical information, which is used to query the data table.

[0007] Optionally, the statistical blocks corresponding to the data table are associated with time information to obtain target statistical blocks, including: The metadata in the statistical block is locked to determine the time information corresponding to the statistical block; The target statistical block is obtained by marking the statistical block with a time identifier based on the time information.

[0008] Optionally, the statistical information is updated based on the target statistical block and the at least one target partition to obtain updated target statistical information, including: The at least one target partition is sampled to obtain the current data distribution information corresponding to the at least one target partition; Based on the historical data distribution information and the current data distribution information of the at least one target partition, the target data distribution information corresponding to the at least one target partition is obtained by fusing the data distribution information of the at least one target partition. The target statistical information is obtained based on the target statistical blocks and target data distribution information.

[0009] Optionally, after updating the statistical information based on the target statistical block and the at least one target partition to obtain updated target statistical information, the method further includes: In response to querying target data in the data table, at least one query path corresponding to the target data is determined; Based on the target statistical information, determine the execution requirement information corresponding to at least one query path; Based on the execution requirement information, a target query path is determined from the at least one query path, and the target query path is used to perform target data query.

[0010] Optionally, the execution requirement information corresponding to at least one query path is determined from the target statistics, including: Based on the target statistical information, selectivity data and access demand data for each field in the at least one query path are determined, wherein the access demand data includes indexing demand and input / output demand. The execution requirement information is determined based on the selection rate data and the access requirement data.

[0011] Optionally, retrieve overall data change information corresponding to multiple data partitions in the data table, including: Obtain the data update records of the data table within the target time period; Based on the data update records, determine the data change information corresponding to the multiple data partitions respectively; The data change information is fused to obtain the overall data change information.

[0012] Secondly, this application provides a statistical information processing apparatus, comprising: The acquisition module is configured to acquire overall data change information corresponding to multiple data partitions in a data table; The determination module is configured to determine at least one target partition from the plurality of data partitions in response to the overall data change information meeting the data change conditions; The association module is configured to associate the statistical blocks corresponding to the data table with time information to obtain the target statistical block; The update module is configured to update the statistical information based on the target statistical block and the at least one target partition to obtain updated target statistical information, which is used to query the data table.

[0013] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the statistical information processing method described in the first aspect.

[0014] Fourthly, this application provides an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the computer program to implement the statistical information processing method described in the first aspect.

[0015] By employing the above technical solutions, this application provides a method, apparatus, storage medium, and electronic device for processing statistical information. This method acquires overall data change information corresponding to multiple data partitions in a data table; in response to the overall data change information meeting data change conditions, it determines at least one target partition from the multiple data partitions where data changes occur; it associates the statistical blocks corresponding to the data table with time information to obtain target statistical blocks; and it updates the statistical information based on the target statistical blocks and at least one target partition to obtain updated target statistical information, which is used for data querying of the data table. Compared with existing technologies, this application, by acquiring overall data change information from multiple data partitions, can comprehensively grasp the dynamic changes of the data table, improving the accuracy of statistical information updates; determining target partitions in response to data change conditions reduces system resource consumption; associating statistical blocks with time information and combining them with target partitions to update statistical information ensures the accuracy of the updated statistical information, providing reliable support for subsequent query optimization and improving the rationality and stability of the query execution plan. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a statistical information processing method provided in an embodiment of this application is shown. Figure 2 A flowchart illustrating a statistical information processing method provided in an embodiment of this application is shown. Figure 3 A schematic diagram illustrating an example provided in an embodiment of this application is shown; Figure 4 This paper shows a schematic diagram of the structure of a statistical information processing device provided in an embodiment of this application; Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0019] The embodiments of this application will now be described in more detail with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0020] To address the technical problem that existing technologies generate static snapshots that lack temporal semantic associations and cannot reflect the evolution of data over time, leading to large cost estimation errors in time-sensitive query scenarios and consequently reducing the accuracy and stability of execution plan selection, this embodiment provides a method for processing statistical information, such as... Figure 1 As shown, the method includes: Step 101: Obtain the overall data change information corresponding to multiple data partitions in the data table.

[0021] In the embodiments of this application, the data table can be a structured data table in a database system that stores business data. The data table can be a single-table structure or a partitioned table structure. For example, the data table in the embodiments of this application may specifically include a user transaction data table in the financial industry, a user call log data table in the telecom operator, an order data table in the e-commerce platform, etc.

[0022] In the embodiments of this application, data partitioning can be a sub-data set after a data table is divided according to preset rules. Data partitioning can include various types such as range partitioning, hash partitioning, and list partitioning. For example, data partitioning in the embodiments of this application can specifically include daily data partitioning and weekly data partitioning divided by time dimension, and can also include data partitioning for different regions and different product types divided by business dimension.

[0023] In this embodiment, the overall data change information can be a comprehensive set of information reflecting the degree of data change in all data partitions of a data table within a specified time period. The overall data change information may include dimensions such as the quantity, proportion, and frequency of data changes. For example, in this embodiment, the overall data change information may specifically include the weighted total change rate of the entire table, the cumulative value of the changed data volume in each partition, etc.

[0024] Step 102: In response to the overall data change information meeting the data change conditions, determine at least one target partition from multiple data partitions where data changes exist.

[0025] In this embodiment, the data change condition can be a threshold condition used to determine whether to trigger an update of statistical information. The data change condition can be a fixed numerical threshold or a dynamically adjusted threshold range. For example, the data change condition in this embodiment can specifically include an overall data change rate threshold, an overall change data volume threshold, etc.

[0026] In this embodiment, the target partition can be a partition among multiple data partitions where data insertion, updating, deletion, or other changes occur. There can be one or more target partitions. For example, in this embodiment, the target partition may specifically include a partition whose partition change rate exceeds a partition-level threshold, or a partition whose changed data volume exceeds a preset number.

[0027] In this embodiment of the application, if the overall data change information meets the preset data change conditions, such as the overall data change rate exceeding a threshold, then the subsequent partition filtering operation is triggered. When determining at least one target partition with data changes from multiple data partitions, the specific amount of changed data in each partition (such as the number of inserted, updated, and deleted records) can be counted by parsing the data update records of each partition. If the amount of changed data in a partition exceeds the preset partition-level change threshold, or the partition change rate Vt(Pi) is greater than the set threshold, then the partition is determined as the target partition.

[0028] It should be noted that, in addition to the above-mentioned screening method based on the rate of change threshold, the target partition can also be determined by comparing the differences in data volume and data content between the partitions before and after the target time period. For example, when the absolute value of the change in the data volume of a partition exceeds a preset number, it is determined as the target partition. The specific implementation method for determining at least one target partition can be flexibly selected.

[0029] Step 103: Associate the statistical blocks corresponding to the data table with the time information to obtain the target statistical blocks.

[0030] In this embodiment of the application, the statistical block can be a basic data unit for storing statistical information of the data table. The statistical block can include statistical data such as field value distribution histogram, number of unique values, proportion of null values, maximum and minimum values. For example, the specific statistical block in this embodiment of the application can include statistical data unit corresponding to a single field or statistical data unit corresponding to multiple fields.

[0031] In this embodiment, the time information can be time-dimensional information associated with the statistical block, including the start time, end time, and data activity level of the statistical block. For example, the time information in this embodiment may specifically include information in the form of timestamps, time density functions, etc.

[0032] In this embodiment of the application, the target statistical block can be a statistical block that possesses temporal semantics after being associated with time information. The target statistical block can be used to reflect the time range corresponding to the statistical data and the evolution characteristics of the data over time. For example, in this embodiment of the application, the target statistical block can specifically provide the query optimizer with statistical data support with a time dimension.

[0033] In the embodiments of this application, associating the statistical blocks corresponding to the data table with time information can first determine the time information corresponding to the statistical blocks. The time information can include the start timestamp Bi and end timestamp Ei of the statistical blocks, and can also include the time density function fi(t) used to characterize the activity level of the data within the time interval. Associating the statistical blocks with time information enables the statistical blocks to have time semantics, thereby generating the target statistical blocks.

[0034] Step 104: Update the statistical information based on the target statistical block and at least one target partition to obtain the updated target statistical information.

[0035] Among them, the target statistics are used to query data in the data table.

[0036] In the embodiments of this application, statistical information can be information reflecting the data distribution characteristics of a data table. Statistical information may include field value distribution histograms, number of unique values, proportion of null values, selection rate, and other information. Statistical information can be used to support the query optimizer in cost estimation and execution path selection. For example, the statistical information in the embodiments of this application may specifically include single-table statistical information, multi-table association statistical information, etc.

[0037] In the embodiments of this application, the target statistical information may be statistical information that integrates the time characteristics of the target statistical block and the latest data distribution of the target partition. For example, the target statistical information in the embodiments of this application may specifically include statistical information after updating a single field, statistical information after jointly updating multiple fields, etc.

[0038] In this embodiment, updating statistical information based on the target statistical block and at least one target partition can combine the temporal semantic features of the target statistical block with the latest data changes of the target partition, and adopt an incremental update method to achieve accurate optimization of statistical information, thereby avoiding the resource consumption caused by full table sampling.

[0039] Compared with existing technologies, the embodiments of this application can comprehensively grasp the dynamic changes of data tables by obtaining overall data change information of multiple data partitions, thereby improving the accuracy of statistical information updates; determining the target partition in response to data change conditions can reduce system resource consumption; associating statistical blocks with time information and updating statistical information in combination with target partitions can ensure the accuracy of updated statistical information, provide reliable support for subsequent query optimization, and improve the rationality and stability of query execution plans.

[0040] As an optional approach, when performing the task of "associating the statistical blocks corresponding to the data table with time information to obtain the target statistical block", the following methods can be used, but are not limited to these: Figure 2 As shown, it includes: Step 201: Lock the metadata in the statistical block to determine the time information corresponding to the statistical block.

[0041] For example, the time density function fi(t) in this application embodiment can specifically adopt an exponential decay model, assuming that data changes are most concentrated at the beginning of the segment and then gradually decrease, which is suitable for periodic business activities, such as data entry at the beginning of the month and review at the end of the month. The time density function fi(t) is shown in Formula 1, where, This can be represented as a data evolution intensity parameter. It can reflect the frequency of data changes (rate of change per unit time) within a time interval. The larger the value, the more "active" the data. Bi can represent the starting timestamp of the statistical block Si, and t can represent any point in time.

[0042] (Formula 1) In the embodiments of this application, metadata can be data describing the basic attributes of a statistical block. For example, the metadata in the embodiments of this application may specifically include the identifier of the statistical block, the corresponding data table and field information, the generation time, the version number, etc. The metadata can be attribute description data of the statistical block, or management and control data of the statistical block, etc.

[0043] In this embodiment, locking can be an operation to fix the state of the metadata of a statistical block. Locking can be used to prevent the statistical block from being accidentally overwritten or automatically updated. For example, locking in this embodiment can be implemented by setting a locking identifier, constructing a locking information structure, etc. For instance, locking can be setting a locking identifier "LOCKED" on the metadata of the statistical block to prevent the system from automatically updating the statistical block.

[0044] In this embodiment, the time information corresponding to the statistical block can be time information related to the data collection and effective range of the statistical block. This time information may include the start and end times of data collection, time parameters related to data evolution characteristics, etc. For example, the time information corresponding to the statistical block in this embodiment may specifically be the actual business time range corresponding to the statistical block data or the generation and effective time range of the statistical block, etc.

[0045] For the embodiments of this application, locking the metadata in the statistical block can be achieved by constructing a locking information structure, Lk, as shown in Formula 2. Here, lock can be identified as the unique locking identifier of the statistical block, which can be a GUID (globally unique identifier) ​​generated by the system. ts can be identified as the timestamp when the statistical block is locked, and the format of ts can be ISO8601. version can represent the version number of the statistical information, which can support semantic version (e.g., v1.2.3) or time version. hash can represent the hash value calculated from the key content of the current statistical block (e.g., histogram Hi(x) and time density function fi(t)). The hash value can be calculated using digest algorithms such as SHA-256. It is obtained by serializing the histogram structure and the time density function, concatenating them, and then performing hash calculation, as shown in Formula 3. Here, Serialize(Hi) can represent serializing the histogram structure into a string according to fixed fields and order, and Serialize(fi(t)) can represent converting the expression of the time density function or the discrete value sequence into a string in order.

[0046] (Formula 2) (Formula 3) For the embodiments of this application, the target statistical block structure can be as shown in Formula 4, where Bi and Ei can both represent time boundaries, Bi and Ei can be used to identify the life cycle of statistical data, Hi(x) can be represented as a histogram of the value distribution of field x, Hi(x) can be used to estimate selectivity, and fi(t) can be represented as a time density function.

[0047] (Formula 4) It should be noted that, in addition to the start timestamp Bi, end timestamp Ei, and time density function fi(t), the time information corresponding to the statistical block can also include the generation time and update time of the statistical block. In addition to the exponential decay model, other forms such as linear model and polynomial model can also be used for modeling the time density function, and the specific choice can be made according to the data evolution characteristics.

[0048] Step 202: Mark the statistical blocks with time identifiers based on time information to obtain the target statistical blocks.

[0049] In this embodiment, the time identifier can be an identifier generated based on time information to mark the time attributes of statistical blocks. The time identifier can be in the form of a timestamp, a time range label, a time density function identifier, etc. For example, the time identifier in this embodiment may specifically include a standardized time string, encoded time feature values, such as ISO8601 format time range labels for start and end times, and data evolution intensity parameters. The corresponding identifier.

[0050] In this embodiment, the target statistical block can be a statistical block marked with a time identifier and possessing clear time attributes and time evolution characteristics. The target statistical block can be used to provide time-aware statistical data support for the query optimizer. For example, the target statistical block in this embodiment may specifically include a single statistical block associated with a time identifier, a combination of multiple statistical blocks with time identifiers, etc.

[0051] In this embodiment, marking statistical blocks with time identifiers based on time information can convert the start timestamp Bi, end timestamp Ei, and time density function fi(t) in the time information into a standardized time identifier format. For example, Bi and Ei are converted into ISO8601 format time strings as time range identifiers, and the parameters of the time density function fi(t) are... Converted into a data activity level indicator.

[0052] In this embodiment, binding a time identifier to a statistical block can be achieved by adding a time attribute field to the metadata of the statistical block, generating a time identifier tag, and storing it in association with the statistical block, ultimately obtaining a target statistical block with time semantics. It should be noted that, in addition to the above-mentioned time identifier format, other methods such as time encoding and time hashing can also be used to mark time information; it is not limited to the event identifier format.

[0053] As an optional approach, when performing the action of "updating statistical information based on target statistical blocks and at least one target partition to obtain updated target statistical information", the following method can be used, but is not limited to: sampling at least one target partition to obtain current data distribution information corresponding to at least one target partition; fusing historical data distribution information and current data distribution information of at least one target partition to obtain target data distribution information corresponding to at least one target partition; and obtaining target statistical information based on target statistical blocks and target data distribution information.

[0054] In the embodiments of this application, sampling processing can be a process of extracting a portion of data samples from the target partition for statistical analysis. Sampling processing can employ various sampling methods such as random sampling, stratified sampling, and systematic sampling. The sampling ratio of the sampling processing can be flexibly configured according to the amount of data and the degree of data change in the target partition. For example, the sampling processing in the embodiments of this application can specifically be a stratified sampling method for the target partition, or it can be local sampling of the areas in the target partition where data changes have occurred.

[0055] In the embodiments of this application, the current data distribution information can be information reflecting the current data distribution characteristics obtained by sampling the target partition. The current data distribution information may include the current value distribution histogram of fields within the target partition, the current number of unique values, the current proportion of null values, etc. For example, the current data distribution information in the embodiments of this application may include the current distribution information of a single field and the joint current distribution information of multiple fields.

[0056] In this embodiment of the application, the historical data distribution information can be the data distribution information of the target partition before the current update. The historical data distribution information can be extracted from the historical statistical data of the corresponding partition from the target statistical block. For example, the historical data distribution information in this embodiment of the application may specifically include a historical value distribution histogram, the number of historical unique values, etc.

[0057] In this embodiment, the fusion process can be a weighted integration of historical data distribution information and current data distribution information. The fusion process can employ algorithms such as time decay weighting and proportional weighting to ensure that the fused information retains both the evolutionary characteristics of historical data and conforms to the actual distribution of current data. For example, the fusion process in this embodiment can specifically include fusion by histogram bucket and fusion based on overall distribution characteristics.

[0058] In this embodiment of the application, the target data distribution information can be information reflecting the complete data distribution characteristics of the target partition after fusion processing. This target data distribution information can be used to provide accurate data support for updating the statistical information of the target partition. Specifically, the target data distribution information in this embodiment may include a fused field value distribution histogram, the number of fused unique values, etc.

[0059] For the embodiments of this application, sampling processing of at least one target partition can be performed by formulating a separate sampling strategy for each target partition, and only local sampling can be performed on the areas within the target partition where data changes have occurred, such as only sampling the areas corresponding to the data rows involved in the insertion and update operations; after obtaining the current data distribution information, the historical data distribution information (such as the historical value distribution histogram Hi(x)) of the corresponding target partition can be extracted from the target statistical block; fusion can be performed using a weighted fusion histogram, and fusion calculation is performed separately for each bucket of the histogram, as shown in Formula 5, where Hi(x) can represent the histogram of field x in the original statistical block, Hi(x) can reflect the data distribution over a period of time, Hi'(x) can represent the new histogram sampled from the newly added or changed data within the update interval, Hi''(x) can represent the fused histogram, Hi''(x) is the weighted summation result, and Hi''(x) serves as the updated statistical information. It can represent weighting coefficients. It can indicate the degree of retention of historical information, if The larger the value, the more it indicates a greater reliance on historical information. It can indicate the degree of confidence in the new sample.

[0060] (Formula 5) In the embodiments of this application, The weighting coefficients can be represented as shown in Formula 6, where, It can represent the intensity of data change within a unit of time within a statistical block or partition. It can be inferred by inserting, updating, or deleting logs. It can represent the interval between the current time and the last time the statistics were updated.

[0061] (Formula 6) For embodiments of this application, if the historical data distribution information or the current data distribution fluctuates greatly and the fused distribution is unstable, a Gaussian perturbation term and a normalization function can be introduced for correction, as shown in Formula 7, wherein, This can be expressed as Gaussian error coefficients. It can be used to compensate for the introduction of sample bias. Gnoise(x) can be expressed as a Gaussian perturbation function to simulate sampling error. Gnoise(x) can add a small fluctuation to each histogram bucket. Normalize() can be expressed as a normalization function to ensure that the sum of the values ​​of each bucket is 1, thus maintaining the validity of the histogram.

[0062] (Formula 7) In this embodiment, after obtaining the target data distribution information corresponding to the target partition through the above fusion process, the original statistical information can be updated by combining the time information of the target statistical block (such as time density function, time boundary, etc.). The target data distribution information replaces the statistical data of the corresponding target partition in the original statistical information, and the time identifier of the target statistical block is retained, thus obtaining the target statistical information. It should be noted that, in addition to the above weighted fusion algorithm, other fusion methods such as mean fusion and median fusion can also be used, and it is not limited to the above weighted fusion algorithm.

[0063] Optionally, after performing the step of "updating the statistical information based on the target statistical block and at least one target partition to obtain the updated target statistical information", the following method may be used, but is not limited thereto: in response to querying the target data in the data table, determining at least one query path corresponding to the target data; determining the execution requirement information corresponding to the at least one query path based on the target statistical information; and determining the target query path from the at least one query path based on the execution requirement information, wherein the target query path is used to query the target data.

[0064] In this embodiment of the application, the target data can be the data content that the user requests to obtain. The target data can be part of the row data, part of the column data, or multi-table related data in a data table. For example, the target data in this embodiment of the application can specifically include a single data record and a batch data set, etc.

[0065] In this embodiment, the query path can be the specific execution path of the database system to obtain target data. The query path can include various types such as a full table scan path, an index scan path, and a multi-table join path. For example, the query path in this embodiment can specifically include a single-step execution path and a multi-step combined execution path.

[0066] In the embodiments of this application, the execution requirement information may be information reflecting characteristics such as resource requirements and execution efficiency during the execution of the query path. For example, the execution requirement information in the embodiments of this application may include execution time, CPU utilization, IO overhead, memory requirements, etc.

[0067] In the embodiments of this application, the target query path can be the query path with the best execution efficiency and lowest resource consumption selected from multiple query paths. For example, the target query path in the embodiments of this application may specifically include a single optimal path or multiple alternative optimal paths.

[0068] In the embodiments of this application, in response to querying target data in a data table, the query statement can be parsed by a database query parser to generate multiple possible query paths. For example, for querying fields containing indexes, an index scan path and a full table scan path can be generated.

[0069] In the embodiments of this application, the execution requirement information corresponding to at least one query path can be determined based on the target statistical information. The precise data distribution characteristics and temporal semantics in the target statistical information can be used to make a detailed estimate of the execution cost of each query path.

[0070] For example, when determining the target query path from at least one query path based on execution requirement information, the shortest execution time, lowest resource consumption, or best overall execution cost can be used as the screening criteria to select the query path that best meets the query requirements as the target query path.

[0071] Optionally, when performing the "determine the execution requirement information corresponding to at least one query path based on target statistics", the following method can be used, but is not limited to: determining the selectivity data and access requirement data of each field in at least one query path based on target statistics, wherein the access requirement data includes index requirements and input / output requirements; and determining the execution requirement information based on the selectivity data and access requirement data.

[0072] In this embodiment, the selection rate data can be information about the filtering ratio of field data by query conditions. The selection rate data can reflect the proportion of data retained after filtering by query conditions to the total amount of data in the field. For example, the selection rate data in this embodiment can specifically include the selection rate of a single field and the combined selection rate of multiple fields.

[0073] In this embodiment, the access requirement data can be resource requirement information when accessing field data during the execution of the query path. The access requirement data can include index requirement data, input / output requirement data, etc. For example, the index requirement data in this embodiment can specifically be the requirement data for whether the query path needs to use an index and what type of index (such as a B-tree index or a hash index) should be used. The input / output requirement data can be the IO resource requirement data for data reading and writing during the execution of the query path.

[0074] In this embodiment of the application, the execution requirement information can be information reflecting the overall execution cost of the query path, obtained by combining the selection rate data and the access requirement data. The execution requirement information can be used to provide a quantitative basis for filtering query paths. For example, the execution requirement information in this embodiment may specifically include specific indicators such as execution time and resource consumption.

[0075] In the embodiments of this application, the selection rate data of each field can be determined based on the target statistical information by using data such as the field value distribution histogram and the number of unique values ​​in the target statistical information, combined with the query conditions. For example, the selection rate data can be determined by the proportion of the histogram interval corresponding to the query conditions.

[0076] In the embodiments of this application, the access requirement data can be determined by analyzing the index information of the fields in the target statistics, the data storage location, and other content to determine the index usage requirements of the query path for that field. At the same time, the selection rate data is combined to estimate the amount of data reading and writing IO and determine the input and output requirements.

[0077] In this embodiment of the application, the execution requirement information determined based on the selection rate data and access requirement data can be comprehensively calculated using a cost estimation model. For example, the amount of data involved in the query can be estimated by using the selection rate data, and the execution requirement information such as the IO overhead and index usage efficiency in the access requirement data can be calculated to determine the execution time and resource consumption of each query path.

[0078] For example, embodiments of this application can employ a boundary-aware multi-block cost estimation mechanism (Time-Aware Multi-Block Cost Estimation) for cost estimation. Specifically, this can include assigning different weights to multiple statistical blocks based on the temporal relationship between the current query time tquery and the statistical block Si, forming a time-adaptive cost evaluation model. The main function for cost estimation can be as shown in Formula 8, where Q can represent the query statement to be optimized, tquery can represent the current query time point, such as the current system time or the time specified by the user, Si can represent the i-th historical statistical block, Eval(Q,Si) can represent the cost evaluation based on Si, Pi can represent the time-related weight of the statistical block Si on tquery, and Cost(Q,tquery) can represent the total cost estimate of multi-block fusion.

[0079] (Formula 8) For example, the cost evaluation within the statistical block can be refined as shown in Formula 9. Here, Qf can represent the field f involved in query Q, Hi(f) can represent the histogram or distribution information of field f in the statistical block Si, Selectivity(Qf,Hi(f)) can represent the selection rate estimate of field f, i.e. the filtering strength of the field, Access_Cost(Qf,Index_Info) can represent the access cost of field f, and Access_Cost(Qf,Index_Info) can be calculated based on whether an index is used, the index type, IO cost, etc.

[0080] (Formula Nine) It should be noted that, in addition to the cost estimation methods mentioned above, other methods such as machine learning model prediction and empirical formula calculation can also be used to determine the execution requirements information, and are not limited to the cost estimation methods mentioned above.

[0081] Optionally, when executing "obtaining the overall data change information corresponding to multiple data partitions in a data table", the following methods can be used, but are not limited to: obtaining the data update records of the data table within the target time period; determining the data change information corresponding to multiple data partitions based on the data update records; and performing fusion processing on the data change information to obtain the overall data change information.

[0082] In the embodiments of this application, the data update record can be log information in the database system that records data operation behavior of data tables. For example, the data update record in the embodiments of this application may specifically include records of operation type (insert, update, delete), operation time, data partition involved, and amount of data operated.

[0083] In the embodiments of this application, the target time period can be a user-preset time interval for statistical data changes. The target time period can include a fixed duration interval or a dynamically adjusted duration interval, etc.

[0084] In this embodiment, the data change information can be the data change information of a single data partition within a target time period. For example, the data change information in this embodiment may specifically include information such as the number of records inserted, updated, and deleted within the partition, and the change ratio.

[0085] For the embodiments of this application, the rate of change of the overall data change information can be calculated by formula ten, where Vt can represent the weighted change rate of the entire table at time t, It can represent the number of newly inserted records during time t, Ut can represent the number of records modified (updated) during time t, Dt can represent the number of records deleted during time t, and Nt can represent the total number of records in the current table (at time t). It can identify tiny values ​​that are prevented from being divided by zero. The specific value can be 0.000001. This can represent the weighting factor being inserted. It can represent the updated weight factor. It can represent the weight factor to be deleted, and satisfies , , or This can represent the degree of impact of different types of changes on statistical information. For example, The specific value can be 0.5. The specific value can be 0.3. The specific value can be 0.2.

[0086] (Formula 10) In this embodiment, the fusion process can be a process of integrating independent data change information from multiple data partitions into a process that reflects the overall change in the data table. For example, the fusion process in this embodiment can be implemented using various algorithms such as weighted calculation and mean calculation.

[0087] In this embodiment of the application, the data update records of the data table within the target time period are first obtained. By parsing the data update records, the number of inserted records, updated records, and deleted records of each partition within the target time period are counted. Combined with the current total number of records in the partition, the change rate of each partition is calculated by the partition change rate. The partition change rate is shown in Formula 11, where Pi can represent the i-th data partition, It(Pi) can represent the number of inserted records in partition Pi, Ut(Pi) can represent the number of updated records in partition Pi, Dt(Pi) can represent the number of deleted records in partition Pi, and Nt(Pi) can represent the current total number of records in partition Pi.

[0088] (Formula Eleven) In this embodiment of the application, the data change information of each partition is fused to obtain the overall data change information. The overall data change rate can be calculated using the full table change rate integration formula, as shown in Formula Twelve. In this formula, Vt_total can represent the global weighted change rate of the entire partition table, and wi can represent the data proportion of partition Pi in the entire table. wi can also be represented as shown in Formula Thirteen, where Nt(Pi) can represent the current total number of records in partition Pi, and Nt(Pj) can represent the current total number of records in partition Pj.

[0089] (Formula 12) (Formula Thirteen) As an optional approach, this application also provides an example, such as... Figure 3 As shown, this example includes five steps: Step 1: Detect and calculate changes in table objects. Monitor insert, update, and delete operations on data tables or partitions in real time, and calculate the change rate Vt_total as the trigger for whether to update statistical information.

[0090] Step 2: Statistical information time boundary modeling mechanism. When sampling statistical information, record its start and end time Bi and Ei, and model the time density function fi(t) to provide time awareness capability for the subsequent optimizer.

[0091] Step 3: Statistical information locking and export / import mechanism. In critical scenarios, statistical blocks (version number and hash signature) can be locked, while cross-environment import and export are supported to ensure the consistency of statistical information.

[0092] Step 4: Incremental statistical information collection and fusion mechanism. When the rate of change exceeds the threshold, local data sampling is performed to generate a new histogram Hi'(x), which is then fused with the original Hi(x) in a time-weighted manner to form Hi'''(x).

[0093] Step 5: Boundary-aware optimizer cost evaluation mechanism. The query optimizer loads multiple statistical blocks Si based on the query time tquery, and weights and merges the estimated values ​​Eval(Q,Si) of multiple blocks according to time distance to improve optimization accuracy.

[0094] Compared with existing technologies, the embodiments of this application ensure the uniqueness and stability of the temporal semantics of statistical information by locking the metadata of statistical blocks and marking them with time identifiers; by combining historical and current data distribution information for fusion updates, the accuracy and continuity of statistical information updates are improved; by filtering the optimal query path based on the target statistical information, the query requirements and data characteristics are accurately matched, which can improve the execution efficiency and stability of target data queries; by extracting field selection rate and access requirement data, more accurate quantitative basis is provided for the selection of the optimal query path; by fusing multi-partition data change information to obtain the overall change situation, the comprehensiveness and accuracy of the statistical information update trigger judgment are ensured, further guaranteeing the rationality of statistical information updates and the efficient use of system resources.

[0095] Furthermore, as Figure 1 and Figure 2 The specific implementation of the method shown in this embodiment provides a statistical information processing device, such as... Figure 4 As shown, the device includes: an acquisition module 31, a determination module 32, an association module 33, and an update module 34.

[0096] The acquisition module 31 is configured to acquire overall data change information corresponding to multiple data partitions in the data table; The determination module 32 is configured to determine at least one target partition from multiple data partitions in response to the overall data change information meeting the data change conditions; The association module 33 is configured to associate the statistical blocks corresponding to the data table with time information to obtain the target statistical block; The update module 34 is configured to update the statistical information based on the target statistical block and at least one target partition to obtain the updated target statistical information, which is used to query the data table.

[0097] In some examples of this embodiment, the association module 33 is specifically configured to lock the metadata in the statistical block, determine the time information corresponding to the statistical block, and mark the statistical block with a time identifier based on the time information to obtain the target statistical block.

[0098] In some examples of this embodiment, the update module 34 is specifically configured to perform sampling processing on at least one target partition to obtain the current data distribution information corresponding to at least one target partition; perform fusion processing based on the historical data distribution information and the current data distribution information of at least one target partition to obtain the target data distribution information corresponding to at least one target partition; and obtain target statistical information based on the target statistical block and the target data distribution information.

[0099] In some examples of this embodiment, the update module 34 is further configured to, in response to querying target data in the data table, determine at least one query path corresponding to the target data; determine execution requirement information corresponding to at least one query path based on target statistical information; and determine a target query path from at least one query path based on the execution requirement information, wherein the target query path is used to query the target data.

[0100] In some examples of this embodiment, the update module 34 is further configured to determine the selectivity data and access demand data for each field in at least one query path based on the target statistics, wherein the access demand data includes indexing demand and input / output demand; and to determine execution demand information based on the selectivity data and access demand data.

[0101] In some examples of this embodiment, the acquisition module 31 is specifically configured to acquire data update records of the data table within a target time period; determine the data change information corresponding to multiple data partitions based on the data update records; and perform fusion processing on the data change information to obtain the overall data change information.

[0102] It should be noted that other corresponding descriptions of the functional units involved in the statistical information processing device provided in this embodiment can be found in [reference]. Figure 1 and Figure 2 The corresponding description in [the document] will not be repeated here.

[0103] Based on the above, Figure 1 and Figure 2 Accordingly, this embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. Figure 1 and Figure 2 The method shown.

[0104] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.

[0105] like Figure 5 The diagram shown is a hardware structure schematic of an electronic device according to the present invention, comprising: At least one processor 401; and, Memory 402 is connected to at least one processor 401 for processing statistical information; wherein, The memory 402 stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the statistical information processing method described above.

[0106] Figure 5 Take a processor 401 as an example.

[0107] The electronic device may also include an input device 403 and an output device 404.

[0108] The processor 401, memory 402, input device 403, and output device 404 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0109] Memory 402, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the statistical information processing method in the embodiments of this application, for example, Figure 1 and Figure 2 The method flow is shown. The processor 401 executes various functional applications and statistical information processing by running non-volatile software programs, instructions, and modules stored in the memory 402, that is, it implements the statistical information processing method in the above embodiments.

[0110] Memory 402 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the statistical information processing method. Furthermore, memory 402 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 402 may optionally include memory remotely located relative to processor 401, and these remote memories may be connected via a network to the apparatus performing the statistical information processing method. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile statistical information processing networks, and combinations thereof.

[0111] Input device 403 can receive user clicks and generate signal inputs related to user settings for statistical information processing methods and function control. Output device 404 may include display devices such as a display screen.

[0112] One or more modules are stored in memory 402, and when run by one or more processors 401, the statistical information processing method in any of the above method embodiments is executed.

[0113] Optionally, the aforementioned physical devices may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0114] Those skilled in the art will understand that the physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0115] The storage medium may also include an operating system and a network statistics processing module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs. The network statistics processing module is used to process statistical information between the various components within the storage medium, as well as between the storage medium and other hardware and software within the information processing physical device.

[0116] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platform, or it can be implemented by hardware. By applying the solution of this embodiment, compared with the existing technology, this application embodiment can comprehensively grasp the dynamic changes of the data table by obtaining the overall data change information of multiple data partitions, thus improving the accuracy of statistical information updates; determining the target partition in response to data change conditions can reduce system resource consumption; associating statistical blocks with time information and updating statistical information in conjunction with the target partition can ensure the accuracy of the updated statistical information, providing reliable support for subsequent query optimization and improving the rationality and stability of the query execution plan; locking the metadata of statistical blocks and marking time identifiers ensures the uniqueness and stability of the time semantics of statistical information; combining historical and current data distribution information for fusion updates improves the accuracy and continuity of statistical information updates; filtering the optimal query path based on the target statistical information achieves accurate matching between query requirements and data characteristics, thus improving the execution efficiency and stability of target data queries; extracting field selection rate and access requirement data provides a more accurate quantitative basis for filtering the optimal query path; and obtaining the overall change situation through the fusion of multi-partition data change information ensures the comprehensiveness and accuracy of statistical information update trigger judgment, further ensuring the rationality of statistical information updates and the efficient use of system resources.

[0117] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0118] The above are merely specific embodiments of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for processing statistical information, characterized in that, include: Retrieve overall data change information corresponding to multiple data partitions in a data table; In response to the overall data change information meeting the data change conditions, at least one target partition with data change is determined from the plurality of data partitions; The statistical blocks corresponding to the data table are associated with time information to obtain the target statistical block; The step of associating the statistical blocks corresponding to the data table with time information to obtain the target statistical block includes: locking the metadata in the statistical block, determining the time information corresponding to the statistical block; and marking the statistical block with a time identifier based on the time information to obtain the target statistical block. Associating the statistical blocks corresponding to the data table with time information is to determine the time information corresponding to the statistical blocks. The time information includes the start timestamp Bi and end timestamp Ei of the statistical blocks, and also includes a time density function fi(t) used to characterize the activity level of the data within the time interval. Associating the statistical blocks with the time information gives the statistical blocks time semantics, and generates the target statistical block. The statistical information is updated based on the target statistical block and the at least one target partition to obtain updated target statistical information, which is used to query the data table. The step of updating the statistical information based on the target statistical block and the at least one target partition to obtain updated target statistical information includes: sampling the at least one target partition to obtain current data distribution information corresponding to the at least one target partition; fusing the historical data distribution information and the current data distribution information of the at least one target partition to obtain target data distribution information corresponding to the at least one target partition; and obtaining the target statistical information based on the target statistical block and the target data distribution information. The fusion process is a weighted integration of the historical data distribution information and the current data distribution information. The fusion process uses time decay weighting and proportional weighting algorithms to ensure that the fused information retains the evolution characteristics of historical data while conforming to the actual distribution of current data.

2. The method according to claim 1, characterized in that, After updating the statistical information based on the target statistical block and the at least one target partition to obtain the updated target statistical information, the method further includes: In response to querying target data in the data table, at least one query path corresponding to the target data is determined; Based on the target statistical information, determine the execution requirement information corresponding to at least one query path; Based on the execution requirement information, a target query path is determined from the at least one query path, and the target query path is used to perform target data query.

3. The method according to claim 2, characterized in that, The step of determining the execution requirement information corresponding to at least one query path based on the target statistical information includes: Based on the target statistical information, selectivity data and access demand data for each field in the at least one query path are determined, wherein the access demand data includes indexing demand and input / output demand. The execution requirement information is determined based on the selection rate data and the access requirement data.

4. The method according to any one of claims 1 to 3, characterized in that, The step of obtaining overall data change information corresponding to multiple data partitions in a data table includes: Obtain the data update records of the data table within the target time period; Based on the data update records, determine the data change information corresponding to the multiple data partitions respectively; The data change information is fused to obtain the overall data change information.

5. A statistical information processing device, characterized in that, include: The acquisition module is configured to acquire overall data change information corresponding to multiple data partitions in a data table; The determination module is configured to determine at least one target partition from the plurality of data partitions that has data changes in response to the overall data change information meeting the data change conditions; The association module is configured to associate the statistical blocks corresponding to the data table with time information to obtain the target statistical block; The association module is also configured to lock the metadata in the statistical block and determine the time information corresponding to the statistical block; The target statistical block is obtained by marking the statistical block with a time identifier based on the time information. Associating the statistical blocks corresponding to the data table with time information is to determine the time information corresponding to the statistical blocks. The time information includes the start timestamp Bi and end timestamp Ei of the statistical blocks, and also includes a time density function fi(t) used to characterize the activity level of the data within the time interval. Associating the statistical blocks with the time information gives the statistical blocks time semantics, and generates the target statistical block. The update module is configured to update the statistical information based on the target statistical block and the at least one target partition to obtain updated target statistical information, which is used to query the data table. The update module is also configured to perform sampling processing on the at least one target partition to obtain the current data distribution information corresponding to the at least one target partition; Based on the historical data distribution information and the current data distribution information of the at least one target partition, the target data distribution information corresponding to the at least one target partition is obtained by fusing the data distribution information of the at least one target partition. Based on the target statistical blocks and target data distribution information, the target statistical information is obtained; The fusion process is a weighted integration of the historical data distribution information and the current data distribution information. The fusion process uses time decay weighting and proportional weighting algorithms to ensure that the fused information retains the evolution characteristics of historical data while conforming to the actual distribution of current data.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

7. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 4.

8. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.

Images