Database data processing method and device, computer device and storage medium

Abstract

The application relates to a database data processing method and device, computer equipment and a storage medium. The method comprises the following steps: for each data attribute of data in a database, based on the frequency index value of various attribute values of the data attribute, the various attribute values of the data attribute are determined as high-frequency attribute values or low-frequency attribute values; for each high-frequency attribute value, a first type of bucket is determined based on the high-frequency attribute value; at least one second type of bucket is determined based on a plurality of low-frequency attribute values; a histogram vector corresponding to the data attribute is constructed according to the first type of bucket and the second type of bucket; and the database feature representation of the database is obtained by joint encoding based on the histogram vectors corresponding to the respective data attributes of the data in the database. The method can save storage costs in the database data processing process.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a database data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the development of database technology, intelligent data management and analysis can be achieved through databases. In the database field, the effective representation of database features is a key foundation for realizing intelligent data management and analysis. For example, in query optimization scenarios, query optimizers often rely on database feature representations to estimate execution costs. Furthermore, in data monitoring scenarios, database feature representations can capture distribution drift caused by the coordinated changes of multiple attributes.

[0003] In related technologies, multidimensional histograms are a common method for modeling database features. However, the space complexity of multidimensional histograms increases exponentially with the number of attributes, resulting in high storage overhead. Summary of the Invention

[0004] Therefore, it is necessary to provide a database data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can save storage overhead in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a database data processing method. The method includes:

[0006] For each data attribute in the database, based on the frequency index value of various attribute values ​​of the data attribute, the various attribute values ​​of the data attribute are respectively determined as high-frequency attribute values ​​or low-frequency attribute values;

[0007] For each high-frequency attribute value, a first type of bucket is determined based on the high-frequency attribute value;

[0008] Based on multiple low-frequency attribute values, at least one second type of bucket is determined by merging them.

[0009] Based on the first type of bucketing and the second type of bucketing, construct the histogram vector corresponding to the data attribute;

[0010] Based on the histogram vectors corresponding to each data attribute in the database, joint encoding is performed to obtain the database feature representation of the database.

[0011] Secondly, this application also provides a database data processing apparatus. The apparatus includes:

[0012] The attribute value classification module is used to determine the various attribute values ​​of the data attribute as high-frequency attribute values ​​or low-frequency attribute values ​​for each data attribute in the database, based on the frequency index values ​​of the various attribute values ​​of the data attribute.

[0013] The first type of bucketing determination module is used to determine a first type of bucket based on each high-frequency attribute value;

[0014] The second type of bucket determination module is used to determine at least one second type of bucket based on multiple low-frequency attribute values.

[0015] The histogram vector construction module is used to construct the histogram vector corresponding to the data attribute based on the first type of bucketing and the second type of bucketing;

[0016] The joint encoding module is used to perform joint encoding based on the histogram vectors corresponding to each data attribute of the data in the database, so as to obtain the database feature representation of the database.

[0017] In some embodiments, the attribute value classification module is further configured to: when the attribute value of the data attribute is a discrete attribute value, classify attribute values ​​with the same value in the set of attribute values ​​of the data attribute into the same type of attribute value to obtain multiple attribute values; determine the frequency index value of each type of attribute value respectively; determine the attribute value whose frequency index value meets the high-frequency condition as a high-frequency attribute value, and determine the attribute value whose frequency index value does not meet the high-frequency condition as a low-frequency attribute value.

[0018] In some embodiments, the attribute value classification module is further configured to: determine the number of times each attribute value appears in the sliding time window based on the attribute values ​​of the data attributes contained in the sliding time window; the sliding step size of the sliding time window is negatively correlated with the data change amplitude in the sliding time window; and determine the occurrence frequency of each attribute value based on the number of times each attribute value appears in the sliding window and the total number of attribute values ​​in the sliding time window.

[0019] In some embodiments, the sliding window is provided with counters for various attribute values, and each counter is used to count the number of times each attribute value appears within the sliding window. In the histogram vector, each first type bucket and each second type bucket corresponds to a vector component. The database data processing device further includes: a histogram update module, used to update the corresponding counter according to the target attribute value category to which the updated attribute value belongs when the sliding time window slides; redetermine the occurrence frequency of the target attribute value category according to the updated counter; and update the vector component corresponding to the target attribute value category in the histogram vector according to the redetermined occurrence frequency of the target attribute value category.

[0020] In some embodiments, the attribute value classification module is further configured to: when the attribute value of the data attribute is a continuous attribute value, divide the range of the attribute value of the data attribute into intervals to obtain multiple data intervals; determine each data interval as an attribute value; determine the frequency index value of each attribute value; determine the attribute value whose frequency index value meets the high-frequency condition as a high-frequency attribute value, and determine the attribute value whose frequency index value does not meet the high-frequency condition as a low-frequency attribute value.

[0021] In some embodiments, the attribute value classification module is further configured to: when the attribute value of the data attribute is a continuous attribute value, cluster the data of the data attribute to obtain multiple clusters; divide the multiple clusters into intervals to obtain multiple data intervals, wherein the multiple data intervals correspond one-to-one with the multiple clusters.

[0022] In some embodiments, the database data processing apparatus further includes: a cardinality estimation module, configured to acquire a query statement, perform feature processing on the query statement to obtain a query feature representation corresponding to the query statement; perform attention processing on the database feature representation based on the query feature representation to obtain an attention processing result; and determine a cardinality estimate of the query statement based on the attention processing result.

[0023] In some embodiments, the cardinality estimation module is further configured to parse the query statement into an abstract syntax tree;

[0024] The connection predicates of the abstract syntax tree are traversed, and the data attributes on both sides of the traversed equivalent connection predicates are grouped according to equivalence to obtain at least one equivalent attribute group; an equivalence class label is generated for each equivalent attribute group, and the equivalence class label is embedded in the abstract syntax tree to obtain an updated abstract syntax tree; the abstract syntax tree is input into a tree convolutional neural network to obtain the query feature representation corresponding to the query statement.

[0025] In some embodiments, the cardinality estimation module is further configured to: perform parallel projection of the query feature representation using multiple attention heads to obtain the query matrix of each attention head; perform parallel projection of the database feature representation using multiple attention heads to obtain the key matrix and value matrix of each attention head; for each attention head, calculate the correlation between the query matrix and the key matrix of the attention head, determine the attention weight matrix based on the correlation calculation result, and perform weighted summation of the value matrix of the attention head using the attention weight matrix to obtain the output result of the attention head; and fuse the output results of each attention head to obtain the attention processing result.

[0026] In some embodiments, the joint encoding module is further configured to: perform parallel projection of the input matrix composed of histogram vectors corresponding to each data attribute of the data in the database with multiple attention heads to obtain the query matrix, key matrix, and value matrix of each attention head; for each attention head, calculate the correlation between the query matrix and the key matrix of the attention head, determine the attention weight matrix based on the correlation calculation result, and perform weighted summation of the value matrix of the attention head using the attention weight matrix to obtain the output result of the attention head; and fuse the output results of each attention head to obtain the database feature representation of the database.

[0027] In some embodiments, the histogram vector includes vector components corresponding one-to-one with each first type of bucket and vector components corresponding one-to-one with each second type of bucket; the second type of bucket determination module is further configured to: obtain a preset target dimension number for the histogram vector; determine the number of buckets of the first type of bucket, and determine the target number of buckets of the second type of bucket based on the target dimension number and the number of buckets of the first type of bucket; and merge the second type of buckets with the target number of buckets determined based on multiple low-frequency attribute values.

[0028] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the aforementioned database data processing method.

[0029] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the above-described database data processing method.

[0030] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the aforementioned database data processing method.

[0031] The aforementioned database data processing methods, apparatus, computer equipment, storage media, and computer program products can construct histogram vectors for each data attribute and jointly encode the histogram vectors to obtain a database feature representation. The resulting database representation can accurately model the database feature distribution. Furthermore, in the process of constructing the histogram vector for a single data attribute, various attribute values ​​of the data attribute are respectively determined as high-frequency attribute values ​​or low-frequency attribute values. Different strategies are used to determine histogram binning for high-frequency and low-frequency attribute values. Since the types of high-frequency attribute values ​​have a significant impact on the accuracy of modeling the database feature distribution, but their number is usually small, determining histogram binning separately for high-frequency attribute values ​​can improve statistical accuracy while having a small impact on storage overhead. Conversely, the types of low-frequency attribute values ​​have a small impact on the accuracy of modeling the database feature distribution, but their number is usually large. Therefore, determining histogram binning based on multiple low-frequency attribute values ​​can significantly reduce storage overhead without affecting statistical accuracy, thereby achieving both improved accuracy of database feature representation and reduced storage overhead. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a diagram illustrating the application environment of the database data processing methods in some embodiments;

[0034] Figure 2 This is a flowchart illustrating the database data processing method in some embodiments;

[0035] Figure 3 This is a schematic diagram illustrating examples of high-frequency and low-frequency attribute values ​​in some embodiments;

[0036] Figure 4 This is an example schematic diagram illustrating the determination of buckets in some embodiments;

[0037] Figure 5 This is an example illustration of constructing histogram vectors in some embodiments;

[0038] Figure 6 This is a flowchart illustrating the cardinality estimation steps in some embodiments;

[0039] Figure 7 This is a schematic diagram illustrating examples of attention processing of database feature representations in some embodiments;

[0040] Figure 8This is a schematic diagram illustrating the database data processing method in some embodiments;

[0041] Figure 9 This is a schematic diagram illustrating the application scenarios of the database data processing method in other embodiments;

[0042] Figure 10 This is a structural block diagram of the database data processing device in some embodiments;

[0043] Figure 11 These are internal structural diagrams of the computer device in some embodiments;

[0044] Figure 12 This is an internal structural diagram of a computer device in some other embodiments. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] First, some key terms used in this application will be explained:

[0047] Data attributes are fields or dimensions that describe the characteristics or properties of a data object, used to characterize the essential features, state, or relationships of the data. Each data attribute corresponds to a column in a data table, and its value reflects the specific information of the data object regarding that characteristic. Data attributes can be divided into discrete and continuous data attributes based on whether their values ​​are continuous. Discrete attributes refer to data attributes whose values ​​are discrete, such as "quantity of goods" or "color" in a product order data table. Continuous attributes refer to data attributes whose values ​​are continuous, such as "price" in a product order data table or "temperature" in equipment data.

[0048] Frequency index value: refers to the index value that represents frequency. For example, it can be the number of times or frequency of a certain attribute value appears in the database, or it can be the ranking of the frequency or number of occurrences.

[0049] Histogram vector: Contains multiple vector components. Each first-type bucket corresponds to one vector component, and each second-type bucket corresponds to one vector component. Each vector component can be the frequency index value of its corresponding bucket record or a normalized value obtained from the frequency index value. Each data attribute corresponds to one histogram vector.

[0050] Discrete attribute values: These are attribute values ​​that are discrete. Discrete attribute values ​​can be integer values ​​or categorical attribute values. Categorical attribute values ​​refer to attribute values ​​that describe a finite, enumerable category. For example, in a product data table, the attribute value of the product color attribute is a categorical attribute value, such as red, yellow, or blue.

[0051] Continuous attribute values: These are attribute values ​​that are consecutive numerical values. For example, in a product data table, the product price attribute is a continuous attribute.

[0052] Sliding time window: refers to a fixed-length data collection interval with time as the dimension. As time goes by, it slides at a set step size to continuously capture the latest data segments, thereby focusing on recent data and filtering out interference from historical information.

[0053] Database states are the result of data characterization, represented as a set of vectors that capture the distribution characteristics of each attribute. These states can be accessed and updated efficiently to support dynamic workloads.

[0054] Cardinality estimation refers to estimating the number of tuples (i.e., cardinality c) in the result set after a given query is executed.

[0055] Dynamic workloads refer to situations where data in the database changes while queries are being executed, including operations such as inserts, deletes, and updates. These changes require cardinality estimation methods to adapt quickly.

[0056] Multi-head attention: A mechanism used in models that allows the model to focus on different parts of the input simultaneously in order to extract richer feature information.

[0057] Mean-Weighted-Squared-Error (MWSE): The loss function used to train the model, which calculates the difference between the predicted cardinality and the true cardinality, and adjusts it according to the weights.

[0058] Tree-based Convolutional Neural Network (TCNN) is a deep learning-based convolutional neural network model with a tree-like structure. By treating the input data as a tree structure, it effectively reduces the model's complexity and improves its expressive power by leveraging the characteristics of the tree structure.

[0059] The database data processing method provided in this application embodiment can be applied to, for example...Figure 1 In the application environment shown, database 102 can store the data that computer device 104 needs to process. Database 102 can be set up independently, integrated into computer device 104, or placed in the cloud or on other computer devices. Computer device 104 can perform operations such as adding, deleting, querying, and modifying data in database 102. Computer device 104 can be a terminal or a server. Terminals can be, but are not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0060] Continue to refer to Figure 1 For each data attribute in the database, the computer device 104 can identify high-frequency and low-frequency attribute values ​​from the various attribute values ​​of the data attribute based on the frequency index values ​​of the various attribute values ​​of the data attribute. For each high-frequency attribute value, a first type of bucket is determined based on the high-frequency attribute value. Based on multiple low-frequency attribute values, at least one second type of bucket is determined by merging. According to the first type of bucket and the second type of bucket, a histogram vector corresponding to the data attribute is constructed. Based on the histogram vectors corresponding to each data attribute in the database, joint encoding is performed to obtain the database feature representation of the database.

[0061] In some embodiments, such as Figure 2 As shown, a database data processing method is provided, which is executed by a computer device and includes the following steps:

[0062] Step 202: For each data attribute in the database, based on the frequency index values ​​of various attribute values ​​of the data attribute, determine the various attribute values ​​of the data attribute as high-frequency attribute values ​​or low-frequency attribute values ​​respectively.

[0063] Data attributes are fields or dimensions that describe the characteristics or properties of data objects, used to characterize the essential features, states, or relationships of the data. Each data attribute corresponds to a column in a data table, and its value reflects the specific information of the data object regarding that characteristic. Data attributes are the foundation for constructing database table structures and the core unit for data querying, analysis, and modeling. Data attributes can be divided into discrete data attributes and continuous data attributes based on whether their values ​​are continuous. Discrete attributes refer to data attributes whose values ​​are discrete, while continuous attributes refer to data attributes whose values ​​are continuous.

[0064] The attribute value of a data attribute refers to the specific value that the data attribute takes. For example, the attribute value of "Quantity of Goods" in a product order data table can be 2, 3, 4, etc. All attribute values ​​for each data attribute can be categorized into multiple attribute values ​​based on their possible values. The categorization method can be pre-defined for different types of data attributes. For example, for discrete data attributes, if the attribute value is an integer, attribute values ​​with the same value can be categorized into the same attribute value. Alternatively, the value range can be divided into multiple numerical intervals, and attribute values ​​belonging to the same numerical interval can be categorized into the same attribute value. For continuous data attributes, the value range can be divided into multiple numerical intervals, and attribute values ​​belonging to the same numerical interval can be categorized into the same attribute value.

[0065] Frequency metrics refer to metrics that characterize frequency. For example, it could be the number of times an attribute value appears in a database, or the ranking of its occurrences. High-frequency attribute values ​​are those with larger frequency metrics, while low-frequency attribute values ​​are those with smaller frequency metrics.

[0066] Specifically, for each data attribute in the database, the computer device can classify the attribute values ​​under that data attribute into multiple categories, count the frequency index value of each attribute value, and identify high-frequency and low-frequency attribute values ​​from the data of that data attribute based on the frequency index value of each attribute value.

[0067] For example, see reference. Figure 3 The diagram illustrates examples of high-frequency and low-frequency attribute values ​​in some embodiments. Suppose there is an e-commerce order database where the "product category" attribute value includes six types: electronic products, clothing, home furnishings, books, cosmetics, and sports equipment. The frequency of each attribute value is 10000, 8000, 6000, 100, 80, and 120, respectively. Then, electronic products, clothing, and home furnishings can be identified as high-frequency attribute values, while books, cosmetics, and sports equipment can be identified as low-frequency attribute values.

[0068] Step 204: For each high-frequency attribute value, determine a first type of bucket based on the high-frequency attribute value.

[0069] Step 206: Based on multiple low-frequency attribute values, at least one second-type bucket is determined by merging.

[0070] In a histogram, a bucket is the basic statistical unit for data distribution. A histogram can include multiple buckets, and a bucket can represent one or more attribute values. The height of a bucket represents the frequency index value of the attribute value it represents. Therefore, buckets can record frequency index values. In this embodiment, the buckets include two types: a first type of bucket for representing high-frequency attribute values ​​and a second type of bucket for representing low-frequency attribute values. A first type of bucket represents one high-frequency attribute value, and a second type of bucket represents multiple low-frequency attribute values.

[0071] Specifically, for each high-frequency attribute value, the computer device can determine a first type of bucket based on the high-frequency attribute value. The first type of bucket can record the frequency index value of the high-frequency attribute value. For all low-frequency attribute values, the computer device can merge multiple low-frequency attribute values ​​to jointly determine a second type of bucket. The second type of bucket can record the sum of the frequency index values ​​of the multiple low-frequency attribute values, thereby reducing the number of buckets, reducing storage overhead, and improving the generalization of the histogram. The first type of buckets and the second type of buckets together form the histogram corresponding to the data attribute.

[0072] For example, see reference. Figure 4 The following is an example illustration of how bins are determined in some embodiments. Continuing with the e-commerce order database example above, for the three attribute values ​​of electronic products, clothing, and home furnishings, three first-type bins are determined respectively: bin 1, bin 2, and bin 3. Bin 1 records the occurrence count of "electronic products" (10,000), bin 2 records the occurrence count of "clothing" (8,000), and bin 3 records the occurrence count of "home furnishings" (6,000). The three attribute values ​​of books, cosmetics, and sports equipment are merged to determine a second-type bin, bin 4. Bin 4 records the occurrence count of these three attribute values ​​(books, cosmetics, and sports equipment) (300). It should be noted that... Figure 4 For illustrative purposes only, the bucket shape is not intended to precisely represent the numerical value it records.

[0073] For example, when the data attribute is a numeric attribute, both the first-type bucket and the second-type bucket can record statistical quantities of the data within the bucket, such as any one of the following: sum, average, maximum, minimum, etc. Here, the data within a bucket refers to all data under the attribute value category used to determine that bucket. For example, in a product order table, for the data attribute "product quantity," a first-type bucket created for one attribute value "10" refers to all orders with a product quantity of 10. This first-type bucket can also record the total quantity of these product orders.

[0074] For example, when determining the second type of bucket, the computer device can divide the low-frequency attribute values ​​into a fixed number of buckets. Assuming that the low-frequency attribute values ​​include 6 attribute values, then every 3 attribute values ​​can be combined to determine a second type of bucket.

[0075] Step 208: Construct histogram vectors corresponding to data attributes based on the first type of bucketing and the second type of bucketing.

[0076] The histogram vector contains multiple vector components. Each first-type bucket corresponds to one vector component, and each second-type bucket corresponds to one vector component. Each vector component can be the frequency index value of its corresponding bucket record or a normalized value obtained from the frequency index value. Each data attribute corresponds to one histogram vector.

[0077] Specifically, the computer device can determine the corresponding vector component based on the frequency index value of each first type of bucket record, and then assemble the vector components into a histogram vector corresponding to the data attribute in a preset order.

[0078] For example, the vector dimension of the histogram vector can be predetermined. When the total number of buckets is less than the predetermined vector dimension, the vector values ​​of the insufficient part can be replaced with 0.

[0079] For example, see reference. Figure 5 The following is an example diagram illustrating the construction of histogram vectors in some embodiments. Continuing with the example of the product order database mentioned above, after determining three first-type bins, namely bin 1, bin 2, and bin 3, and one second-type bin, namely bin 4, the corresponding vector components 10000, 8000, and 6000 are determined according to the frequency index values ​​recorded by bins 1, 2, and 3, respectively. The corresponding vector component 300 is determined according to the frequency index value recorded by bin 4. Finally, the histogram vector (10000, 8000, 6000, 300) is formed.

[0080] Step 210: Based on the histogram vectors corresponding to each data attribute in the database, perform joint encoding to obtain the database feature representation of the database.

[0081] Joint encoding is the process of fusing the histogram vectors through encoding, while database feature representation is used to represent the data distribution of the database.

[0082] Specifically, computer equipment can perform joint encoding based on the histogram vectors corresponding to each data attribute in the database to model the data distribution of the database and obtain a database feature representation.

[0083] For example, a computer device can fuse the histogram vectors corresponding to each data attribute to obtain a database feature representation of the database. In other embodiments, the computer device can also use an attention mechanism to jointly encode the histogram vectors corresponding to each data attribute to obtain a database feature representation of the database.

[0084] In the aforementioned database data processing method, a histogram vector can be constructed for each data attribute, and the database feature representation can be obtained by jointly encoding the histogram vectors. The resulting database representation can accurately model the database feature distribution. In the process of constructing the histogram vector for a single data attribute, various attribute values ​​of the data attribute are respectively determined as high-frequency attribute values ​​or low-frequency attribute values. Different strategies are used to determine the histogram bins for high-frequency and low-frequency attribute values. Since the types of high-frequency attribute values ​​have a significant impact on the accuracy of modeling the database feature distribution, but their number is usually small, determining the histogram bins separately for high-frequency attribute values ​​improves statistical accuracy while having a small impact on storage overhead. The types of low-frequency attribute values ​​have a small impact on the accuracy of modeling the database feature distribution, but their number is usually large. Therefore, determining the histogram bins based on multiple low-frequency attribute values ​​can significantly reduce storage overhead without affecting statistical accuracy. This method is suitable for long-tailed distributed data, thereby achieving both improved accuracy of database feature representation and reduced storage overhead.

[0085] In some embodiments, determining various attribute values ​​of a data attribute as high-frequency or low-frequency attribute values ​​based on the frequency index values ​​of various attribute values ​​of the data attribute includes: when the attribute values ​​of the data attribute are discrete attribute values, classifying attribute values ​​with the same value in the set of attribute values ​​of the data attribute into the same type of attribute value to obtain multiple attribute values; determining the frequency index value of each attribute value; determining attribute values ​​whose frequency index values ​​meet the high-frequency condition as high-frequency attribute values, and determining attribute values ​​whose frequency index values ​​do not meet the high-frequency condition as low-frequency attribute values.

[0086] Discrete attribute values ​​refer to attribute values ​​that are discrete in nature. These can be integer values ​​or categorical attribute values. Categorical attribute values ​​describe a finite, enumerable category. For example, in a product data table, the attribute value for the product color is a categorical attribute value, such as red, yellow, or blue. High-frequency conditions refer to the conditions that determine an attribute value as a high-frequency attribute value. For example, a high-frequency condition could be an indicator value greater than a preset threshold, a frequency indicator value ranked before a preset sorting threshold, or the difference between a frequency indicator value and the maximum frequency indicator value being less than a preset threshold. It could also be a frequency indicator value's percentage being greater than a preset proportion threshold, etc. Specific settings can be configured as needed.

[0087] Specifically, when the attribute value of a data attribute is a discrete attribute value, the computer device can classify attribute values ​​with the same value in the set of attribute values ​​of the data attribute into the same type of attribute value, thereby obtaining multiple attribute values. For example, when the attribute value of a data attribute is an integer value, each integer value can be classified into one type of attribute value. When the attribute value of a data attribute is a categorical attribute value, each categorical attribute value can be classified into one type of attribute value.

[0088] Furthermore, the computer device can determine the frequency index value of each attribute value separately. For example, it can determine the number of times or frequency of occurrence of each attribute value. It should be noted that the frequency index values ​​of various attribute values ​​adopt the same type of frequency index value. For example, the frequency index values ​​of various attribute values ​​are both the number of occurrences or both the frequency of occurrence. Furthermore, for each attribute value, the computer device determines whether the frequency index value of that attribute value meets the high-frequency condition. If it does, the attribute value is determined to be a high-frequency attribute value. If it does not, the attribute value is determined to be a low-frequency attribute value.

[0089] In the above embodiments, by dividing attribute values ​​with the same value in the set of attribute values ​​of data attributes into the same type of attribute value, multiple attribute values ​​can be obtained. This allows for a more accurate division of high-frequency and low-frequency attribute values, thereby further improving the accuracy of database feature representation.

[0090] In some embodiments, determining the frequency index values ​​of each attribute value includes: determining the number of times each attribute value appears in the sliding time window based on the attribute values ​​of the data attributes contained in the sliding time window; the sliding step size of the sliding time window is negatively correlated with the data change amplitude in the sliding time window; and determining the frequency of each attribute value based on the number of times each attribute value appears in the sliding window and the total number of attribute values ​​in the sliding time window.

[0091] The sliding time window refers to a fixed-length data collection interval based on time. It slides along a set step size over time, continuously capturing the latest data segments, thus focusing on recent data and filtering out historical information. The attribute values ​​of a data attribute within the sliding time window refer to all possible values ​​for that attribute. For example, within a sliding time window of 10:00-10:10, the attribute values ​​for the "Order Status" data attribute include: Paid, Pending Shipment, Cancelled, Pending Shipment, Paid, etc. The sliding step size refers to the time interval at which the sliding time window moves forward each time, such as 1 minute or 5 minutes. The sliding step size is negatively correlated with the magnitude of data change within the sliding time window; that is, the sliding step size can change dynamically. The greater the magnitude of data change within the sliding time window, the smaller the step size (e.g., shortening from 10 minutes to 1 minute), and vice versa, to balance computational efficiency and data sensitivity. The frequency of occurrence refers to the ratio of the number of times a certain attribute value appears to the total number of attribute values ​​within the window, reflecting the proportion of that value's distribution within the window. The calculation formula can be: Frequency of occurrence = Number of times a certain attribute value appears / Total number of attribute values ​​within the sliding time window.

[0092] Specifically, in this embodiment, the computer device can dynamically collect data attribute values ​​through a sliding time window mechanism and adaptively adjust the sliding step size based on the magnitude of data changes within the window to achieve accurate calculation of attribute value frequency. Specifically, the computer device can traverse the data within the sliding time window, accumulate and count identical attribute values ​​to determine the frequency of occurrence of various attribute values ​​within the sliding time window, and then calculate the frequency of occurrence by combining this with the total number of attribute values. The sliding step size is dynamically adjusted according to the magnitude of data changes within the window; that is, the greater the data fluctuation, the smaller the step size, and vice versa. This effectively balances the real-time nature of data processing with computational resource overhead, making it particularly suitable for scenarios requiring real-time monitoring of data distribution changes, such as e-commerce order monitoring and financial transaction analysis, significantly improving data processing efficiency and decision-making accuracy.

[0093] For example, suppose we need to determine the frequency of occurrence of various attribute values ​​for the data attribute "order status". We set the sliding window length to 1 hour and the initial step size to 15 minutes. Within the 09:00-10:00 window, the set of "order status" attribute values ​​is {Paid: 500 times, Pending Shipment: 300 times, Cancelled: 100 times}, with a total of 900 attribute values. If the data fluctuates drastically within the 10:00-11:00 window (for example, the number of "Cancelled" orders suddenly increases to 500), then according to the negative correlation rule, the sliding step size can be shortened to 5 minutes to capture changes more frequently. Within the 09:00-10:00 window, the frequency of occurrence of the "Paid" status is approximately 50 / 900 ≈ 0.56.

[0094] In the above embodiments, the latest data is continuously collected by a sliding time window to ensure that the analysis results always reflect recent trends. By adjusting the step size with the negative correlation between the step size and the magnitude of data changes, the consumption of computing resources is reduced when the data is stable and a timely response is made when the data changes abruptly. This balances efficiency and accuracy and avoids the problems of "insufficient sampling when the data changes drastically" or "over-computation when the data is stable" in the fixed step size scheme. This reduces the system load and is especially suitable for high-concurrency, large-data-volume business environments.

[0095] In some embodiments, the sliding window is provided with counters for various attribute values, and each counter is used to count the number of times each attribute value appears within the sliding window. In the histogram vector, each first-type bucket and each second-type bucket corresponds to a vector component. The database data processing method of this application further includes:

[0096] As the sliding time window slides, the corresponding counter is updated according to the target attribute value category to which the updated attribute value belongs within the sliding time window; the occurrence frequency of the target attribute value category is re-determined based on the updated counter; and the vector components corresponding to the target attribute value category in the histogram vector are updated according to the re-determined occurrence frequency of the target attribute value category.

[0097] Here, "attribute value type" refers to the different value categories of a data attribute. For example, the "order status" attribute includes categories such as "paid," "pending shipment," and "cancelled." "Counter" is a variable used to record the number of times a specific attribute value type appears within the sliding time window. It updates the frequency statistics of attribute value types in real time through incrementing or decrementing operations. The attribute values ​​updated within the sliding time window include attribute values ​​entering the sliding time window and attribute values ​​moving out of the sliding time window.

[0098] Specifically, in this embodiment, a linkage mechanism between attribute value type counters and histogram vectors is constructed to achieve real-time monitoring of data distribution characteristics. Specifically, during the sliding time window, the computer device can update the corresponding counters based on newly entering or leaving attribute values. Then, based on the updated counter values, the frequency of occurrence of attribute value types is recalculated. Finally, the new frequencies are synchronized to the corresponding components of the histogram vector, completing the instant refresh of the visualized data. This local update strategy significantly reduces computational complexity. Combined with the timeliness advantage of the sliding window, it effectively solves the problem that traditional static statistical methods cannot adapt to dynamic data changes. It is well-suited for business scenarios with high real-time requirements, significantly improving data analysis efficiency and decision-making accuracy.

[0099] For example, suppose we need to update the histogram vector for the "order status" attribute, and the initial settings are: sliding time window: 1 hour; counter: paid: 0, pending shipment: 0, cancelled: 0; histogram vector: [0,0,0];

[0100] If 100 new order records are added within the first time window, including 60 "Paid", 30 "Pending Shipment", and 10 "Cancelled", then the counters are updated as follows: Paid: 60, Pending Shipment: 30, Cancelled: 10. The frequencies are calculated as follows: "Paid" frequency = 60 / 100 = 0.6; "Pending Shipment" frequency = 30 / 100 = 0.3; "Cancelled" frequency = 10 / 100 = 0.1. The histogram vector is then updated to [0.6, 0.3, 0.1].

[0101] After the window slides, 10 new "Paid" orders enter, and 5 "Pending Shipment" orders leave. Counter update: Paid: 60 + 10 = 70, Pending Shipment: 30 - 5 = 25, Cancelled: 10. Recalculate the frequency and update the vector: [70 / 100, 25 / 100, 10 / 100] = [0.7, 0.25, 0.1].

[0102] In the above embodiments, by linking the sliding time window with the counter, millisecond-level response of attribute value frequency can be achieved, accurately capturing instantaneous changes in data distribution. Furthermore, since the counter and frequency can be adjusted only for the updated attribute values ​​within the window, full data recalculation is avoided, significantly reducing computational overhead and making it suitable for high-concurrency data scenarios.

[0103] In some embodiments, based on the frequency index values ​​of various attribute values ​​of a data attribute, the various attribute values ​​of the data attribute are respectively determined as high-frequency attribute values ​​or low-frequency attribute values, including:

[0104] When the attribute value of a data attribute is a continuous attribute value, the range of the attribute value is divided into intervals to obtain multiple data intervals; each data interval is determined as an attribute value; the frequency index value of each attribute value is determined; the attribute value whose frequency index value meets the high-frequency condition is determined as a high-frequency attribute value, and the attribute value whose frequency index value does not meet the high-frequency condition is determined as a low-frequency attribute value.

[0105] Continuous attribute values ​​refer to attributes whose values ​​are consecutive numerical values. For example, in a product data table, the product price attribute is a continuous attribute. The range of attribute values ​​refers to the range of all possible values ​​within that attribute. In practical implementation, the minimum valid value (MinValid Value) and maximum valid value (Max Valid Value) of an attribute can be identified through mathematical statistics, business rules, or algorithmic logic, forming a closed interval [Min, Max]. Taking the "logistics timeliness" attribute as an example, a reasonable range is set based on the transportation method, such as [1, 15] days for regular express delivery. Interval partitioning refers to dividing the range of continuous attribute values ​​into multiple non-overlapping sub-intervals, each corresponding to a discrete category. Through interval partitioning, continuous data can be transformed into a discrete form.

[0106] Specifically, when the attribute value of a data attribute is a continuous attribute value, the computer device can divide the range of the attribute value into intervals to obtain multiple data intervals. For example, the value range can be divided into multiple data intervals of equal width, and the span of each data interval can be Width=(Max−Min) / N (N is the number of intervals). For example, the value range of the "consumption amount" attribute is [18,60], which can be divided into N=4 intervals, such as: [18,27), [27,36), [36,45), [45,60].

[0107] Furthermore, the computer device can determine the frequency index value of each attribute value separately. For example, it can determine the number of times or frequency of occurrence of each attribute value. It should be noted that the frequency index values ​​of various attribute values ​​adopt the same type of frequency index value. For example, the frequency index values ​​of various attribute values ​​are both the number of occurrences or both the frequency of occurrence. Furthermore, for each attribute value, the computer device determines whether the frequency index value of that attribute value meets the high-frequency condition. If it does, the attribute value is determined to be a high-frequency attribute value. If it does not, the attribute value is determined to be a low-frequency attribute value.

[0108] In the above embodiments, by dividing the range of attribute values ​​of data attributes into intervals, multiple data intervals are obtained, and each data interval is determined as an attribute value. This allows for a more accurate division of high-frequency and low-frequency attribute values, thereby further improving the accuracy of database feature representation.

[0109] In some embodiments, when the attribute value of a data attribute is a continuous attribute value, the range of the attribute value of the data attribute is divided into intervals to obtain multiple data intervals, including: when the attribute value of a data attribute is a continuous attribute value, clustering the data of the data attribute to obtain multiple clusters; dividing the data into intervals according to the multiple clusters to obtain multiple data intervals, with each data interval corresponding to one of the multiple clusters.

[0110] Clustering refers to the unsupervised learning process of dividing data objects into multiple "clusters." Data within the same cluster has high similarity, while differences between different clusters are significant. Clusters are the grouping results output by clustering algorithms, and each cluster represents a category or distribution feature of the data.

[0111] For each data attribute, if the data attribute is a continuous attribute value, the computer device can use a clustering algorithm to cluster all attribute values ​​of the data attribute, generating multiple clusters with similar characteristics. For example, any clustering algorithm such as K-Means, DBSCAN (Density-Based Spatial Clustering of Applications with Noise), or hierarchical clustering can be used for clustering. Then, based on the value range of each cluster, the attribute values ​​are divided into corresponding data intervals, so that each interval accurately maps the inherent distribution characteristics of the data.

[0112] For example, suppose the K-Means algorithm is used to aggregate consumption data into three clusters: Cluster 1: contains small consumption data (e.g., 10-50 yuan); Cluster 2: contains medium consumption data (e.g., 50-200 yuan); Cluster 3: contains large consumption data (e.g., over 200 yuan). Then, based on the clustering results, the consumption amount can be divided into three data intervals: Interval 1: [10 yuan, 50 yuan) → corresponding to Cluster 1; Interval 2: [50 yuan, 200 yuan) → corresponding to Cluster 2; Interval 3: [200 yuan, +∞) → corresponding to Cluster 3.

[0113] In the above embodiments, the adaptive determination of interval boundaries is achieved through data-driven clustering algorithms. The interval boundaries can be automatically determined based on the inherent distribution of the data, which can significantly improve the accuracy of data discretization and the efficiency of analysis.

[0114] The above embodiments describe the process of generating database feature representations using the database data processing method of this application. The specific application of the generated database feature representations will be introduced below in conjunction with the cardinality estimation scenario.

[0115] In some embodiments, the database data processing method of this application further includes a cardinality estimation step, referring to... Figure 6The diagram below illustrates the cardinality estimation process in some embodiments, specifically including the following steps:

[0116] Step 602: Obtain the query statement, perform feature processing on the query statement, and obtain the query feature representation corresponding to the query statement.

[0117] In this context, the instructions sent by users or applications to the database system for retrieving, inserting, updating, or deleting data are typically presented in SQL (Structured Query Language) form. Feature processing refers to extracting key information with semantic or structural features from the query statement, converting unstructured text into structured feature vectors. Query feature representation refers to the abstract representation of the features of the query statement in the form of numerical vectors, symbol sequences, or graph structures.

[0118] Specifically, after obtaining a query statement, the computer device can parse the query statement into an abstract syntax tree, extract features from the abstract syntax tree and encode them to obtain a query feature representation of the query statement.

[0119] Step 604: Perform attention processing on the database feature representation based on the query feature representation to obtain the attention processing result.

[0120] Specifically, the computer device can calculate the correlation between the query feature representation and the database feature representation to obtain the attention weight, and then perform a weighted summation of the database feature representation based on the attention weight to obtain the attention processing result.

[0121] Step 606: Determine the cardinality estimate of the query statement based on the attention processing results.

[0122] Cardinality estimation refers to estimating the number of tuples (i.e., cardinality c) in the result set after a given query is executed.

[0123] Specifically, after obtaining the attention processing results, the computer device can map the attention processing results through linear regression to obtain the cardinality estimate of the query statement, thereby achieving accurate prediction of the size of the query result set through dynamic weighted association between query features and database features.

[0124] For example, a computer device can first perform lexical and syntactic analysis on the input query statement to generate an abstract syntax tree, from which syntactic features such as table operations, conditional filtering, and aggregation sorting are extracted; then, the query semantics are parsed in conjunction with database metadata to identify deep features such as business themes and data ranges; finally, multi-dimensional features are converted into a unified vector representation through techniques such as one-hot encoding and word embedding.

[0125] In the above embodiments, since attention processing can be performed on the database feature representation based on the query feature representation to obtain the attention processing result, and the cardinality estimate of the query statement can be determined based on the attention processing result, the dependence of traditional cardinality estimation methods on static statistical information can be broken. Through the data-driven attention mechanism, the estimation accuracy in complex query scenarios can be significantly improved by adapting to changes in query semantics and data distribution.

[0126] In some embodiments, the query statement is characterized to obtain a query feature representation corresponding to the query statement, including:

[0127] The query statement is parsed into an abstract syntax tree (AST). The connection predicates of the AST are traversed, and the data attributes on both sides of the traversed equivalent connection predicates are grouped according to equivalence to obtain at least one equivalent attribute group. An equivalence class label is generated for each equivalent attribute group, and the equivalence class label is embedded in the AST to obtain an updated AST. The AST is then input into a tree convolutional neural network to obtain the query feature representation corresponding to the query statement.

[0128] The Abstract Syntax Tree (AST) is a tree structure generated after parsing a query statement. Nodes represent operators (such as SELECT and JOIN), table names, column names, or predicates, while edges represent hierarchical relationships. Join predicates are conditional expressions used in table join operations. They can exist as ON or WHERE clauses and are used to determine the matching rules for the joined rows. Join predicates include equivalent join predicates and non-equivalent join predicates. Equivalent join predicates are conditional expressions connected using the equality comparison operator (=) to determine whether two attribute values ​​or constants are equal. The core characteristic of equivalent join predicates is an "equality relationship," indicating that the attributes or values ​​on both sides are logically completely equivalent. For example, in the query statement SELECT * FROM orders WHERE user_id=123, user_id=123 is an equivalent join predicate used to filter out order data where user_id equals 123. In table join scenarios, `JOIN orders ON orders.user_id=users.user_id` is an equivalence join predicate in `JOIN orders ON orders.user_id=users.user_id`, used to define an equivalence join between two tables via the `user_id` field. Non-equivalence join predicates are conditional expressions joined using non-equivalence comparison operators (such as >, <, >=, <=, !=, LIKE, BETWEEN, etc.), used to describe the order, range, or fuzzy matching relationships between attribute values. For example, in `SELECT * FROM products WHERE price > 100`, `price > 100` is a non-equivalence join predicate, filtering products with prices greater than 100; in `SELECT * FROM logs WHERE timestamp BETWEEN '2025-01-01' AND '2025-12-31'`, the `BETWEEN` expression is a non-equivalence join predicate used to limit the time range.

[0129] Grouping by equivalence refers to grouping attributes associated through equivalence join predicates into the same group based on the transitivity principle. For example, if there are two join predicates: users.id=orders.user_id and orders.user_id=reviews.user_id, then the equivalence attribute group {users.id, orders.user_id, reviews.user_id} can be generated. An equivalence class label is a unique identifier assigned to each equivalence join predicate group, used to replace the original attribute references and uniformly represent the semantically equivalent set of attributes. Equivalence class labels can consist of any one or more of letters, numbers, or characters. For example, for the equivalence join predicate group {users.id, orders.user_id, reviews.user_id}, the equivalence class label "EC1" can be generated. Tree Convolutional Neural Networks (Tree CNNs) are neural networks specifically designed to process tree-structured data, generating structured representations by recursively aggregating features of nodes and their subtrees.

[0130] Specifically, computer devices can decompose query statements into tokens, such as SELECT, FROM, WHERE, table names, column names, etc., and then construct an Abstract Syntax Tree (AST) according to SQL syntax rules. The node types include operator nodes (such as JOIN, WHERE), relation nodes (such as tables, subqueries), and expression nodes (such as col1=col2). The tree structure represents the nested level of operations through parent-child relationships. For example, the FROM subtree contains JOIN nodes, and the JOIN node contains the ON clause. The computer device can further traverse the AST, extract all equivalent join predicates from the join predicates of the ON clause and WHERE clause, group the data attributes on both sides of the equivalent join predicates according to equivalence, obtain at least one equivalent attribute group, and the computer device further generates an equivalence class label for each equivalent attribute group and records the member table. For example, if A.id=B.id and B.id=C.id, then {A.id,B.id,C.id} belongs to the same equivalent attribute group. An equivalence class label, such as EquivClass1, is assigned to this equivalent attribute group, and the member list [A.id,B.id,C.id] is recorded.

[0131] The computer device further replaces the original join conditions in the AST with equivalence class references and adds equivalence class information to the AST nodes, thereby embedding equivalence class tags and obtaining an updated abstract syntax tree. For example, A.id=B.id can be replaced with the equivalence class reference EquivClass1, and equivalence class information can be added to the ON clause node as follows:

[0132] ON clause node:{

[0133] type: "EquivClassRef",

[0134] class_id: "EquivClass1",

[0135] members: ["A.id", "B.id", "C.id"]

[0136] }

[0137] Furthermore, the computer device can input the abstract syntax tree into a tree convolutional neural network, where the tree structure is transformed into a linear sequence, for example, by generating a sequence of nodes through depth-first traversal. Each element in the sequence is an operator, relation, or equivalence class reference. A fixed-dimensional feature vector can be generated for each node. For example, for operator nodes, predefined embedding vectors are used, such as JOIN → [0.2, 0.5, ...]. For equivalence class nodes, features such as membership count, selectivity estimation, and connection type are encoded. Long-distance dependencies in the sequence are captured using convolutional kernels with different dilation rates. Then, residual connections are used to mitigate the gradient vanishing problem in deep networks, preserving the original features. Finally, pooling operations compress the variable-length sequence into a fixed-length feature vector, thereby obtaining the query feature representation.

[0138] In the above embodiments, multiple equivalence connection predicates are merged into a single equivalence class to reduce the complexity of the AST. Equivalence class labels are embedded in the abstract syntax tree to obtain an updated abstract syntax tree, enabling the model to recognize queries that are semantically equivalent but syntactically different. Furthermore, the hierarchical structure of the AST is directly processed through a tree convolutional neural network, which can preserve the logical nesting relationship of the queries, thereby obtaining a more accurate representation of query features.

[0139] In some embodiments, attention processing is performed on the database feature representation based on the query feature representation to obtain the attention processing result, including:

[0140] The query feature representation is projected in parallel by multiple attention heads to obtain the query matrix of each attention head; the database feature representation is also projected in parallel by multiple attention heads to obtain the key matrix and value matrix of each attention head; for each attention head, the correlation between the query matrix and the key matrix of the attention head is calculated, and the attention weight matrix is ​​determined based on the correlation calculation result; the value matrix of the attention head is weighted and summed using the attention weight matrix to obtain the output result of the attention head; the output results of each attention head are fused to obtain the attention processing result.

[0141] In deep learning, the attention head is the fundamental unit of the attention mechanism. Essentially, it's a parallel feature analysis channel that maps input data to subspaces using independent parameters, capturing relationships within the data from different perspectives. Multi-attention head parallel projection refers to the process of mapping input features to multiple subspaces through multiple linear transformation matrices, with each subspace corresponding to one attention head.

[0142] The query feature matrix is ​​a projection of the query feature representation, used to measure its relevance to database features. The key matrix is ​​a projection of the database feature representation, used to calculate its similarity to the query matrix. The value matrix is ​​a projection of the database feature representation, used to store information that needs to be focused on. The attention weight matrix is ​​a weight calculated using the similarity between the query matrix and the key matrix, used to perform weighted aggregation on the value matrix.

[0143] Specifically, the computer device can perform parallel projection of the query feature representation using multiple attention heads to obtain the query matrix of each attention head, and perform parallel projection of the database feature representation using multiple attention heads to obtain the key matrix and value matrix of each attention head. Then, based on the query matrix, key matrix, and value matrix, cross-attention calculation can be performed on each attention head separately. Specifically, for each attention head, the correlation between the query matrix and key matrix of that attention head can be calculated, and the attention weight matrix can be determined based on the correlation calculation result. The value matrix of the attention head is then weighted and summed using the attention weight matrix to obtain the output result of that attention head. The computer device can further concatenate the output results of each attention head and perform a linear transformation to obtain the attention processing result.

[0144] For example, see reference. Figure 7 This is an example illustration of attention processing on database feature representations in some embodiments. Assuming there are 2 attention heads, after projecting the query feature representations in parallel with multiple attention heads, we can obtain the query matrix Q1 of attention head 1 and the query matrix Q2 of attention head 2. After projecting the database feature representations in parallel with multiple attention heads, we can obtain the key matrix K1 and value matrix V1 of attention head 1, and the key matrix K2 and value matrix V2 of attention head 2. For attention head 1, the correlation between the query matrix Q1 and the key matrix K1 is calculated, and the attention weight matrix is ​​determined based on the correlation calculation result. The value matrix V1 is weighted and summed using the attention weight matrix to obtain the output result of attention head 1. For attention head 2, the correlation between the query matrix Q2 and the key matrix K2 is calculated, and the attention weight matrix is ​​determined based on the correlation calculation result. The value matrix V2 is weighted and summed using the attention weight matrix to obtain the output result of attention head 2. Finally, the output results of attention head 1 and attention head 2 are fused to obtain the attention processing result.

[0145] In the above embodiments, the multi-head attention mechanism is used to capture the multi-dimensional correlation between the query and database statistics in parallel, which enhances the model's ability to selectively focus on database statistics, thereby significantly improving the accuracy of cardinality estimation results.

[0146] In some exemplary embodiments, joint encoding is performed based on the histogram vectors corresponding to each data attribute in the database to obtain a database feature representation, including:

[0147] The input matrix, composed of histogram vectors corresponding to each data attribute in the database, is subjected to parallel projection using multiple attention heads to obtain the query matrix, key matrix, and value matrix of each attention head. For each attention head, the correlation between the query matrix and the key matrix is ​​calculated, and the attention weight matrix is ​​determined based on the correlation calculation results. The value matrix of the attention head is then weighted and summed using the attention weight matrix to obtain the output result of the attention head. The output results of each attention head are then fused to obtain the database feature representation of the database.

[0148] Specifically, the computer device can construct a database state set X={x1,x2,…,xT} from the histogram vectors corresponding to each data attribute in the database, where xi is the histogram vector of the i-th data attribute and T is the total number of attributes. The database state set is represented as a matrix X, which serves as the input to the self-attention mechanism. Based on this input matrix, multi-head self-attention processing is then performed. Specifically, the computer device can perform parallel projection of the input matrix using multiple attention heads to obtain the query matrix, key matrix, and value matrix of each attention head. For each attention head, the correlation between the query matrix and the key matrix of that attention head is calculated. The attention weight matrix is ​​determined based on the correlation calculation result. The value matrix of the attention head is weighted and summed using the attention weight matrix to obtain the output result of that attention head. The computer device can further concatenate the output results of each attention head and perform a linear transformation to obtain the database feature representation of the database.

[0149] In the above embodiments, by jointly encoding the histogram vectors of each data attribute through a self-attention mechanism, the implicit joint distribution information between attributes can be captured, improving the accuracy of the obtained database feature representation. Furthermore, since attention processing is performed under multiple attention heads, selective attention can be paid to the correlation information between attributes from multiple dimensions, further improving the accuracy of the database feature representation.

[0150] In some embodiments, the histogram vector of each attribute data includes vector components corresponding to each first type of bucket and vector components corresponding to each second type of bucket. Based on multiple low-frequency attribute values, at least one second type of bucket is determined by merging these components, including:

[0151] Obtain the target number of dimensions for the histogram vector; determine the number of buckets for the first type of bucketing, and based on the target number of dimensions and the number of buckets for the first type of bucketing, determine the target number of buckets for the second type of bucketing; based on multiple low-frequency attribute values, merge the second type of buckets whose target number of buckets has been determined.

[0152] The target dimension refers to the fixed number of dimensions set for the histogram vector. The target dimension can be set as needed, for example, the target dimension can be 40.

[0153] In this embodiment, a fixed dimension is set for each histogram vector. When determining the number of bins for the second type of bins, the target number of dimensions for the histogram vector is first obtained, and then the number of bins for the first type of bins is determined. Further, the target number of dimensions is subtracted from the number of bins for the first type of bins to obtain the target number of bins for the second type of bins. Then, based on multiple low-frequency attribute values, the second type of bins with the target number of bins are merged to determine the desired number of bins. It is understood that in some other embodiments, the target number of bins for the second type of bins may also be less than the value obtained by subtracting the number of bins for the first type of bins from the target number of dimensions. In this case, when constructing the histogram vector, dimensions without corresponding bins can be padded with zeros. In other words, the target number of bins for the second type of bins is less than or equal to the value obtained by subtracting the number of bins for the first type of bins from the target number of dimensions.

[0154] For example, suppose the target dimension is 40. For a certain data attribute, 26 high-frequency attribute values ​​are identified from the various attribute values ​​of the data attribute. Thus, the number of first-type buckets is 26. Then, the computer device can calculate the target dimension by subtracting the number of first-type buckets (40-26=14). Therefore, the target number of second-type buckets is less than or equal to 14.

[0155] In the above embodiments, since the number of buckets for the first type and the second type can be determined based on the target dimension preset in the histogram vector, it can be ensured that the histogram vector is set to a fixed dimension. On the one hand, this avoids the histogram vector dimension from being too large, saving storage overhead. On the other hand, the histogram vector dimension is always fixed, which ensures that the feature dimension input to the neural network is stable during the joint encoding process, without the need to readjust the model structure due to data changes. In addition, when the data changes (such as inserting a record), only the histogram vector of the corresponding attribute needs to be updated, which greatly reduces the time complexity and supports real-time incremental updates.

[0156] In some specific embodiments, such as Figure 8 As shown, a database data processing method is provided, reference Figure 8Computer devices can perform cardinality estimation for a given SQL query, specifically:

[0157] First, computer equipment can perform histogram feature generation on each data attribute in the database to construct histogram vectors.

[0158] Specifically, for each data attribute, when the attribute value of the data attribute is a discrete attribute value, the computer device can divide the attribute values ​​with the same value in the set of attribute values ​​of the data attribute into the same type of attribute value, thereby obtaining multiple attribute values. Based on the attribute values ​​of the data attribute contained in the sliding time window, the computer device determines the number of times each attribute value appears in the sliding time window. The sliding step size of the sliding time window is negatively correlated with the data change amplitude within the sliding time window. The computer device can further determine the occurrence frequency of each attribute value based on the number of times each attribute value appears in the sliding window and the total number of attribute values ​​in the sliding time window. Attribute values ​​whose frequency index values ​​meet the high-frequency condition are determined as high-frequency attribute values, and attribute values ​​whose frequency index values ​​do not meet the high-frequency condition are determined as low-frequency attribute values.

[0159] When the attribute value of a data attribute is a continuous attribute value, the computer device can cluster the data attribute data to obtain multiple clusters; divide the multiple clusters into intervals to obtain multiple data intervals, with each data interval corresponding to one of the multiple clusters; determine each data interval as an attribute value; determine the frequency index value of each attribute value; determine the attribute value whose frequency index value meets the high-frequency condition as high-frequency attribute value, and determine the attribute value whose frequency index value does not meet the high-frequency condition as low-frequency attribute value.

[0160] Furthermore, for each high-frequency attribute value, the computer device can determine a first-type bucket based on the high-frequency attribute value, and combine and determine at least one second-type bucket based on multiple low-frequency attribute values. Specifically, combining and determining at least one second-type bucket based on multiple low-frequency attribute values ​​includes: obtaining a preset target dimension number for the histogram vector; determining the number of buckets for the first-type bucket, and determining the target number of buckets for the second-type bucket based on the target dimension number and the number of buckets for the first-type bucket; and combining and determining the target number of second-type buckets based on multiple low-frequency attribute values.

[0161] Furthermore, the computer device can construct histogram vectors corresponding to data attributes based on the first type of bucketing and the second type of bucketing.

[0162] Furthermore, as the sliding time window slides, the computer device can update the corresponding counter based on the target attribute value category to which the updated attribute value belongs within the sliding time window, redetermine the occurrence frequency of the target attribute value category based on the updated counter, and update the vector component corresponding to the target attribute value category in the histogram vector based on the redetermined occurrence frequency of the target attribute value category.

[0163] Second, computer equipment can use an autoencoder based on a self-attention mechanism to jointly encode each histogram vector to obtain the database feature representation of the database.

[0164] Specifically, the computer device can perform parallel projection of the input matrix composed of histogram vectors corresponding to each data attribute of the data in the database using multiple attention heads to obtain the query matrix, key matrix, and value matrix of each attention head; for each attention head, the correlation between the query matrix and the key matrix of the attention head is calculated, and the attention weight matrix is ​​determined based on the correlation calculation result. The value matrix of the attention head is weighted and summed using the attention weight matrix to obtain the output result of the attention head; the output results of each attention head are fused to obtain the database feature representation of the database.

[0165] Third, computer equipment can parse SQL queries into abstract syntax trees, obtain updated abstract syntax trees through equivalent recombination, and input the updated abstract syntax trees into TCNN to obtain query feature representations.

[0166] Specifically, the computer device can parse the SQL query into an abstract syntax tree (AST) according to the SQL syntax rules. The computer device can then traverse the AST, traverse the join predicates of the abstract syntax tree, group the data attributes on both sides of the traversed equivalent join predicates according to equivalence, obtain at least one equivalent attribute group, and the computer device can further generate equivalence class labels for each equivalent attribute group and record them in the member table.

[0167] The computer device can further replace the original connection conditions in the AST with equivalence class references and add equivalence class information to the AST nodes, thereby embedding equivalence class labels and obtaining an updated abstract syntax tree. The computer device can then input the abstract syntax tree into a tree convolutional neural network, where the tree structure is transformed into a linear sequence, for example, by generating a node sequence through depth-first traversal. Each element in the sequence is an operator, relation, or equivalence class reference. Furthermore, a fixed-dimensional feature vector can be generated for each node, and then long-distance dependencies in the sequence are captured using convolutional kernels with different dilation rates. Residual connections are then used to mitigate the gradient vanishing problem in deep networks, preserving the original features. Finally, pooling operations are used to compress the variable-length sequence into a fixed-length feature vector, thereby obtaining the query feature representation.

[0168] Fourth, the computer equipment further determines the cardinality estimate by using a query parser based on a cross-attention mechanism, according to the database feature representation and the query feature representation.

[0169] Specifically, in the query parser, the computer device can perform parallel projection of the query feature representation using multiple attention heads to obtain the query matrix of each attention head, and perform parallel projection of the database feature representation using multiple attention heads to obtain the key matrix and value matrix of each attention head. For each attention head, the correlation between the query matrix and the key matrix of the attention head is calculated, and the attention weight matrix is ​​determined based on the correlation calculation result. The value matrix of the attention head is weighted and summed using the attention weight matrix to obtain the output result of the attention head. The output results of each attention head are fused to obtain the attention processing result. Finally, the attention processing result is mapped using linear regression to obtain the cardinality estimate of the SQL query.

[0170] In the above embodiments, histogram data featureization and incremental update mechanisms achieve efficient adaptation to dynamic workloads. Only local updates of relevant histogram vectors are needed to capture changes in data distribution, which greatly improves computational efficiency compared to traditional full reconstruction methods. At the same time, by combining tree convolutional neural networks and equivalence class recombination technology, complex SQL queries are transformed into feature vectors with high information density, which greatly reduces the cardinality estimation error rate of multi-table join queries. By establishing a dynamic hybrid histogram association between queries and data through a multi-head attention mechanism, the estimation accuracy is greatly improved and interpretability is enhanced. In addition, the first similar bucketing and the second type of bucketing divide-and-conquer strategy are adopted, which greatly reduces the statistical error of long-tailed distributed data under the same storage overhead.

[0171] In some embodiments, this application also provides an application scenario in which SQL queries are optimized through cardinality estimation.

[0172] In related technologies, cardinality estimation can be performed using histogram cardinality estimators. However, this method typically relies on simplified assumptions, such as the independence of attributes. Therefore, its accuracy is limited when data changes dynamically, making it difficult for the optimizer to select a good query plan. Machine learning-based cardinality estimation methods are also available; however, these methods often do not support dynamic workloads or require a complete model rebuild during model maintenance, a process that is typically time-consuming and impractical.

[0173] Therefore, it is evident that existing technologies have limited capacity to handle dynamic workloads, exhibiting significant performance degradation when dealing with dynamic workloads, especially when data distribution changes drastically, making it impossible to guarantee the accuracy of cardinality estimation. Furthermore, existing methods are inefficient in updating feature sets, failing to adapt to data changes in real time, resulting in poor query optimization performance.

[0174] To address the aforementioned problems, this application provides a database data processing method that can efficiently capture changes in data distribution. By constructing a histogram vector for each attribute and updating only the relevant relation vectors when data changes, it significantly improves the efficiency of processing dynamic workloads. The solution in this application can effectively estimate the cardinality of SPJ queries under dynamic workloads. (Reference) Figure 9 This scheme achieves efficient cardinality prediction by characterizing the database and queries, and then inputting the characterization results into an attention-based cardinality estimation model. The specific implementation is as follows:

[0175] 1. Data Featurization:

[0176] Data characterization is a method for compressing the entire database, effectively describing the data of each attribute and the relationships between them. In this embodiment, the histogram of each attribute can be used as database states, represented as a set of histogram vectors of the same dimension. The DB states are a dynamic set composed of histograms of each attribute. Each histogram can be flexibly adjusted according to the complexity of the data distribution to capture more distribution information. When information in the database is updated by CRUD operations, only the relevant DB state vectors need to be modified. This method not only improves the computational efficiency of characterization but also enables it to cope with dynamic workloads.

[0177] In this embodiment, for each data attribute, high-frequency and low-frequency attribute values ​​are identified. High-frequency attribute values ​​are recorded using a precise method similar to a frequency histogram, while low-frequency attribute values ​​are recorded using an approximate statistical method similar to a contour histogram. "Special buckets" are created for high-frequency attribute values, with each high-frequency attribute value occupying a separate bucket to record its precise frequency of occurrence. The remaining low-frequency attribute values ​​are grouped into multiple "regular buckets," each containing a certain range of values. This hybrid approach significantly reduces storage overhead while maintaining statistical accuracy, making it particularly suitable for long-tailed distribution data common in the real world.

[0178] 2. Query Featurization:

[0179] After parsing the SQL query into an abstract syntax tree (AST), an updated AST is obtained through equivalent recombination. This updated AST is then input into a TCNN to obtain the query feature representation. Query featureization effectively captures key query features and reduces storage overhead because the featureized database state contains only a small number of floating-point numbers, typically less than 100.

[0180] 3. Model Architecture:

[0181] The core of the cardinality estimation model in this application is an attention-based model, comprising an encoder based on self-attention and a query parser based on cross-attention, wherein:

[0182] By using an encoder based on a self-attention mechanism, the input matrix composed of histogram vectors corresponding to each data attribute in the database is subjected to parallel projection by multiple attention heads to obtain the query matrix, key matrix, and value matrix of each attention head. For each attention head, the correlation between the query matrix and the key matrix of the attention head is calculated, and the attention weight matrix is ​​determined based on the correlation calculation results. The value matrix of the attention head is then weighted and summed using the attention weight matrix to obtain the output result of the attention head. Finally, the output results of each attention head are fused to obtain the database feature representation of the database.

[0183] A query parser based on a cross-attention mechanism is used to project the query feature representation into multiple attention heads in parallel to obtain the query matrix of each attention head. Similarly, the database feature representation is projected into multiple attention heads in parallel to obtain the key matrix and value matrix of each attention head. For each attention head, the correlation between the query matrix and the key matrix is ​​calculated. The attention weight matrix is ​​determined based on the correlation calculation results. The value matrix of each attention head is weighted and summed using the attention weight matrix to obtain the output result of the attention head. Finally, the output results of each attention head are fused to obtain the attention processing result.

[0184] Furthermore, the attention processing result output by the query parser is passed through a linear regression layer to calculate the cardinality estimate.

[0185] During training, the cardinality estimation model can be trained using query statements with true cardinality labels. The training process can employ the Minimum Weighted Mean Square Error (MWSE) loss function to consider the difference between the batch cardinality prediction and the true cardinality in order to adjust the parameters of the encoder and query parser.

[0186] 4. Dynamic workload support:

[0187] The solution proposed in this application can adapt to dynamic changes in data. By capturing the distribution characteristics of each attribute through feature extraction, the model can provide different predictions based on different database states when the data changes. Its performance under dynamic workloads significantly outperforms other existing machine learning cardinality estimation methods. Specifically:

[0188] On the one hand, the histogram vectors of each data attribute in this application collectively constitute database states (DB states). In the process of generating histogram vectors for each data attribute, the number of bins for the first type of bins is determined by obtaining the target dimension number preset for the histogram vector. Based on the target dimension number and the number of bins for the first type of bins, the target number of bins for the second type of bins is determined. Based on multiple low-frequency attribute values, the data features of the second type of bins that determine the target number of bins are merged. This makes the dimension of the histogram vector of each data attribute fixed. When data is inserted, deleted, or updated, only the histogram vector of the corresponding data attribute needs to be modified. Even if the data distribution drifts, the model input remains stable, avoiding model structure adjustments due to data changes and eliminating the need to rebuild the global model. For example, if a database update inserts 100 records involving data attribute A and data attribute B, then the histogram vectors corresponding to data attribute A and data attribute B can be modified accordingly. Since inserting data only requires updating the histogram vectors of the relevant relationships, the time complexity is O(v) (where v is the number of records involved), which is much lower than the global update cost of traditional methods.

[0189] Furthermore, the encoder module based on the self-attention mechanism dynamically calculates the real-time correlation weights between data attributes through a multi-head self-attention mechanism. For example, when the distribution of the attribute values ​​of a certain data attribute changes, the self-attention weights will adaptively adjust, enabling the model to capture the dynamic changes in the joint distribution between attributes.

[0190] Furthermore, the query analyzer module based on the cross-attention mechanism interacts with the dynamically updated database state and query features in real time through the cross-attention mechanism. For example, when processing queries containing "price filtering + brand connection", the model will dynamically adjust the attention allocation to relevant attributes based on the latest price distribution histogram and brand attribute association weights, thereby outputting a cardinality estimate that adapts to the current data state.

[0191] The cardinality estimate predicted in this application embodiment can be applied to the query optimizer. Based on the cardinality estimate and the SQL query, the query optimizer can generate a query plan and provide it to the execution engine. After the execution engine executes the query plan, it can feed back the execution time to the cardinality estimation model, thereby optimizing the parameters of the cardinality estimation model.

[0192] In some embodiments, this application also provides another application scenario. In this scenario, the database data processing method of this application is applied to data anomaly detection and distribution drift monitoring. By monitoring abnormal changes in the data distribution in the database in real time (such as attribute skew caused by sudden insertion), model updates or alarms are triggered. Specifically, after initializing the database, the computer device can determine the various attribute values ​​of each data attribute in the database as high-frequency attribute values ​​or low-frequency attribute values ​​based on the frequency index values ​​of the various attribute values. For each high-frequency attribute value, a first type of bucket is determined based on the high-frequency attribute value. Based on multiple low-frequency attribute values, at least one second type of bucket is determined by merging. According to the first type of bucket and the second type of bucket, the histogram vector corresponding to the data attribute is constructed. Based on the histogram vectors corresponding to each data attribute in the database, joint encoding is performed to obtain the database feature representation of the database. This database feature representation is used as the baseline Zbase. When the database data changes, the corresponding histogram vector is updated and the database feature representation Z is regenerated. The difference between the current Z and the historical baseline Zbase (such as cosine distance, KL divergence) is calculated to quantify the degree of distribution change. When the difference exceeds the threshold, it is determined to be a distribution drift and an alarm is triggered.

[0193] In some embodiments, this application also provides another application scenario. In this application scenario, the database data processing method of this application can also be applied to semantic query similarity measurement to determine the semantic similarity between two queries, for query caching, optimization rule sharing, etc. Specifically, after obtaining the database feature representation Z of the database, the computer device can store the database feature representation. When semantic query similarity measurement is required, the database feature representation Z is obtained, and query statements q1 and q2 are respectively cross-attention calculated with Z to obtain response vectors y1 and y2. The semantic similarity of the queries is measured by vector similarity (such as cosine similarity) on y1 and y2. Similar queries can reuse historical optimization plans or result caching.

[0194] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0195] Based on the same inventive concept, this application also provides a database data processing apparatus for implementing the database data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more database data processing apparatus embodiments provided below can be found in the limitations of the database data processing method described above, and will not be repeated here.

[0196] In some embodiments, such as Figure 10 As shown, a database data processing device 1000 is provided, comprising:

[0197] The attribute value classification module 1002 is used to determine the various attribute values ​​of the data attribute as high-frequency attribute values ​​or low-frequency attribute values ​​based on the frequency index values ​​of the various attribute values ​​of the data attribute for each data attribute in the database.

[0198] The first type of bucket determination module 1004 is used to determine a first type of bucket based on the high-frequency attribute value for each type of high-frequency attribute value.

[0199] The second type of bucket determination module 1006 is used to determine at least one second type of bucket based on multiple low-frequency attribute values.

[0200] The histogram vector construction module 1008 is used to construct histogram vectors corresponding to data attributes based on the first type of bucketing and the second type of bucketing.

[0201] The joint encoding module 1010 is used to perform joint encoding based on the histogram vectors corresponding to each data attribute in the database to obtain the database feature representation.

[0202] The aforementioned database data processing device can construct histogram vectors for each data attribute and jointly encode the histogram vectors to obtain a database feature representation. The resulting database representation can accurately model the database feature distribution. Furthermore, in the process of constructing the histogram vector for a single data attribute, various attribute values ​​are identified as either high-frequency or low-frequency attribute values. Different strategies are used to determine histogram binning for high-frequency and low-frequency attribute values. Since the types of high-frequency attribute values ​​have a significant impact on the accuracy of modeling the database feature distribution, but their number is usually small, determining histogram binning separately for high-frequency attribute values ​​can improve statistical accuracy while having a small impact on storage overhead. Conversely, the types of low-frequency attribute values ​​have a small impact on the accuracy of modeling the database feature distribution, but their number is usually large. Therefore, determining histogram binning based on multiple low-frequency attribute values ​​can significantly reduce storage overhead without affecting statistical accuracy. This is suitable for long-tailed distributed data, thereby achieving both improved accuracy of database feature representation and reduced storage overhead.

[0203] In some embodiments, the attribute value classification module 1002 is further configured to: when the attribute value of a data attribute is a discrete attribute value, classify attribute values ​​with the same value in the set of attribute values ​​of the data attribute into the same type of attribute value to obtain multiple attribute values; determine the frequency index value of each type of attribute value respectively; determine the attribute value whose frequency index value meets the high-frequency condition as a high-frequency attribute value, and determine the attribute value whose frequency index value does not meet the high-frequency condition as a low-frequency attribute value.

[0204] In some embodiments, the attribute value classification module 1002 is further configured to: determine the number of times each attribute value appears in the sliding time window based on the attribute values ​​of the data attributes contained in the sliding time window; the sliding step size of the sliding time window is negatively correlated with the data change amplitude in the sliding time window; and determine the frequency of occurrence of each attribute value based on the number of times each attribute value appears in the sliding window and the total number of attribute values ​​in the sliding time window.

[0205] In some embodiments, the sliding window is provided with counters for various attribute values, and each counter is used to count the number of times each attribute value appears within the sliding window. In the histogram vector, each first type bucket and each second type bucket corresponds to a vector component. The database data processing device further includes: a histogram update module, used to update the corresponding counter according to the target attribute value category to which the updated attribute value belongs when the sliding time window slides; redetermine the occurrence frequency of the target attribute value category according to the updated counter; and update the vector component corresponding to the target attribute value category in the histogram vector according to the redetermined occurrence frequency of the target attribute value category.

[0206] In some embodiments, the attribute value classification module 1002 is further configured to: when the attribute value of a data attribute is a continuous attribute value, divide the range of the attribute value of the data attribute into intervals to obtain multiple data intervals; determine each data interval as an attribute value; determine the frequency index value of each type of attribute value; determine the attribute value whose frequency index value meets the high-frequency condition as a high-frequency attribute value, and determine the attribute value whose frequency index value does not meet the high-frequency condition as a low-frequency attribute value.

[0207] In some embodiments, the attribute value classification module 1002 is further configured to: cluster the data of the data attribute when the attribute value of the data attribute is a continuous attribute value, to obtain multiple clusters; divide the data into intervals according to the multiple clusters, to obtain multiple data intervals, and the multiple data intervals correspond one-to-one with the multiple clusters.

[0208] In some embodiments, the database data processing apparatus further includes: a cardinality estimation module, configured to acquire a query statement, perform feature processing on the query statement to obtain a query feature representation corresponding to the query statement; perform attention processing on the database feature representation based on the query feature representation to obtain an attention processing result; and determine a cardinality estimate of the query statement based on the attention processing result.

[0209] In some embodiments, the cardinality estimation module is also used to parse the query statement into an abstract syntax tree;

[0210] Traverse the connection predicates of the abstract syntax tree, group the data attributes on both sides of the traversed equivalent connection predicates according to equivalence, and obtain at least one equivalent attribute group; generate equivalence class labels for each equivalent attribute group, embed the equivalence class labels in the abstract syntax tree, and obtain the updated abstract syntax tree; input the abstract syntax tree into the tree convolutional neural network to obtain the query feature representation corresponding to the query statement.

[0211] In some embodiments, the cardinality estimation module is further configured to: perform parallel projection of the query feature representation onto multiple attention heads to obtain the query matrix of each attention head; perform parallel projection of the database feature representation onto multiple attention heads to obtain the key matrix and value matrix of each attention head; for each attention head, calculate the correlation between the query matrix and the key matrix of the attention head, determine the attention weight matrix based on the correlation calculation result, and perform weighted summation of the value matrix of the attention head using the attention weight matrix to obtain the output result of the attention head; and fuse the output results of each attention head to obtain the attention processing result.

[0212] In some embodiments, the joint encoding module is further configured to: perform parallel projection of the input matrix composed of histogram vectors corresponding to each data attribute of the data in the database with multiple attention heads to obtain the query matrix, key matrix, and value matrix of each attention head; for each attention head, calculate the correlation between the query matrix and the key matrix of the attention head, determine the attention weight matrix based on the correlation calculation result, and obtain the output result of the attention head by weighted summing of the value matrix of the attention head through the attention weight matrix; and fuse the output results of each attention head to obtain the database feature representation of the database.

[0213] In some embodiments, the histogram vector includes vector components corresponding one-to-one with each first type of bucket and vector components corresponding one-to-one with each second type of bucket; the second type of bucket determination module is further configured to: obtain a preset target dimension number for the histogram vector; determine the number of buckets of the first type of bucket, and determine the target number of buckets of the second type of bucket based on the target dimension number and the number of buckets of the first type of bucket; and merge the second type of buckets whose target number of buckets is determined based on multiple low-frequency attribute values.

[0214] Each module in the aforementioned database data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0215] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 11 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores various types of business data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a database data processing method.

[0216] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows:Figure 12 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a database data processing method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0217] Those skilled in the art will understand that Figure 11 , Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0218] In some embodiments, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the database data processing method described above.

[0219] In some embodiments, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the database data processing method described above.

[0220] In some embodiments, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the database data processing method described above.

[0221] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0222] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logic devices, etc., and are not limited to these.

[0223] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0224] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A database data processing method, characterized in that, The method includes: For each data attribute in the database, based on the frequency index value of various attribute values ​​of the data attribute, the various attribute values ​​of the data attribute are respectively determined as high-frequency attribute values ​​or low-frequency attribute values; For each high-frequency attribute value, a first type of bucket is determined based on the high-frequency attribute value; Based on multiple low-frequency attribute values, at least one second type of bucket is determined by merging them. Based on the first type of bucketing and the second type of bucketing, construct the histogram vector corresponding to the data attribute; Based on the histogram vectors corresponding to each data attribute in the database, joint encoding is performed to obtain the database feature representation of the database.

2. The method according to claim 1, characterized in that, The frequency index values ​​of various attribute values ​​based on the data attributes, which determine the various attribute values ​​of the data attributes as high-frequency attribute values ​​or low-frequency attribute values, include: When the attribute value of the data attribute is a discrete attribute value, attribute values ​​with the same value in the set of attribute values ​​of the data attribute are divided into the same type of attribute value to obtain multiple attribute values; Determine the frequency index value for each of the various attribute values; Attribute values ​​whose frequency index values ​​meet the high-frequency condition are determined as high-frequency attribute values, and attribute values ​​whose frequency index values ​​do not meet the high-frequency condition are determined as low-frequency attribute values.

3. The method according to claim 2, characterized in that, The process of determining the frequency index values ​​for each of the various attribute values ​​includes: Based on the attribute values ​​of the data attributes contained within the sliding time window, the frequency of occurrence of each attribute value within the sliding time window is determined; the sliding step size of the sliding time window is negatively correlated with the magnitude of data change within the sliding time window. The frequency of occurrence of each attribute value is determined based on the number of times each attribute value appears in the sliding window and the total number of attribute values ​​in the sliding time window.

4. The method according to claim 3, characterized in that, The sliding window is equipped with counters for various attribute values. Each counter is used to count the number of times each attribute value appears in the sliding window. In the histogram vector, each first type bucket and each second type bucket corresponds to a vector component. The method further includes: As the sliding time window slides, the corresponding counter is updated according to the target attribute value category to which the updated attribute value within the sliding time window belongs; The frequency of occurrence of the target attribute value type is re-determined based on the updated counter; Based on the frequency of occurrence of the newly determined target attribute value types, update the vector components corresponding to the target attribute value types in the histogram vector.

5. The method according to claim 1, characterized in that, The frequency index values ​​of various attribute values ​​based on the data attributes, which determine the various attribute values ​​of the data attributes as high-frequency attribute values ​​or low-frequency attribute values, include: When the attribute value of the data attribute is a continuous attribute value, the range of the attribute value of the data attribute is divided into intervals to obtain multiple data intervals; Each data range is assigned a specific attribute value; Determine the frequency index value for each of the various attribute values; Attribute values ​​whose frequency index values ​​meet the high-frequency condition are defined as high-frequency attribute values, and attribute values ​​whose frequency index values ​​do not meet the high-frequency condition are defined as low-frequency attribute values.

6. The method according to claim 5, characterized in that, When the attribute value of the data attribute is a continuous attribute value, the range of the attribute value of the data attribute is divided into intervals to obtain multiple data intervals, including: When the attribute value of the data attribute is a continuous attribute value, the data of the data attribute is clustered to obtain multiple clusters; The data intervals are divided according to the multiple clusters, and each data interval corresponds one-to-one with the multiple clusters.

7. The method according to claim 1, characterized in that, The method further includes: Obtain the query statement, perform feature processing on the query statement, and obtain the query feature representation corresponding to the query statement; Attention processing is performed on the database feature representation based on the query feature representation to obtain the attention processing result; Based on the attention processing results, the cardinality estimate of the query statement is determined.

8. The method according to claim 7, characterized in that, The step of performing feature processing on the query statement to obtain the query feature representation corresponding to the query statement includes: The query statement is parsed into an abstract syntax tree; Traverse the connection predicates of the abstract syntax tree, group the data attributes on the left and right sides of the traversed equivalent connection predicates according to equivalence, and obtain at least one equivalent attribute group; For each group of equivalent attributes, generate an equivalence class tag and embed the equivalence class tag into the abstract syntax tree to obtain an updated abstract syntax tree; The abstract syntax tree is input into a tree convolutional neural network to obtain the query feature representation corresponding to the query statement.

9. The method according to claim 7, characterized in that, The step of performing attention processing on the database feature representation based on the query feature representation to obtain the attention processing result includes: The query feature representation is subjected to parallel projection with multiple attention heads to obtain the query matrix of each attention head; The database feature representation is subjected to parallel projection with multiple attention heads to obtain the key matrix and value matrix of each attention head; For each attention head, the correlation between the query matrix and the key matrix of the attention head is calculated. The attention weight matrix is ​​determined based on the correlation calculation result. The value matrix of the attention head is weighted and summed using the attention weight matrix to obtain the output result of the attention head. The outputs of each attention head are combined to obtain the attention processing result.

10. The method according to claim 1, characterized in that, The method of jointly encoding the histogram vectors corresponding to each data attribute in the database to obtain the database feature representation includes: The input matrix, composed of the histogram vectors corresponding to each data attribute in the database, is subjected to parallel projection with multiple attention heads to obtain the query matrix, key matrix, and value matrix of each attention head. For each attention head, the correlation between the query matrix and the key matrix of the attention head is calculated. The attention weight matrix is ​​determined based on the correlation calculation result. The value matrix of the attention head is weighted and summed using the attention weight matrix to obtain the output result of the attention head. By fusing the outputs of each attention head, a database feature representation of the database is obtained.

11. The method according to any one of claims 1 to 10, characterized in that, The histogram vector includes vector components corresponding one-to-one with each of the first type of buckets and vector components corresponding one-to-one with each of the second type of buckets. The step of determining at least one second type of bucket based on multiple low-frequency attribute values ​​includes: Obtain the preset target dimension for the histogram vector; Determine the number of buckets for the first type of bucketing, and based on the target number of dimensions and the number of buckets for the first type of bucketing, determine the target number of buckets for the second type of bucketing; Based on various low-frequency attribute values, the second type of buckets is combined to determine the target number of buckets.

12. A database data processing device, characterized in that, The device includes: The attribute value classification module is used to determine the various attribute values ​​of the data attribute as high-frequency attribute values ​​or low-frequency attribute values ​​for each data attribute in the database, based on the frequency index values ​​of the various attribute values ​​of the data attribute. The first type of bucketing determination module is used to determine a first type of bucket based on each high-frequency attribute value; The second type of bucket determination module is used to determine at least one second type of bucket based on multiple low-frequency attribute values. The histogram vector construction module is used to construct the histogram vector corresponding to the data attribute based on the first type of bucketing and the second type of bucketing; The joint encoding module is used to perform joint encoding based on the histogram vectors corresponding to each data attribute of the data in the database, so as to obtain the database feature representation of the database.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

Images