Data sampling method

Abstract

An embodiment of the present specification provides a data sampling method, wherein the data sampling method includes: receiving a data sampling request for a target data table, wherein the data sampling request carries a target index position; based on the target index position, using a prediction model to obtain a target predicted sampling probability corresponding to the target index position, wherein the prediction model is trained based on sample pairs, and the sample pairs include a sample index position and a sample predicted sampling probability corresponding to the sample index position; based on the target predicted sampling probability, sampling the data to be sampled at the target index position to obtain target data, thereby significantly improving sampling efficiency and sampling accuracy.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of data processing technology, and in particular to a data sampling method. Background Art

[0002] With the continuous development of computer technology, various performance tuning tools are generally configured in databases to avoid various performance issues that may arise in databases. For example, to address the issue of missing database indexes, index recommendation tuning tools are configured in the database. Constructing accurate statistical information is the cornerstone of index recommendation database tuning tools. Based on this, the existing technology generally constructs statistical information using data obtained using a uniform sampling method. However, the uniform sampling method has very low sampling efficiency and sampling accuracy in frequently updated databases, resulting in the inability to quickly and accurately construct statistical information. Therefore, there is an urgent need to provide a method that can significantly improve sampling efficiency and sampling accuracy. Summary of the Invention

[0003] In view of this, embodiments of this specification provide a data sampling method. One or more embodiments of this specification also relate to a data sampling apparatus, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a data sampling method is provided, including:

[0005] receiving a data sampling request for a target data table, wherein the data sampling request carries a target index position;

[0006] According to the target index position, using a prediction model to obtain a target predicted sampling probability corresponding to the target index position, wherein the prediction model is trained based on sample pairs, and the sample pairs include a sample index position and a sample predicted sampling probability corresponding to the sample index position;

[0007] Based on the target predicted sampling probability, the data to be sampled at the target index position is sampled to obtain target data.

[0008] According to a second aspect of the embodiments of this specification, a data sampling device is provided, including:

[0009] A receiving module is configured to receive a data sampling request for a target data table, wherein the data sampling request carries a target index position;

[0010] A prediction module is configured to obtain, based on the target index position, a target predicted sampling probability corresponding to the target index position using a prediction model, wherein the prediction model is trained based on sample pairs, and the sample pairs include a sample index position and a sample predicted sampling probability corresponding to the sample index position;

[0011] The sampling module is configured to sample the data to be sampled at the target index position based on the target predicted sampling probability to obtain target data.

[0012] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0013] memory and processor;

[0014] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the data sampling method are implemented.

[0015] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the steps of the data sampling method are implemented.

[0016] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the data sampling method.

[0017] The data sampling method provided in this specification includes: receiving a data sampling request for a target data table, wherein the data sampling request carries a target index position; based on the target index position, using a prediction model to obtain a target predicted sampling probability corresponding to the target index position, wherein the prediction model is trained based on sample pairs, and the sample pairs include a sample index position and a sample predicted sampling probability corresponding to the sample index position; based on the target predicted sampling probability, sampling the data to be sampled at the target index position to obtain target data.

[0018] Specifically, when receiving a data sampling request for a target data table, the method uses a prediction model to determine the corresponding target predicted sampling probability for the target index position carried in the data sampling request, and samples the data to be sampled in the target data table based on the target predicted sampling probability, thereby significantly improving the sampling efficiency and sampling accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a schematic diagram of statistical information provided by an embodiment of this specification;

[0020] Figure 2 This is a schematic diagram of an application scenario of a data sampling method provided by an embodiment of this specification;

[0021] Figure 3 is a flow chart of a data sampling method provided by one embodiment of this specification;

[0022] Figure 4 This is a flow chart of a processing process of a data sampling method provided by one embodiment of this specification;

[0023] Figure 5 This is a structural diagram of a data sampling device provided by an embodiment of this specification;

[0024] Figure 6 This is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0025] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0026] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0027] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0028] First, the terms involved in one or more embodiments of this specification are explained.

[0029] Accept-reject Sampling: A commonly used sampling method for sampling elements from a distribution.

[0030] Inverser Probablity Weighting: A statistical method to correct for sampling bias.

[0031] RDS: short for Relational Database Service, is a ready-to-use, stable, reliable, and elastically scalable online database service.

[0032] In the cloud computing era, open-source relational databases (such as MySQL) are becoming the preferred choice for a growing number of small and medium-sized enterprises migrating to the cloud. However, when using open-source relational databases, users may encounter various performance issues, such as missing indexes. Currently, there are no performance tuning tools specifically for open-source databases. Alternatively, due to enterprise scale constraints, users often lack experienced database administrators to perform performance tuning.

[0033] Some databases generally have built-in database tuning tools, such as index recommendation tools, which can help users solve database performance problems. However, these tools are customized for specific commercial databases. For users of open source relational databases, such automatic database tuning tools are currently lacking.

[0034] For index recommendation tools, constructing accurate statistical information is the cornerstone. Different indexes need to be recommended for different data distributions. Furthermore, to ensure that statistical information is constructed without impacting the normal operation of database services such as data writing, querying, adding, and deleting, sampling is typically used. However, typical uniform sampling methods have low sampling efficiency for frequently updated relational databases.

[0035] For example, this specification provides a uniform sampling solution. Specifically, given a table T in a database instance, assuming the table has at least one indexed numeric column (typically an auto-incrementing primary key) named id, elements of a row in the table can be retrieved by performing a range query SQL on the indexed column. However, considering the sampling budget or sampling cost, it is desirable to use a series of range query SQLs to obtain statistical information about other columns in the table, such as a histogram, at a given sampling cost.

[0036] The value range of the index column is, for example, 1 to 1,000,000. This solution uses uniform sampling to retrieve elements from the table. To improve sampling efficiency, we assume that the sampling range for each sample is " / ", where " / " is equivalent to l in the following formula. We can randomly select a number n in [1, 1,000,000] and use the following SQL statement to retrieve elements from the table and construct statistical information using these elements.

[0037] SELECT*FROM T WHERE id BETWEEN n - / / 2 AND n + / / 2.

[0038] For example, if the size of table T is similar to the range of the index column, for example, T has 1,000,000 rows, and the range of ids in the index column is 1 to 1,000,000, that is, the range of the index column values ​​is 1 to 1,000,000, then uniform sampling will have good sampling efficiency in this table T.

[0039] However, relational databases are constantly changing, with numerous insert and delete operations. Consequently, the range of table IDs can be much larger than the table size. For example, table T may have only 10,000 rows, but its IDs still range from 1 to 1,000,000. This is called a sparse table. For such a sparse table, sparsity is defined as {table size} / {index column range}. In the example above, the sparsity of table T can be 10,000 / 1,000,000 = 1%.

[0040] By analyzing the tables provided by some companies in the RDS service, we know that sparse tables are very common. For example, an analysis of 10,000 tables in 1,000 instances of some companies' RDS services shows that more than 25% of the tables have a sparsity of less than 5%. This shows that sparse tables are very common in actual applications. Figure 1 , Figure 1 This is a schematic diagram of statistical information provided by an embodiment of this specification. Figure 1 The table shows the sparsity of the data in the sparse table, as well as the proportion of data in different ranges.

[0041] For this sparse table, it is foreseeable that the general uniform sampling method has very low sampling efficiency. In particular, when the IDs in the table are not evenly distributed in the range of 1 to 1,000,000, but are concentrated in a certain area, many SQL statements will return null, thereby wasting sampling costs (such as sampling time and computing resources).

[0042] Based on this, the data sampling method provided in this specification proposes a method for constructing sparse table statistical information of a relational database based on active sampling. When receiving a data sampling request for a target data table, the target predicted sampling probability corresponding to the target index position is obtained using a prediction model based on the target index position carried in the data sampling request, wherein the prediction model is trained based on a sample pair, and the sample pair includes a sample index position and a sample predicted sampling probability corresponding to the sample index position.

[0043] Based on the target predicted sampling probability, the data to be sampled at the target index position is sampled to obtain the target data. This significantly improves sampling efficiency and achieves higher sampling accuracy with less sampling cost. This method constructs statistical information by executing SQL to sample database records and is applicable to all relational databases. It is also suitable for tuning tools that require database statistics.

[0044] Specifically, in this specification, a data sampling method is provided. This specification also relates to a data sampling device, a computing device, a computer-readable storage medium, and a computer program, which are described in detail one by one in the following embodiments.

[0045] Figure 2 A schematic diagram of an application scenario of a data sampling method provided according to an embodiment of this specification is shown. Figure 2 It can be seen that the data sampling method provided in this specification can be applied to the database 204, which is connected to the client 202. Based on this, when the client 202 needs to sample a data table in the database, it can send a data sampling request for a target data table in the database 204 to the database 204. The target data table can be a table stored in the database 204 and capable of performing a sampling operation.

[0046] Upon receiving a data sampling instruction from a client, database 204 first determines the data table targeted by the data sampling instruction, i.e., the table on which the sampling operation is to be performed, from one or more data tables stored within the database. The method for determining the data table can be configured based on the actual application scenario and is not specifically limited in this specification. For example, the data sampling request can include identification information of the target data table, and database 204 can determine the target data table on which the sampling operation is to be performed based on the identification information. In actual applications, the target data table can be a sparse table.

[0047] After the database 204 determines the sparse table that needs to be sampled, for the sparse table T, assuming that the sparse table T has N rows, and the value range of the index column of the sparse table T is [1, M], N<<M. M can be any value. This specification specifically limits this replenishment. For any index position n∈[1, M], if there is a function p(n) that can tell the database 204 the probability of having a data record at position n, then the database 204 can use this probability to perform sampling that is more efficient than even sampling. Therefore, the database 204 is configured with a pre-trained prediction model, which is obtained based on sample pair training. The sample pair includes a sample index position and the sample prediction sampling probability corresponding to the sample index position.

[0048] Based on this, the database 204 obtains the index position n carried in the data sampling request, and uses the prediction model to predict the sampling probability of the index position n to obtain the predicted sampling probability corresponding to the index position n. That is, input the index position n into the prediction model to obtain the predicted sampling probability corresponding to the index position n

[0049] The database 204 then calculates the predicted sampling probability based on the The data to be sampled at the index position n in the sparse table T is sampled to obtain the target data, thereby improving the sampling efficiency for the sparse table and avoiding the problem of low sampling efficiency of the sparse table due to equal sampling.

[0050] Subsequently, the database 204 returns the target data to the client 202 so that the client 202 can construct statistical information of the sparse table T based on the sampled target data. Alternatively, the database 204 can also construct statistical information of the sparse table T based on the target data and send the statistical information to the client 202.

[0051] Figure 3 A flow chart of a data sampling method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0052] Step 302: Receive a data sampling request for a target data table, wherein the data sampling request carries a target index position.

[0053] Among them, the target data table can be understood as a table stored in a database, and the database needs to perform data sampling operations. The target data table can be a sparse table; the database can be any type of database, and this specification does not make specific restrictions on this. For example, the database can be a relational database.

[0054] Among them, the target index position can be understood as any value in the index range (i.e., value range) corresponding to the index column of the target data table. For example, if the value range in the index column of the target data table is 1 to 1000000, then the target index position can be one or more values ​​in [1,1000000].

[0055] Based on this, the database can receive a data sampling request for the target data table, and the data sampling request also carries the target index position.

[0056] In the embodiments provided herein, before using a prediction model to obtain the target predicted sampling probability corresponding to a target index position, it is necessary to train the prediction model based on sample pairs, thereby achieving the goal of predicting the target predicted sampling probability corresponding to the target index position based on the pre-trained prediction model, thereby further improving data sampling efficiency. Specifically, before using the prediction model to obtain the target predicted sampling probability corresponding to the target index position based on the target index position, the following steps are also included:

[0057] Determine an index range of the sample data table, and determine a sample index position based on the index range, and a predicted sampling probability corresponding to the sample index position;

[0058] Using the sample index position as a training sample, and using the predicted sampling probability corresponding to the sample index position as the sample label corresponding to the training sample;

[0059] A sample pair is constructed based on the training samples and the corresponding sample labels, and the prediction model to be trained is trained according to the sample pair to obtain a trained prediction model.

[0060] The sample data table can be understood as a data table stored in a database and used to train a prediction model. In practical applications, the sample data table may be different from the target data table.

[0061] In one embodiment of this specification, during the data sampling process, the database allocates limited sampling resources to a data sampling task. A data sampling task can be understood as a task that, after receiving a data sampling request, performs a sampling operation on a target data table in response to the data sampling request. The sampling resources can be understood as the sampling budget or sampling cost required to perform the data sampling. Examples include the total sampling time allocated by the database to a data sampling task, the database response time, the number of SQL statements allocated to a data sampling task, and so on.

[0062] Based on this, considering the limited sampling budget, in order to improve the prediction effect of the prediction model under the limited sampling budget, and also to improve the prediction sampling probability obtained based on the prediction model, when sampling the target data table, the target data table can be used as a sample data table, and the prediction model is trained based on the target data table. In other words, the sample data table can be the target data table, or the sample data table can be a subset of the target data table; wherein, the subset of the target data table can be understood as a table containing some of the data records recorded in the target data table.

[0063] The index range of the sample data table may be understood as the value range corresponding to the index column of the sample data table. For example, the index range may be 1 to 1,000,000.

[0064] The predicted sampling probability corresponding to the sample index position can be understood as the probability of data that can be sampled at the sample index position. The predicted sampling probability can be any value in the range [0, 1].

[0065] Specifically, during the process of training the prediction model, the database determines the index range of the sample data table, determines the sample index position from the index range, and determines the predicted sampling probability corresponding to the sample index position based on the index range.

[0066] Afterwards, the sample index position is used as the training sample, and the predicted sampling probability corresponding to the sample index position is used as the sample label corresponding to the training sample. Sample pairs are constructed based on the training samples and their corresponding sample labels, and the prediction model to be trained is trained based on the sample pairs until the training stop condition is met, thereby obtaining a trained prediction model. The training stop condition can be set according to the actual application scenario, for example, after a specific number of training iterations have been performed, or after the model has reached convergence, so that the target predicted sampling probability corresponding to the target index position can be accurately determined based on the prediction model, further improving the accuracy and efficiency of data sampling.

[0067] Furthermore, determining the sample index position based on the index range and the predicted sampling probability corresponding to the sample index position includes:

[0068] Determining a sample index position and a sampling range corresponding to the sample index position based on the index range, and determining sample sampling data corresponding to the sample index position based on the sampling range;

[0069] Based on the attribute information of the sample sampling data and the sampling range, a predicted sampling probability corresponding to the sample index position is determined.

[0070] The sampling range corresponding to the sample index position can be understood as the surrounding range of the sample index position in the index range. For example, if the index range is 1 to 1,000,000 and the sample index position is 500, the sampling range corresponding to the sample index position can be [400, 600].

[0071] The attribute information of the sample sampling data can be understood as the quantity information of the sample sampling data. Based on this, based on the attribute information of the sample sampling data and the sampling range, the predicted sampling probability corresponding to the sample index position is determined. It can be understood that the quantity information of the sample sampling data is divided by the sampling range, and the result of the division operation is used as the predicted sampling probability corresponding to the sample index position.

[0072] Specifically, in the process of training the prediction model, the database first determines the sample index position and the sampling range corresponding to the sample index position from the index range, and determines the sample sampling data corresponding to the sample index position based on the sampling range; then, based on the attribute information of the sample sampling data and the sampling range, determines the predicted sampling probability corresponding to the sample index position.

[0073] Furthermore, in the embodiments provided herein, determining a sample index position and a sampling range corresponding to the sample index position based on the index range, and determining sample sampling data corresponding to the sample index position based on the sampling range, includes:

[0074] Determining a preset uniform sampling rule corresponding to the sample data table, and selecting a sample index position from the sample data table based on the preset uniform sampling rule;

[0075] Determining a preset range parameter corresponding to the sample index position, and determining a corresponding sample sampling range for the sample index position from the index range based on the preset range parameter;

[0076] The sample data to be sampled corresponding to the sample sampling range is obtained from the sample data table as the sample sampling data corresponding to the sample index position.

[0077] The preset range parameter may be understood as a parameter for determining a corresponding sampling range for a sample index position within an index range.

[0078] The following uses the application of data sampling methods in the scenario of building statistical information of relational databases as an example to further explain how to determine the sample index position based on the index range and determine the predicted sampling probability based on the sampling range.

[0079] In practical applications, before sampling a sparse table, the prediction model needs to be trained. This training step can be the initial step in constructing statistical information. Specifically, the database allocates a portion of the sampling budget (e.g., 10%) from the data sampling task to perform uniform sampling to obtain initial data. The uniform sampling process is as described in the above embodiment. That is, given a sparse table T in a database instance, the value range of the index column of the sparse table T is 1 to 1,000,000. Assuming that the sampling range for each sampling is " / ", we can randomly select a number n in [1, M], where M is equal to 1,000,000. Then use the following SQL statement to obtain elements in the table.

[0080] SELECT*FROM T WHERE id BETWEEN n - / / 2 AND n + / / 2;

[0081] The 2 in the above formula can be understood as the prediction range parameter. The data sampling method provided in this specification estimates the probability of a data record at a location by using the recorded data at other locations within the sampling range corresponding to the sample index location. If the above SQL returns m records, the following formula is used to define the predicted sampling probability corresponding to the sample index location.

[0082]

[0083] In the initial stage, a series of sampling locations n are randomly selected, the corresponding p(n) is calculated and recorded until the initial sampling budget is used up, thus ending the data sampling in the initial stage.

[0084] Afterwards, the database takes position n as a sample, constructs a sample pair based on the predicted sampling probability p(n) as a label, and trains the Gaussian process regression model based on the sample until the model reaches the stopping condition, thereby obtaining a trained Gaussian process regression model.

[0085] Step 304: According to the target index position, a prediction model is used to obtain a target prediction sampling probability corresponding to the target index position.

[0086] The prediction model is obtained based on sample pair training, and the sample pair includes a sample index position and a sample prediction sampling probability corresponding to the sample index position;

[0087] Specifically, the database inputs the target index position into the trained prediction model to obtain the target prediction probability corresponding to the target index position.

[0088] The prediction model can be understood as a model that can predict the target prediction sampling probability corresponding to the target index position. In practical applications, the prediction model can be any model that can realize the sampling probability corresponding to the index position prediction. This specification does not make specific limitations on this. For example, the prediction model can be a Gaussian process regression model.

[0089] The target predicted sampling probability can be understood as the probability of data that can be sampled at the target index position. The target predicted sampling probability can be any value in the interval [0, 1].

[0090] Furthermore, in the embodiment provided in this specification, the sampling resources allocated by the database to a data sampling task are limited; in the case of limited sampling budget, on the one hand, more sampling data are needed to construct statistical information, and on the other hand, we need to focus on predicting the sampling probability. Therefore, the database will have a parameter to control the proportion of the sampling budget. For example, 10% of the sampling budget can be used for Bayesian optimization to improve the accuracy of the prediction model, and 90% of the sampling can be used to sample data to construct statistical information. In this way, under a limited sampling budget, the model accuracy can be improved while completing the construction of statistical information. Specifically, according to the target index position, using the prediction model to obtain the target prediction sampling probability corresponding to the target index position includes:

[0091] Determining a data sampling resource corresponding to the data sampling request;

[0092] Determining a preset processing rule for the data sampling request and processing parameters corresponding to the preset processing rule;

[0093] When it is determined that the data sampling resource is not empty and the processing parameters meet the preset processing rules, the target index position is input into the prediction model to obtain the target prediction sampling probability corresponding to the target index position.

[0094] The preset processing rule can be understood as the rule of using 10% of the sampling budget for Bayesian optimization to improve the accuracy of the prediction model, and using 90% of the sampling to sample data to construct statistical information. The processing parameter can be understood as a parameter that controls the proportional distribution of the sampling budget. The processing parameter can be a random value in the interval [0, 1]. The random value has a 10% probability of being distributed in the interval [0, 0.1] and a 90% probability of being distributed in the interval (0.1, 1).

[0095] Data sampling resources can be understood as sampling budget.

[0096] The processing parameter meeting the preset processing condition can be understood as determining that the processing parameter meets the preset processing condition when the processing parameter is greater than a preset threshold (such as 0.1).

[0097] Continuing with the above example, the database will determine whether there is a sampling budget and the processing parameter is greater than a preset threshold (such as 0.1). If so, the index position n is input into the prediction model to obtain the target predicted sampling probability corresponding to the target index position n.

[0098] It should be noted that the database will use the recorded n and p(n) to train a Gaussian process regression model, which can be used to estimate the probability of all positions, recorded as At the same time, it is important to note that during the model optimization process, since only a new n and p(n) are added with each sampling, the Gaussian process regression model does not need to be updated after each sampling step. The Gaussian process model can be updated periodically, for example, after every 32 or 64 samplings, which can reduce the cost of Gaussian process training. The Gaussian process regression model is chosen because it can estimate not only the function value but also the corresponding variance (uncertainty), which is crucial for the Bayesian optimization to be used later.

[0099] Step 306: Based on the target predicted sampling probability, sample the data to be sampled at the target index position to obtain target data.

[0100] In the embodiment provided in this specification, sampling the data to be sampled at the target index position based on the target predicted sampling probability to obtain target data includes:

[0101] Determine an initial sampling range corresponding to the target index position, and adjust the initial sampling range based on the target predicted sampling probability to obtain a target sampling range;

[0102] The data to be sampled corresponding to the target sampling range is obtained from the target data table as the target data corresponding to the target index position.

[0103] The initial sampling range and target sampling range corresponding to the target index position can be referred to the above explanation and will not be described in detail here.

[0104] In practical applications, for sparse tables, a probability distribution function is known You can use Get more efficient sampling. When the database knows that a certain area has a greater probability of sampling data, it can not only implement more sampling in that area, but also dynamically adjust the sampling range. For sparse tables, when The smaller it is, the more likely it is that no data will be sampled, but when When the value is larger, the data in the same sampling range are not independent and identically distributed, which will significantly reduce the accuracy of the statistical information to be constructed later. Can be based on Dynamically adjust the sampling range.

[0105] Specifically, after determining the target predicted sampling probability corresponding to the target index position, the database can determine the initial sampling range corresponding to the target index position, and based on the target predicted sampling probability, expand or reduce the initial sampling range to obtain the target sampling range.

[0106] The target sampling range is determined from the index range of the target data table, and the to-be-sampled data corresponding to the target sampling range is obtained as the target data corresponding to the target index position.

[0107] Continuing with the above example, use the following formula to adjust the sampling range.

[0108]

[0109] when When using To sample, when When , we use 1 / β to sample approximately. Where β is a preset parameter. In actual data, we find that β = 8 is a stable choice for sparse tables.

[0110] In one embodiment provided in this specification, it is known that We can use the normalized probability But this requires calculating for all n Since M is often very large, a single calculation may exhaust the entire sampling budget. Therefore, the data sampling method provided in this specification proposes the use of an accept-reject method for sampling. Specifically, based on the target predicted sampling probability, the data to be sampled at the target index position is sampled to obtain the target data, including:

[0111] Determining a selected target index position from the at least two target index positions based on target predicted sampling probabilities corresponding to the at least two target index positions;

[0112] The data to be sampled at the selected target index position is sampled to obtain target data.

[0113] Specifically, based on at least two target index positions, after determining the target predicted sampling probability corresponding to each target index position, a selected target index position is determined from the at least two target index positions based on the target predicted sampling probability; then, a sampling range corresponding to the selected target index position is determined (for example, the above-mentioned initial sampling range or target sampling range), and the data to be sampled corresponding to the sampling range is obtained from the target data table as the target data corresponding to the selected target index position.

[0114] Continuing with the above example, for each index position n, determine the corresponding Afterwards, based on the The selected index position n is determined from the index position n by the accept-reject method; then the sampling range corresponding to the selected target index position n is determined, and the data to be sampled corresponding to the sampling range is obtained from the sparse table as the sampling data of the selected index position n.

[0115] Furthermore, determining a selected target index position from the at least two target index positions based on the target predicted sampling probabilities corresponding to the at least two target index positions includes:

[0116] Determining preset selection rules for at least two target index positions, and preset selection thresholds corresponding to the preset selection rules;

[0117] determining a preset sampling probability for the at least two target index positions;

[0118] Calculate the selection parameters corresponding to each target index position according to the target predicted sampling probability and the preset sampling probability;

[0119] A target selection parameter that meets the preset selection threshold is determined from the selection parameters, and the target index position corresponding to the target selection parameter is determined as the selected target index position.

[0120] In practical applications, when the accept-reject method is used for sampling, a point n (i.e., index position) is randomly selected first, and the corresponding (can be obtained by the above prediction model), and then generate a random number Z∈[0,1], if Then accept (accept) this sampling and perform sampling for position n, otherwise reject (reject) this sampling, and the rejection process will continue until there is an accepted sampling. t is the maximum value of p(n).

[0121] The preset selection rule can be understood as the accept-reject method, the preset selection threshold can be understood as a random number Z∈[0,1], and the preset sampling probability can be understood as V t , the selected parameters can be understood as The selection parameter meeting the preset selection threshold can be understood as the selection parameter being greater than or equal to the preset selection threshold.

[0122] In one embodiment provided in this specification, when the sampling budget is limited, 10% of the sampling budget can be used for Bayesian optimization to improve the accuracy of the prediction model. Specifically, the following steps are performed:

[0123] The data sampling method provided in this specification, after receiving the data sampling request for the target data table, further includes:

[0124] Determining a corresponding random index position for the prediction model based on the target data table;

[0125] Obtaining, according to the random index position, an initial predicted sampling probability corresponding to the random index position using a prediction model, wherein the prediction model is trained based on sample pairs, the sample pairs including a sample index position and a predicted sampling probability of the sample corresponding to the sample index position;

[0126] Based on the initial predicted sampling probability, sampling the data to be sampled at the random index position in the target data table to obtain model optimization data;

[0127] The prediction model is optimized based on the model optimization data to obtain an optimized prediction model.

[0128] The model optimization request can be understood as a request to perform Bayesian optimization on the model. The model optimization data can be understood as record data sampled from the target data table based on a random index position. The random index position can be understood as an index position randomly determined based on the index range of the target data table, and this random index position can be determined based on Bayesian optimization.

[0129] In practice, the data sampling method provided in this specification uses a variant of Bayesian optimization, assuming that the Gaussian process regression model estimates the mean and variance of each n as μ n and σ n , then randomly sample, for example, 1000 n, and then sample μ respectively n +σ n The largest point (i.e. index position) and σ n Among them, μ n +σ n The largest point can be understood as μ n and σn are the largest index positions respectively, and the operation of determining the index position is Bayesian optimization, in order to find the maximum value V of p(n) t , the latter σ n The purpose is to improve the prediction accuracy of the Gaussian process regression model.

[0130] Specifically, the database randomly determines a corresponding random index position for the prediction model based on the index range of the target data table, and the random index position is within the index range; the random index position is input into the prediction model to obtain an initial prediction sampling probability corresponding to the random index position; based on the initial prediction sampling probability, the data to be sampled at the random index position in the target data table is sampled to obtain model optimization data; then, the prediction model is optimized based on the model optimization data to obtain an optimized prediction model.

[0131] Furthermore, determining a corresponding random index position for the prediction model based on the target data table includes:

[0132] Determining a data sampling resource corresponding to the data sampling request;

[0133] Determining a preset processing rule for the data sampling request and processing parameters corresponding to the preset processing rule;

[0134] When it is determined that the data sampling resource is not empty and the processing parameter does not satisfy the preset processing rule, a corresponding random index position is determined for the prediction model based on the index range of the target data table.

[0135] The fact that the processing parameter does not satisfy the preset processing condition can be understood as determining that the processing parameter does not satisfy the preset processing condition when the processing parameter is less than or equal to a preset threshold (such as 0.1).

[0136] Continuing with the above example, the database will determine if the processing parameter is less than or equal to a preset threshold (such as 0.1). If so, it will randomly determine the corresponding random index position for the prediction model based on the index range of the target data table.

[0137] In one embodiment provided in this specification, determining a target random index position from the random index positions based on the initial predicted sampling probability, and sampling the data to be sampled at the target random index position to obtain model optimization data includes:

[0138] Determining an initial predicted sampling probability and an initial probability prediction parameter corresponding to each random index position in at least two random index positions, wherein the initial probability prediction parameter is obtained using the prediction model;

[0139] Determining a target random index position from the at least two random index positions based on the initial predicted sampling probability and the initial probability prediction parameter;

[0140] Determine the random sampling range corresponding to the target random index position, and obtain the data to be sampled corresponding to the random sampling range from the target data table as model optimization data.

[0141] Among them, the initial probability prediction parameter can be understood as the variance of the prediction model output, and the target random index position can be understood as μ in the random index position. n and σ n are the maximum index positions respectively.

[0142] Following the above example, the database inputs multiple random index positions into the Gaussian process regression model to obtain the μ corresponding to the random index position. n and σ n , from multiple random index positions, determine μ n and σ n are respectively the largest random index positions, and the sampling range corresponding to the random index position is determined, the sampling range is determined from the sparse table, and the recorded data corresponding to the sampling range is obtained, and the recorded data is used as the data required in the process of optimizing the Gaussian process regression model, thereby realizing subsequent optimization of the prediction model and improving the accuracy of the prediction model.

[0143] In the embodiments provided in this specification, optimizing the prediction model based on the model optimization data to obtain the optimized prediction model includes:

[0144] Determining an actual sampling probability corresponding to the random index position based on the model optimization data;

[0145] Based on the actual sampling probability and the initial predicted sampling probability, adjusting the initial probability prediction parameter corresponding to the random index position to obtain a target probability prediction parameter;

[0146] The pre-trained prediction model is optimized based on the target probability prediction parameters to obtain an optimized prediction model.

[0147] Continuing with the above example, after the database obtains the sampling data based on the random index position, it determines the actual sampling probability corresponding to the random index position based on the sampling data, and adjusts the variance parameter of the prediction model based on the actual sampling probability and the prediction model to obtain the predicted sampling probability. The Gaussian process regression model is updated and optimized based on the adjusted variance to obtain the optimized Gaussian process regression model, thereby improving the prediction accuracy of the prediction model.

[0148] In the data processing method provided in this specification, after obtaining the target data, statistical information can be constructed based on the target data. Specifically, based on the target predicted sampling probability, the data to be sampled at the target index position is sampled. After obtaining the target data, the method further includes:

[0149] Statistical information corresponding to the target data table is constructed based on the target data.

[0150] Continuing with the above example, after sampling the target data table through the target index position to obtain the target data, the target data is constructed into a histogram.

[0151] The data sampling method provided in this specification, when receiving a data sampling request for a target data table, uses a prediction model to determine the corresponding target predicted sampling probability for the target index position carried in the data sampling request, and samples the data to be sampled in the target data table based on the target predicted sampling probability, thereby significantly improving sampling efficiency and sampling accuracy.

[0152] The following combined Figure 4 , taking the application of the data sampling method provided in this specification in the scenario of constructing statistical information of a relational database based on active sampling as an example, the data sampling method is further explained. Figure 4 The flowchart of the processing process of a data sampling method provided by an embodiment of the present specification is shown. In practical applications, for a sparse table, it is assumed that the sparse table T has N rows, and the value range of the index column of the sparse table T is [1, M], N<<M. For any index position n∈[1, M], if there is a function p(n) that can tell the database the probability of having a data record at position n, then the database can use this probability to perform sampling more efficiently than even sampling. To achieve this, first, p(n) is defined, and then the Gaussian process regression model is used to estimate And the accept-reject method is used to achieve efficient sampling. During this period, a variant of Bayesian optimization is used to continuously improve the accuracy of the estimation and the maximum value of the search, so that the accept-reject method can approach the optimal sampling. At the same time, an adaptive sampling range adjustment method is proposed to further improve sampling efficiency. Finally, due to the need to perform more sampling in large areas, which brings about sampling bias, this solution proposes the use of inverse probability weighting to correct the sampling bias in the process of constructing statistical information. Specifically, it includes the following steps.

[0153] Step 402: Initially perform uniform sampling and record the corresponding p(n).

[0154] Specifically, before sampling a sparse table, the Gaussian process regression model needs to be trained. This training step can be the initial step of the entire statistical information construction, specifically including: the database allocates a portion of the sampling budget (for example, 10%) from the sampling budget allocated to the data sampling task to perform uniform sampling to obtain initial data. The uniform sampling process is as described in the above embodiment, that is, given a sparse table T in a database instance, the value range of the index column of the sparse table T is 1 to 1000000. Assuming that the sampling range of each sampling is " / ", we can randomly select a number n in [1, M], where M is equal to 1000000. Then use the following SQL statement to obtain the elements in the table.

[0155] SELECT*FROM T WHERE id BETWEEN n - / / 2 AND n + / / 2;

[0156] The "2" in the above formula can be understood as a parameter for adjusting the sampling range. The data sampling method provided in this specification estimates the probability of a data record at position n by using the recorded data at other positions within the sampling range corresponding to the sample index position. If the above SQL returns m records, the following formula is used to define the predicted sampling probability corresponding to the sample index position.

[0157]

[0158] In the initial stage, a series of sampling locations n are randomly selected, the corresponding p(n) is calculated and recorded until the initial sampling budget is used up, thus ending the data sampling in the initial stage.

[0159] Step 404 : Determine whether there is a sampling budget. If so, execute step 406 ; if not, execute step 420 .

[0160] In practical applications, the sampling budget allocated to each data sampling task is limited. Therefore, during the active sampling process, it is necessary to determine whether there is still sampling budget. If so, it is determined that there is still sampling budget remaining, and step 406 is executed to continue using active sampling. If not, it is determined that the sampling budget has been used up, and step 420 is executed to end sampling.

[0161] Step 406: Train the Gaussian process regression model.

[0162] Specifically, the sampling position n recorded in the initial stage is used as the sample data, and the p(n) corresponding to the sampling position n is used as the sample label. The Gaussian process regression model is trained based on the sample data and the sample label until the Gaussian process regression model reaches the training stop condition.

[0163] The trained Gaussian process regression model can be used to estimate the probability of all positions, which is recorded as At the same time, it should be noted that during the model optimization process, since only a new n and p(n) are added for each sampling, the Gaussian process regression model does not need to be updated after each sampling step. The Gaussian process model can be updated periodically, such as updating the model after every 32 or 64 sampling times. This can reduce the cost of Gaussian process training.

[0164] Furthermore, the data sampling method provided in this specification selects the Gaussian process regression model because the model can not only estimate the function value, but also estimate the corresponding variance (uncertainty), which is crucial for the Bayesian optimization to be used later.

[0165] Step 408: Determine whether Bayesian optimization is used. If so, execute step 410; if not, execute step 412.

[0166] Specifically, the sampling resources allocated to a data sampling task by the database are limited. When the sampling budget is limited, on the one hand, more sampling data is needed to construct statistical information, and on the other hand, we need to focus on predicting the sampling probability. accuracy.

[0167] Therefore, the database will have a parameter to control the proportion of the sampling budget. For example, 10% of the sampling budget can be used for Bayesian optimization to improve the accuracy of the prediction model, and 90% of the sampling can be used to sample data to construct statistical information. In this way, within a limited sampling budget, it is possible to improve the accuracy of the model while completing the construction of statistical information.

[0168] Based on this, the way to determine whether to use Bayesian optimization can be:

[0169] 1. Determine the random parameters that control the proportional distribution of the sampling budget.

[0170] The random parameter may be a random value in the interval [0, 1], wherein the random parameter has a 10% probability of being distributed in the interval [0, 0.1] and a 90% probability of being distributed in the interval (0.1, 1).

[0171] 2. When the database determines that the random parameter is greater than the preset threshold (0.1), the database performs an active sampling step, which is step 412.

[0172] Alternatively, when the database determines that the random parameter is less than or equal to the preset threshold (0.1), the database performs the Bayesian optimization step, that is, step 410 .

[0173] Step 410: Sample the point with the largest estimated variance and estimated value, and record the corresponding p(n).

[0174] Specifically, the data sampling method provided in this specification uses a variant of Bayesian optimization, assuming that the estimated mean and variance of each n in the Gaussian process regression model is μ n and σ n , where the estimated mean μ n It can be understood as the above predicted sampling probability

[0175] After that, randomly sample, for example, 1000 n, and then sample μ separately n +σ n The largest point (i.e. index position) and σ n Among them, μ n +σ n The largest point can be understood as μ n and σ n are the largest index positions respectively; and the operation of determining the index position is Bayesian optimization, which is to find the maximum value V of p(n) t , the latter σ n The purpose is to improve the prediction accuracy of the Gaussian process regression model.

[0176] The specific way to perform Bayesian optimization is as follows:

[0177] The database inputs multiple randomly obtained random index positions n into the Gaussian process regression model to obtain μ corresponding to the random index position n and σ n .

[0178] Then, from multiple random index positions, determine μ n and σ n They are respectively the maximum random index position MAX.

[0179] After executing the Bayesian optimization, first, the sampling range corresponding to the random index position MAX is determined, and the record data corresponding to the sampling range is obtained from the sparse table.

[0180] Secondly, the actual sampling probability corresponding to the random index position MAX is determined based on the recorded data, and the actual sampling probability and the estimated mean μ are used to calculate the probability of the random index position MAX. n σ n Make adjustments.

[0181] Finally, based on the adjusted σ n The parameters of the Gaussian process regression model are adjusted to achieve update and optimization of the Gaussian process regression model.

[0182] Step 412: Dynamically adjust the sampling range.

[0183] Specifically, for sparse tables, a probability distribution function is known You can use Get more efficient sampling. When the database knows that a certain area has a higher probability of sampling data, it can not only implement more sampling in that area, but also dynamically adjust the sampling range.

[0184] For sparse tables, when The smaller it is, the more likely it is that no data will be sampled, but when When the value is larger, the data in the same sampling range are not independent and identically distributed, which will significantly reduce the accuracy of the statistical information to be constructed later. Can be based on Dynamically adjust the sampling range corresponding to each index position n.

[0185] Specifically use the following formula to adjust the sampling range.

[0186]

[0187] when When using To sample, when When , we use 1 / β to sample approximately. Where β is a preset parameter. In actual data, we find that β = 8 is a stable choice for sparse tables.

[0188] Step 414: Sampling using the accept-reject method.

[0189] In practical applications, After that, the normalized probability can be used But this requires calculating for all n Since M is often very large, a single calculation may use up all the sampling budget. Therefore, the data sampling method provided in this specification proposes the use of the accept-reject method for sampling.

[0190] The specific way to use the accept-reject method for sampling is:

[0191] When the accept-reject method is used for sampling, first, a point n (i.e., index position) is randomly selected and the corresponding (which can be obtained by calculating the above Gaussian process regression model), and then generate a random number Z∈[0,1].

[0192] if Then accept (accept) this sampling and perform sampling for position n, otherwise reject (reject) this sampling, and the rejection process will continue until there is an accepted sampling. t is the maximum value of p(n).

[0193] Step 416: Record sampling data and sampling probability.

[0194] Specifically, in the final sampling range You need to record all returned data and the corresponding sampling probability Only the data sampled in this step will be used to construct statistics.

[0195] Step 418: The first sampling is completed and the sampling cost is calculated.

[0196] Specifically, after a sampling is completed, the database will calculate the sampling cost and execute step 404 to determine whether there is still a sampling budget.

[0197] Step 420: Sampling is completed and statistical information is constructed.

[0198] Specifically, when the sampling budget is exhausted, the database uses the sampled data to construct the statistics of the target column. Here, inverse probability weighting is used to correct the sampling offset. For example, each sampled data is not recorded as 1 when calculating the histogram, but is recorded as Second-rate

[0199] Based on the above steps, given a database table, its corresponding index columns, the columns for which statistics need to be generated, and the sampling budget required for sampling the database table (such as total sampling time or database response time), the data sampling method provided in this specification can execute SQL sampling on the database within a limited sampling budget to obtain statistical information, such as a histogram, for the target columns.

[0200] Furthermore, the data sampling method provided in this specification has been verified in actual applications and it is found that, compared with uniform sampling, 30% to 50% of the sampling budget can be used to obtain the same statistical information accuracy.

[0201] The data sampling method provided in this specification proposes an active sampling method based on Bayesian optimization for constructing sparse table statistics. By utilizing techniques such as accept-reject sampling, Bayesian optimization, inverse probability weighting, and adaptive sampling range adjustment, the database can sample more data to construct statistical information at a limited sampling cost. Furthermore, the data sampling method provided in this specification theoretically guarantees that unbiased estimates can be obtained for constructing statistical information. This method, compared to uniform sampling, achieves higher sampling accuracy at a lower sampling cost, significantly improving the sampling efficiency of traditional uniform sampling.

[0202] It should also be noted that the data sampling method provided in this specification uses SQL to construct table statistics outside the database kernel and is applicable to all relational databases. An alternative to the data sampling method provided in this specification is to implement the statistics module within the database kernel. However, this solution requires implementing the corresponding modules for different versions of different open source databases, and users must upgrade to the corresponding version to use it, making it not universally applicable.

[0203] Corresponding to the above method embodiment, this specification also provides a data sampling device embodiment, Figure 5 FIG1 shows a schematic diagram of the structure of a data sampling device provided by an embodiment of this specification. Figure 5 As shown, the device includes:

[0204] The receiving module 502 is configured to receive a data sampling request for a target data table, wherein the data sampling request carries a target index position;

[0205] A prediction module 504 is configured to obtain, based on the target index position, a target predicted sampling probability corresponding to the target index position using a prediction model, wherein the prediction model is trained based on sample pairs, the sample pairs including a sample index position and a sample predicted sampling probability corresponding to the sample index position;

[0206] The sampling module 506 is configured to sample the data to be sampled at the target index position based on the target predicted sampling probability to obtain target data.

[0207] Optionally, the data sampling device further includes a model training module configured to:

[0208] Determine an index range of the sample data table, and determine a sample index position based on the index range, and a predicted sampling probability corresponding to the sample index position;

[0209] Using the sample index position as a training sample, and using the predicted sampling probability corresponding to the sample index position as the sample label corresponding to the training sample;

[0210] A sample pair is constructed based on the training samples and the corresponding sample labels, and the prediction model to be trained is trained according to the sample pair to obtain a trained prediction model.

[0211] Optionally, the model training module is further configured to:

[0212] Determining a sample index position and a sampling range corresponding to the sample index position based on the index range, and determining sample sampling data corresponding to the sample index position based on the sampling range;

[0213] Based on the attribute information of the sample sampling data and the sampling range, a predicted sampling probability corresponding to the sample index position is determined.

[0214] Optionally, the model training module is further configured to:

[0215] Determining a preset uniform sampling rule corresponding to the sample data table, and selecting a sample index position from the sample data table based on the preset uniform sampling rule;

[0216] Determining a preset range parameter corresponding to the sample index position, and determining a corresponding sample sampling range for the sample index position from the index range based on the preset range parameter;

[0217] The sample data to be sampled corresponding to the sample sampling range is obtained from the sample data table as the sample sampling data corresponding to the sample index position.

[0218] Optionally, the prediction module 504 is further configured to:

[0219] Determining a data sampling resource corresponding to the data sampling request;

[0220] Determining a preset processing rule for the data sampling request and processing parameters corresponding to the preset processing rule;

[0221] When it is determined that the data sampling resource is not empty and the processing parameters meet the preset processing rules, the target index position is input into the prediction model to obtain the target prediction sampling probability corresponding to the target index position.

[0222] Optionally, the sampling module 506 is further configured to:

[0223] Determine an initial sampling range corresponding to the target index position, and adjust the initial sampling range based on the target predicted sampling probability to obtain a target sampling range;

[0224] The data to be sampled corresponding to the target sampling range is obtained from the target data table as the target data corresponding to the target index position.

[0225] Optionally, the sampling module 506 is further configured to:

[0226] Determining a selected target index position from the at least two target index positions based on target predicted sampling probabilities corresponding to the at least two target index positions;

[0227] The data to be sampled at the selected target index position is sampled to obtain target data.

[0228] Optionally, the sampling module 506 is further configured to:

[0229] Determining preset selection rules for at least two target index positions, and preset selection thresholds corresponding to the preset selection rules;

[0230] determining a preset sampling probability for the at least two target index positions;

[0231] Calculate the selection parameters corresponding to each target index position according to the target predicted sampling probability and the preset sampling probability;

[0232] A target selection parameter that meets the preset selection threshold is determined from the selection parameters, and the target index position corresponding to the target selection parameter is determined as the selected target index position.

[0233] Optionally, the data sampling device further includes a model optimization module configured to:

[0234] Determining a corresponding random index position for the prediction model based on the target data table;

[0235] Obtaining, according to the random index position, an initial predicted sampling probability corresponding to the random index position using a prediction model, wherein the prediction model is trained based on sample pairs, the sample pairs including a sample index position and a predicted sampling probability of the sample corresponding to the sample index position;

[0236] Determining a target random index position from the random index positions based on the initial predicted sampling probability, and sampling the data to be sampled at the target random index position to obtain model optimization data;

[0237] The prediction model is optimized based on the model optimization data to obtain an optimized prediction model.

[0238] Optionally, the model optimization module is further configured to:

[0239] Determining a data sampling resource corresponding to the data sampling request;

[0240] Determining a preset processing rule for the data sampling request and processing parameters corresponding to the preset processing rule;

[0241] When it is determined that the data sampling resource is not empty and the processing parameter does not satisfy the preset processing rule, a corresponding random index position is determined for the prediction model based on the index range of the target data table.

[0242] Optionally, the sample data table is the target data table, or a subset of the target data table.

[0243] Optionally, the data sampling device further includes a statistical information building module configured to:

[0244] Statistical information corresponding to the target data table is constructed based on the target data.

[0245] The above is a schematic diagram of a data sampling device according to this embodiment. It should be noted that the technical solution of the data sampling device and the technical solution of the above-mentioned data sampling method are based on the same concept. For details not described in detail in the technical solution of the data sampling device, please refer to the description of the technical solution of the above-mentioned data sampling method.

[0246] Figure 6 6 shows a block diagram of a computing device 600 according to one embodiment of the present disclosure. Components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0247] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0248] In one embodiment of the present specification, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 6 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0249] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 600 may also be a mobile or stationary server.

[0250] The processor 620 is configured to execute the following computer-executable instructions, which implement the steps of the above-mentioned data sampling method when executed by the processor 620.

[0251] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the above-mentioned data sampling method are based on the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-mentioned data sampling method.

[0252] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which implement the steps of the above-mentioned data sampling method when executed by a processor.

[0253] The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data sampling method described above are based on the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data sampling method described above.

[0254] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data sampling method.

[0255] The above is an illustrative solution of a computer program of this embodiment. It should be noted that the technical solution of the computer program and the technical solution of the above-mentioned data sampling method are based on the same concept. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the above-mentioned data sampling method.

[0256] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0257] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0258] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0259] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0260] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data sampling method, comprising: receiving a data sampling request for a target data table, wherein the data sampling request carries a target index position; According to the target index position, using a prediction model to obtain a target predicted sampling probability corresponding to the target index position, wherein the prediction model is trained based on sample pairs, and the sample pairs include a sample index position and a sample predicted sampling probability corresponding to the sample index position; Based on the target predicted sampling probability, sampling the data to be sampled at the target index position to obtain target data; Optimize forecasting models, including: Determining a corresponding random index position for the prediction model based on the target data table; According to the random index position, using a prediction model to obtain an initial predicted sampling probability corresponding to the random index position; Determining a target random index position from the random index positions based on the initial predicted sampling probability, and sampling the data to be sampled at the target random index position to obtain model optimization data; Optimizing the prediction model based on the model optimization data to obtain an optimized prediction model; Wherein, determining a corresponding random index position for the prediction model based on the target data table includes: Determining a data sampling resource corresponding to the data sampling request; Determining a preset processing rule for the data sampling request and processing parameters corresponding to the preset processing rule; When it is determined that the data sampling resource is not empty and the processing parameter does not satisfy the preset processing rule, a corresponding random index position is determined for the prediction model based on the index range of the target data table.

2. The data sampling method according to claim 1, before obtaining the target predicted sampling probability corresponding to the target index position using a prediction model based on the target index position, further comprising: Determine an index range of the sample data table, and determine a sample index position based on the index range, and a predicted sampling probability corresponding to the sample index position; Using the sample index position as a training sample, and using the predicted sampling probability corresponding to the sample index position as the sample label corresponding to the training sample; A sample pair is constructed based on the training samples and the corresponding sample labels, and the prediction model to be trained is trained according to the sample pair to obtain a trained prediction model.

3. The data sampling method according to claim 2, wherein determining the sample index position based on the index range and the predicted sampling probability corresponding to the sample index position comprises: Determining a sample index position and a sampling range corresponding to the sample index position based on the index range, and determining sample sampling data corresponding to the sample index position based on the sampling range; Based on the attribute information of the sample sampling data and the sampling range, a predicted sampling probability corresponding to the sample index position is determined.

4. The data sampling method according to claim 3, wherein determining a sample index position and a sampling range corresponding to the sample index position based on the index range, and determining the sample data corresponding to the sample index position based on the sampling range comprises: Determining a preset uniform sampling rule corresponding to the sample data table, and selecting a sample index position from the sample data table based on the preset uniform sampling rule; Determining a preset range parameter corresponding to the sample index position, and determining a corresponding sample sampling range for the sample index position from the index range based on the preset range parameter; The sample data to be sampled corresponding to the sample sampling range is obtained from the sample data table as the sample sampling data corresponding to the sample index position.

5. The data sampling method according to claim 1, wherein obtaining the target predicted sampling probability corresponding to the target index position using a prediction model according to the target index position comprises: Determining a data sampling resource corresponding to the data sampling request; Determining a preset processing rule for the data sampling request and processing parameters corresponding to the preset processing rule; When it is determined that the data sampling resource is not empty and the processing parameters meet the preset processing rules, the target index position is input into the prediction model to obtain the target prediction sampling probability corresponding to the target index position.

6. The data sampling method according to claim 1, wherein sampling the data to be sampled at the target index position based on the target predicted sampling probability to obtain the target data comprises: Determine an initial sampling range corresponding to the target index position, and adjust the initial sampling range based on the target predicted sampling probability to obtain a target sampling range; The data to be sampled corresponding to the target sampling range is obtained from the target data table as the target data corresponding to the target index position.

7. The data sampling method according to claim 1, wherein sampling the data to be sampled at the target index position based on the target predicted sampling probability to obtain the target data comprises: Determining a selected target index position from the at least two target index positions based on target predicted sampling probabilities corresponding to the at least two target index positions; The data to be sampled at the selected target index position is sampled to obtain target data.

8. The data sampling method according to claim 7, wherein determining a selected target index position from the at least two target index positions based on target predicted sampling probabilities corresponding to the at least two target index positions comprises: Determining preset selection rules for at least two target index positions, and preset selection thresholds corresponding to the preset selection rules; determining a preset sampling probability for the at least two target index positions; Calculate the selection parameters corresponding to each target index position according to the target predicted sampling probability and the preset sampling probability; A target selection parameter that meets the preset selection threshold is determined from the selection parameters, and the target index position corresponding to the target selection parameter is determined as the selected target index position. 9 . The data sampling method according to claim 2 , wherein the sample data table is the target data table or a subset of the target data table.

10. The data sampling method according to claim 1, wherein after sampling the data to be sampled at the target index position based on the target predicted sampling probability and obtaining the target data, the method further comprises: Statistical information corresponding to the target data table is constructed based on the target data.

11. A computing device comprising: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the data sampling method according to any one of claims 1 to 10 are implemented.

12. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the data sampling method according to any one of claims 1 to 10.

Images