Data sampling

Abstract

A data sampling method and apparatus, and an electronic device and a storage medium, which relate to the technical field of data processing. The data sampling method comprises: on the basis of a received data sampling instruction, determining a target storage area to be subjected to sampling; dividing said target storage area to obtain a plurality of logic sub-intervals; on the basis of a sampling probability and interval ranges respectively corresponding to the logic sub-intervals, determining target sampling numbers respectively corresponding to the logic sub-intervals; performing screening to obtain logic sub-intervals, the target sampling numbers of which are not zero; and performing data sampling on each of the selected logic sub-intervals, so as to obtain target sampling data respectively corresponding to the logic sub-intervals. In this way, by means of performing interval division on a target storage area, whether each logic sub-interval is required to be subjected to sampling can be quickly determined, thereby reducing sampling operations and improving efficiency.

Description

Data Sampling Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for data sampling. Background Art

[0002] In database applications, it is often necessary to sample the data stored in the database. Sampling is used to obtain an approximate distribution of the overall data using partial data.

[0003] In the existing technology, a hash value corresponding to each row is usually generated, and data sampling is performed based on the hash value of each row. However, this method requires scanning all the data to be sampled, which has low sampling efficiency. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a data sampling method, apparatus, electronic device, and storage medium, so as to improve the efficiency of data sampling when performing data sampling.

[0005] On the one hand, an embodiment of the present application provides a method for data sampling, including: determining a target storage area to be sampled based on a received data sampling instruction; dividing the target storage area to obtain multiple logical sub-intervals; determining the target sampling number corresponding to each logical sub-interval based on the sampling probability and the interval range corresponding to each logical sub-interval; screening out logical sub-intervals whose target sampling number is not zero; and performing data sampling on each screened logical sub-interval to obtain target sampling data corresponding to each logical sub-interval.

[0006] In one embodiment, the target storage area includes at least one data record; each data record is stored row storage data or an incremental operation record for row storage data; each row storage data and its corresponding incremental operation record correspond to the same logical row; the target storage area is divided into intervals and data is sampled according to logical rows.

[0007] In one embodiment, the target storage area is divided to obtain multiple logical sub-intervals, including looping the following steps until the interval division condition is reached: the target storage area is divided into intervals according to the sub-interval size; when it is determined that the size adjustment condition is met, the sub-interval size is adjusted.

[0008] In one embodiment, the target sampling number of each logical sub-interval is determined separately, including performing the following steps for each logical sub-interval: determining the first row of the sub-interval and the last row of the sub-interval of the logical sub-interval; rounding down the product of the last row of the sub-interval and the sampling probability to obtain a first sampling number; rounding down the product of the first row of the sub-interval and the sampling probability to obtain a second sampling number; and obtaining the target sampling number of the logical sub-interval based on the difference between the first sampling number and the second sampling number.

[0009] In one embodiment, data sampling is performed on each of the screened logical sub-intervals to obtain target sampling data corresponding to each of the logical sub-intervals, including performing the following steps for each of the screened logical sub-intervals: determining a sampling offset corresponding to the logical sub-interval; determining a sampling start row corresponding to the logical sub-interval based on the sum of the first row of the sub-interval and the sampling offset; determining a sampling end row corresponding to the logical sub-interval based on the sum of the sampling start row and the target number of samples corresponding to the logical sub-interval; determining a sampling interval of the logical sub-interval based on the sampling start row and the sampling end row; and using the data within the sampling interval as the target sampling data corresponding to the logical sub-interval.

[0010] In one embodiment, determining the sampling offset corresponding to the logical sub-interval includes: performing a hash operation based on the interval number of the logical sub-interval and a specified hash random seed value to obtain a hash value; determining a sampling difference based on the difference between the sub-interval size corresponding to the logical sub-interval and the target sampling number corresponding to the logical sub-interval; and performing a modulo operation based on the hash value and the sampling difference to obtain a sampling offset.

[0011] In one embodiment, the target storage area corresponds to an index tree of at least one ordered string table; the index tree contains multiple nodes; the leaf nodes in the index tree are data microblocks; the non-leaf nodes in the index tree are index microblocks; each data microblock is used to store data of each logical row in the target storage area; the index microblock contains index information of the child node, which is used for querying and reading data.

[0012] In one embodiment, before data sampling is performed on each filtered logical sub-interval and the target sampling data corresponding to each logical sub-interval is obtained, the method also includes: performing the following steps for each node within the primary key query range in each index tree: determining the current row scan total displacement of at least one index tree; the row scan total displacement is determined based on the number of node rows of each node currently scanned; if the node is a non-filtered node, determining the node row interval of the node based on the row scan total displacement and the number of node rows of the node; if the node row interval does not intersect with all sampling intervals and the primary key value corresponding to the node does not exist in other ordered string tables or memory tables, determining the node as a filtered node, otherwise, pre-fetching the node into the memory so that data sampling is performed through the node in the memory.

[0013] On the one hand, an embodiment of the present application provides a data sampling device, including: a receiving unit, used to determine a target storage area to be sampled according to a received data sampling instruction; a dividing unit, used to divide the target storage area to obtain multiple logical sub-intervals; a determining unit, used to determine the target sampling number corresponding to each logical sub-interval according to the sampling probability and the interval range corresponding to each logical sub-interval; a screening unit, used to screen out logical sub-intervals whose target sampling number is not zero; and a sampling unit, used to perform data sampling on each screened logical sub-interval to obtain target sampling data corresponding to each logical sub-interval.

[0014] In one embodiment, the target storage area includes at least one data record; each data record is stored row storage data or an incremental operation record for row storage data; each row storage data and its corresponding incremental operation record correspond to the same logical row; the target storage area is divided into intervals and data is sampled according to logical rows.

[0015] In one embodiment, the partitioning unit is configured to cyclically execute the following steps until a condition for ending the interval partitioning is met: partitioning the target storage area according to sub-interval sizes; and adjusting the sub-interval sizes when a size adjustment condition is determined to be met.

[0016] In one embodiment, the determination unit is used to perform the following steps for each logical sub-interval respectively: determine the first row of the sub-interval and the last row of the sub-interval of the logical sub-interval; round down the product of the last row of the sub-interval and the sampling probability to obtain a first sampling number; round down the product of the first row of the sub-interval and the sampling probability to obtain a second sampling number; and obtain the target sampling number of the logical sub-interval based on the difference between the first sampling number and the second sampling number.

[0017] In one embodiment, the sampling unit is used to perform the following steps for each logical sub-interval screened out: determine the sampling offset corresponding to the logical sub-interval; determine the sampling starting row corresponding to the logical sub-interval based on the sum of the first row of the sub-interval and the sampling offset; determine the sampling last row corresponding to the logical sub-interval based on the sum of the sampling starting row and the target sampling number corresponding to the logical sub-interval; determine the sampling interval of the logical sub-interval based on the sampling starting row and the sampling last row; and use the data in the sampling interval as the target sampling data corresponding to the logical sub-interval.

[0018] In one embodiment, the sampling unit is used to: perform a hash operation based on the interval number of the logical sub-interval and a specified hash random seed value to obtain a hash value; determine a sampling difference value based on the difference between the sub-interval size corresponding to the logical sub-interval and the target sampling number corresponding to the logical sub-interval; and perform a remainder operation based on the hash value and the sampling difference value to obtain a sampling offset.

[0019] In one embodiment, the target storage area corresponds to an index tree of at least one ordered string table; the index tree contains multiple nodes; the leaf nodes in the index tree are data microblocks; the non-leaf nodes in the index tree are index microblocks; each data microblock is used to store data of each logical row in the target storage area; the index microblock contains index information of the child node, which is used for querying and reading data.

[0020] In one embodiment, the sampling unit is further used to: perform the following steps for each node within the primary key query range in each index tree: determine the current total row scan displacement of at least one index tree; the total row scan displacement is determined based on the number of node rows of each node currently scanned; if the node is a non-filtered node, determine the node row interval of the node based on the total row scan displacement and the number of node rows of the node; if the node row interval does not intersect with all sampling intervals and the primary key value corresponding to the node does not exist in other ordered string tables or memory tables, determine the node as a filter node, otherwise, pre-fetch the node into the memory so that data sampling is performed through the node in the memory.

[0021] On the one hand, an embodiment of the present application provides an electronic device, including a processor and a memory storing computer instructions, where the computer instructions are used to enable the processor to execute the steps of the method provided in any of the various optional implementations of data sampling described above.

[0022] On the one hand, an embodiment of the present application provides a storage medium storing computer instructions, which are used to enable a computer to execute the steps of the method provided in any of the various optional implementations of data sampling as described above.

[0023] In the data sampling method, device, electronic device, and storage medium provided in the embodiments of the present application, a target storage area to be sampled is determined based on a received data sampling instruction; the target storage area is divided to obtain multiple logical sub-intervals; the target sampling quantity corresponding to each logical sub-interval is determined based on the sampling probability and the interval range corresponding to each logical sub-interval; logical sub-intervals with a non-zero target sampling quantity are screened out; and data is sampled for each screened out logical sub-interval to obtain target sampling data corresponding to each logical sub-interval. In this way, the target storage area is divided into intervals, so that it is possible to quickly determine whether each logical sub-interval needs to be sampled, reducing sampling operations and improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] FIG1 is a flow chart of a data sampling method according to an embodiment of the present application.

[0026] FIG2 is an example diagram of an operator node in an embodiment of the present application.

[0027] FIG3 is an example diagram of an index tree in an embodiment of the present application.

[0028] FIG4 is an example diagram of a node storage information update according to an embodiment of the present application.

[0029] FIG5 is a structural block diagram of a data sampling device in an embodiment of the present application.

[0030] FIG6 is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solutions of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0032] First, some of the terms involved in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0033] Terminal device: can be a mobile terminal, fixed terminal or portable terminal, such as a mobile phone, station, unit, device, multimedia computer, multimedia tablet, Internet node, communicator, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, personal communication system device, personal navigation device, personal digital assistant, audio / video player, digital camera / camcorder, positioning device, television receiver, radio broadcast receiver, e-book device, gaming device or any combination thereof, including accessories and peripherals of these devices or any combination thereof. It is also foreseeable that the terminal device can support any type of user interface (such as wearable device), etc.

[0034] Server: It can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0035] The technical concept of this application is described below.

[0036] In database applications, it's often necessary to sample the data stored in the database. Sampling is used to approximate the overall data distribution using a subset of the data. Traditionally, hash values ​​are generated for each row, and sampled data is extracted from each row based on the hash values. For example, the hash values ​​are sorted, and the rows corresponding to the first few hash values ​​are used as the sampled data. However, this approach requires scanning the entire data to be sampled or calculating hash values ​​row by row, resulting in low sampling efficiency.

[0037] Based on the defects of the above-mentioned related technologies, the embodiments of the present application provide a method, device, electronic device and storage medium for data sampling, aiming to improve the sampling efficiency when performing data sampling.

[0038] The present application provides a data sampling method that can be applied to an electronic device provided with a database. Optionally, the database can be a distributed database (OceanBase, OB). The present application does not limit the type of electronic device, which can be any device type suitable for implementation, such as a smartphone, tablet computer, etc., and the present application will not elaborate on this.

[0039] 1 , which is a flow chart of a data sampling method in an embodiment of the present application, the method will be described below in conjunction with FIG1 . The specific implementation process of the method is as follows.

[0040] Step 100: Determine a target storage area to be sampled according to a received data sampling instruction.

[0041] In one embodiment, a log-structured merge tree (LSM-Tree) is used in the database to store data. LSM-Tree is a data structure for a key-value storage engine that is used to achieve high performance and high throughput. When LSM-Tree writes data, it first writes the data into a memory table (MemTable). When the amount of data in the MemTable reaches the set memory size, the data in the MemTable is transferred to an ordered string table (SSTable) on disk in the order of the primary key. When it is determined that the set table merge conditions are met (such as when the merge cycle is reached or the storage capacity reaches the set merge capacity), multiple SSTables that meet the conditions are merged.

[0042] In actual applications, the memory size and table merging conditions can be set according to the actual application scenario and are not restricted here.

[0043] The target storage area is the storage range of the database to be sampled, determined according to the data sampling instruction, and the purpose is to extract part of the data from the target storage area as the target sampling data. The data sampling instruction is an instruction for data query and sampling.

[0044] It should be noted that the target storage area includes at least one data record; each data record is stored row storage data or an incremental operation record for row storage data; each row storage data and its corresponding incremental operation record correspond to the same logical row. The key values ​​of each data record corresponding to the same logical row are the same. The incremental operation record is a record of update operations (such as data deletion and data modification) on the stored data. Each incremental operation record can also be called a version corresponding to the row storage data, and each row storage data can correspond to one or more incremental operation records.

[0045] Furthermore, upon receiving a data sampling instruction, if it is determined that the sampling fusion condition is met, the data to be sampled in the target storage area is fused to obtain a new target storage area. Specifically, the row storage data corresponding to the same logical row and the incremental operation records are fused to obtain the data of the logical row.

[0046] In one embodiment, the sampling fusion condition may be that the data to be sampled comes from multiple MemTables or SSTables. This is because only after the fusion process can the complete data corresponding to each logical row be obtained.

[0047] Step 101: Divide the target storage area into multiple logical sub-areas.

[0048] It should be noted that the target storage area is sorted according to the primary key.

[0049] In one embodiment, when executing step 101, the following steps may be executed in a loop until the interval division end condition is met: the target storage area is divided into intervals according to sub-interval sizes; and when it is determined that the size adjustment condition is met, the sub-interval size is adjusted.

[0050] Optionally, the end condition for interval division can be the end of data sampling, the end of data prefetching, or the end of data scanning. The initial sub-interval size can be a small fixed value. After the interval division is performed, each logical sub-interval is scanned sequentially. During the scanning process, the sub-interval size can be increased or decreased based on the current number of scan lines. The size adjustment condition is based on the current number of scan lines.

[0051] As an example, based on the initial sub-interval size, (one or more) logical sub-intervals are divided from the target storage area, and the divided logical sub-intervals are scanned in sequence, and the sub-interval size is periodically optimized and adjusted based on the number of logical rows currently scanned, and the unscanned area in the target storage area is divided again based on the adjusted sub-interval size until the scan is completed.

[0052] In actual applications, the end interval division condition, sub-interval size, and size adjustment condition can be set according to the actual application scenario and are not limited here.

[0053] In this way, the size of the subinterval can be adaptively adjusted during the process of dividing the logical subinterval, thereby obtaining logical subintervals of different sizes.

[0054] Step 102: Determine the target sampling quantity corresponding to each logical sub-interval according to the sampling probability and the interval range corresponding to each logical sub-interval.

[0055] In one implementation, the following steps may be performed for each logical sub-interval.

[0056] S1021: Determine the first row of the sub-interval and the last row of the sub-interval of the logical sub-interval.

[0057] S1022: Round down the product of the last row of the subinterval and the sampling probability to obtain a first sampling quantity.

[0058] S1023: Round down the product of the first row of the subinterval and the sampling probability to obtain a second sampling quantity.

[0059] S1024: Obtain a target sampling number of the logical subinterval according to the difference between the first sampling number and the second sampling number.

[0060] In one implementation, the following formula may be used to determine the target number of samples for any logical subinterval:

[0061] Among them, c is the target sampling number, p is the sampling probability, L is the first row of the subinterval, and r is the last row of the subinterval.

[0062] In this way, it can be guaranteed that when sampling reaches the end row of each logical subinterval, the sampling probability satisfies p.

[0063] Step 103: Filter out logical subintervals where the target sampling quantity is not zero.

[0064] In this way, when sampling, the logical subspace where no sampling data exists can be skipped, which reduces the sampling operations and improves the sampling efficiency.

[0065] Step 104: performing data sampling on each of the screened logical sub-intervals to obtain target sampling data corresponding to each of the logical sub-intervals.

[0066] In one embodiment, a sampling filter operator can be used to perform data sampling on each of the selected logical subintervals to obtain target sampled data corresponding to each logical subinterval. Furthermore, if other data filter operators are determined to exist, the sampling filter operator and other data filter operators can be combined to perform data sampling to obtain target sampled data corresponding to each logical subinterval.

[0067] The following describes how operators and sampling can be combined with Figure 2. Figure 2 shows an example diagram of an operator node, including an "and" node, a sampling filter operator, and a data filter operator. The sampling filter operator is used for data filtering and sampling, while the data filter operator is used for data filtering. The sampling filter operator and the data filter operator are two child nodes of the "and" node. Therefore, the sampling filter operator and the data filter operator in Figure 2 can be combined to perform data filtering and sampling.

[0068] In this way, sampling is designed as a filtering operator and embedded into the query process, which utilizes query optimization methods to improve the efficiency of data sampling.

[0069] In one embodiment, when executing step 104 , the following steps may be performed for each of the screened logical sub-intervals.

[0070] S1041: Determine a sampling offset corresponding to a logical subinterval. In one embodiment, when executing S1041, the following steps may be used.

[0071] S1041-1: Perform a hash operation based on the interval number iid of the logical sub-interval and the specified hash random seed value seed to obtain a hash value.

[0072] S1041 - 2 : Determine a sampling difference value according to a difference between a subinterval size corresponding to the logical subinterval and a target number of samples corresponding to the logical subinterval.

[0073] In one embodiment, the difference between the subinterval size w corresponding to the logical subinterval and the target sampling number c corresponding to the logical subinterval is determined, and the sum of the difference and a specified value (eg, 1) is used as the sampling difference.

[0074] S1041-3: Perform a modulo operation based on the hash value and the sampling difference to obtain a sampling offset.

[0075] In one implementation, when determining the sampling offset corresponding to any logical subinterval, the following formula may be used: offset=murmurhash(iid,seed)%(w-c+1).

[0076] Where offset is the sampling offset, murmurhash is the hash function, iid is the interval number of the logical subinterval, that is, the sequence number of the logical subinterval, seed is the specified hash random seed value, w is the subinterval size, and c is the target number of samples.

[0077] S1042: Determine the sampling start row corresponding to the logical sub-interval according to the sum of the first row of the sub-interval and the sampling offset.

[0078] S1043: Determine the last sampling row corresponding to the logical sub-interval according to the sum of the sampling start row and the target number of samples corresponding to the logical sub-interval.

[0079] S1044: Determine a sampling interval of a logical sub-interval according to the sampling start row and the sampling end row.

[0080] In one implementation, the sampling interval may be expressed as: [L+offset, L+offset+c).

[0081] Furthermore, the relationship between the logical row and each logical sub-interval can be explained as follows. The row number of the logical row can be expressed as row_num, and the interval number of the logical sub-interval where the logical row row_num is located is: iid = row_num / w; It should be noted that this formula represents rounding the result of dividing row_num by w to obtain iid, and iid is a natural number. The first row of the sub-interval of the logical sub-interval where it is located L = w*iid; the last row of the sub-interval of the logical sub-interval where it is located r = w*(iid+1)-1. The following function can also be used to determine whether the logical row is sampled data located in the sampling interval: is_smple = row_num∈[L+offset, L+offset+c).

[0082] Among them, is_smple indicates whether the logical row row_num is within the sampling interval.

[0083] S1045: Use the data in the sampling interval as the target sampling data corresponding to the logical sub-interval.

[0084] Furthermore, before data sampling, in order to improve data access speed and reduce system input and output (IO) overhead, the data microblocks that need to be sampled in the target storage area and their corresponding index microblocks can be pre-fetched into the memory in advance according to the index tree, so that the data in the data microblocks can be directly sampled in the memory through the index microblocks later.

[0085] The data in the target storage area is stored in at least one SSTable or MemTable. If the data in the target storage area is stored in at least one SSTable, the target storage area corresponds to an index tree of at least one ordered string table. Each SSTable corresponds to an index tree, which contains multiple nodes. From top to bottom, they are the root node of level 0, the nodes of level 1, ..., the leaf nodes of level n, where n is a positive integer. The leaf nodes in the index tree are data microblocks. The non-leaf nodes in the index tree are index microblocks. Each SSTable contains multiple data microblocks, each of which is used to store the data of each logical row in the target storage area. The index microblock contains the index information of the child nodes, which is used for querying and reading data.

[0086] The relationship between index microblocks and data microblocks is illustrated below with reference to Figure 3. Figure 3 is an example diagram of an index tree. In Figure 3, the index tree contains multiple layers of index microblocks. The layers in the index tree can be represented as level 0, level 1, level 2, ..., level n-1, and level n, respectively. The nodes in level n (i.e., leaf nodes) are data microblocks. The parent node in the index tree is used to record the storage information of the corresponding child nodes. The data of each layer is allocated to the nodes of that layer in the order of the primary key value from low to high.

[0087] It should be noted that when LSM-Tree scans data, it scans each node of the SSTable in the order of the primary key. If it corresponds to multiple SSTables, there may be multiple SSTables open at the same time. Then determine the row with the smallest primary key value of each SSTable. If the row with the smallest primary key value only comes from one SSTable, then there is no need to fuse it; if the row with the smallest primary key value comes from multiple tables (that is, SSTable or MemTable), then the row with the smallest primary key value will be fused.

[0088] In one embodiment, before sampling data for each selected logical subinterval and obtaining target sampled data corresponding to each logical subinterval, the following steps may be performed for each node within the primary key query range in each index tree.

[0089] S1045-1: Determine the total displacement of the current row scan of at least one index tree; the total displacement of the row scan is determined based on the number of node rows of each node currently scanned.

[0090] In one implementation, the total row scan displacement is obtained based on the current row scan displacement of each index tree.

[0091] For example, the current row scan displacement of the first index tree is 10, and the current row scan displacement of the second index tree is 10, then the total row scan displacement is 20.

[0092] This is because there may be multiple SSTables that are filtering and prefetching, so the total number of rows currently scanned in each SSTable is summarized.

[0093] Furthermore, since the data within the primary key query range may come from multiple SSTables or MemTables, the data records corresponding to the same primary key value are also fused, and the total row scan displacement is updated based on the fusion result.

[0094] S1045-2: If the node is a non-filtered node, determine the node row interval of the node according to the total row scan displacement and the number of node rows of the node.

[0095] In one embodiment, if the node is a non-filtered node, the node starting row of the node is determined based on the total row scan displacement, and the node ending row corresponding to the node is determined based on the node starting row and the number of node rows of the node to obtain the node row interval corresponding to the node.

[0096] S1045-3: If the node row interval does not have an intersection with all sampling intervals, the node is determined to be a filtered node; otherwise, the node is pre-fetched into a memory so as to perform data sampling through the node in the memory.

[0097] Among them, the primary key values ​​corresponding to the node only exist in the SSTable where the node is located, and do not exist in other SSTables or MemTables.

[0098] In one implementation, in the memory, a corresponding data microblock is determined based on the pre-fetched index microblock, and data in the data microblock is sampled.

[0099] Furthermore, regardless of whether the node is a filtering node or a non-filtering node, the row scan displacement of the index tree is updated according to the number of node rows of the node, and then the total row scan displacement is updated according to the updated row scan displacement of the index tree and the latest row scan displacements of other index trees.

[0100] In one embodiment, the sum of the number of node rows of the node and the row scan displacement of the index tree is used as the row scan displacement after the index tree is updated.

[0101] In this way, we can determine whether there is sampled data in the data microblock covered by the node based on the logical row range and sampling interval covered by the node. If there is no sampled data in the node, we can directly skip pre-fetching the data microblock and index microblock corresponding to the node, thereby saving IO overhead.

[0102] As an example, the target storage area corresponds to only one index tree, which contains a parent node and its corresponding three child nodes, namely child node 0, node 1, and child node 2. Child node 0 is the first child node of the parent node. The logical row range corresponding to the parent node is [100, 137]. The starting row of the parent node and node 0 is 100. The number of node rows of node 0 is 20. After scanning node 0, the row scan displacement of the parent node is updated to 20. The following is an example of the scanning and storage information update process of node 1 and child node 2 with reference to Figure 4. Figure 4 is an example diagram of a node storage information update. Figure 4 includes three sub-graphs, namely Figure (a), Figure (b), and Figure (c). In Figure (a), the node starting behavior corresponding to the parent node is 100, and the parent node's current row scan displacement is 20. Therefore, the node starting behavior of node 1 is determined to be 100 + 20 = 120. After scanning node 1, based on the number of node rows of node 1 = 10, the row scan displacement in the parent node is updated to 20 + 10 = 30, resulting in Figure (b). Similarly, based on the updated parent node in Figure (b), the node starting behavior of node 2 is determined and updated to 100 + 30 = 130, resulting in Figure (c).

[0103] Since the logical row range covered by each data micro-block is different when prefetching data micro-blocks, and the number of prefetched logical rows will decrease after skipping the data micro-blocks corresponding to some filter nodes, the number of logical rows that have been scanned cannot be determined by simple accumulation. For this reason, in an embodiment of the present application, the logical row coverage range of each child node is determined separately by the row scan displacement within a single index tree and the current total row scan displacement. If the number of sampled rows within the logical row coverage range is zero, it is not necessary to prefetch them into the memory, thereby reducing unnecessary IO operations.

[0104] It should be noted that when sampling is performed based on a fixed p through logical area division, the larger the sub-interval size of the logical sub-interval, the more uneven the sampled data will be. However, the filtering probability of data prefetching will be higher. Therefore, in practical applications, the size of the logical sub-interval can be adjusted according to actual needs.

[0105] In an embodiment of the present application, the size of the logical sub-interval can be dynamically and adaptively adjusted, and then the size of the sampling interval can be adjusted, so that the randomness of sampling and the system performance can be taken into account for a target storage area of ​​any size (such as a table); furthermore, the target storage area is divided into intervals to obtain multiple logical sub-intervals, so that the logical sub-intervals where no sampling data exists can be skipped, and it can be quickly determined whether there is data that needs to be sampled within each logical row range, and, based on the sampling interval in each logical sub-interval, when it is determined that there is no sampling data in the data micro-block covered by each node in the index tree, the pre-fetching of the nodes that do not need data sampling can be skipped, so that it can be quickly determined whether the data micro-block or the index micro-block is covered with sampling data, and the nodes that do not cover the sampling data are filtered. In combination with the index aggregation information, it is quickly determined whether the data block needs to be sampled based on the sampling interval of the logical sub-interval and the row number interval covered by the data block, further reducing unnecessary IO consumption and data sampling operations, and improving data sampling efficiency.

[0106] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0107] Based on the same inventive concept, the embodiments of the present application also provide a data sampling device. Since the principles of the above-mentioned device and equipment for solving the problem are similar to those of a data sampling method, the implementation of the above-mentioned device can refer to the implementation of the method, and the repeated parts will not be repeated. The device can be applied to electronic devices. This application does not limit the type of electronic device. It can be any type of device suitable for implementation, such as a smartphone, tablet computer, etc., and this application will not repeat them.

[0108] Refer to Figure 5, which is a block diagram of a data sampling device in an embodiment of the present application. In some embodiments, the data sampling device in the example of the present application includes: a receiving unit 501, which is used to determine a target storage area to be sampled based on a received data sampling instruction; a dividing unit 502, which is used to divide the target storage area into multiple logical sub-intervals; a determining unit 503, which is used to determine the target sampling number corresponding to each logical sub-interval based on the sampling probability and the interval range corresponding to each logical sub-interval; a screening unit 504, which is used to screen out logical sub-intervals whose target sampling number is not zero; and a sampling unit 505, which is used to sample data from each screened logical sub-interval to obtain target sampling data corresponding to each logical sub-interval.

[0109] In one embodiment, the target storage area includes at least one data record; each data record is stored row storage data or an incremental operation record for row storage data; each row storage data and its corresponding incremental operation record correspond to the same logical row; the target storage area is divided into intervals and data is sampled according to logical rows.

[0110] In one embodiment, the partitioning unit 502 is configured to cyclically execute the following steps until a condition for ending interval partitioning is met: partitioning the target storage area according to sub-interval sizes; and adjusting the sub-interval sizes when a size adjustment condition is determined to be met.

[0111] In one embodiment, the determination unit 503 is used to perform the following steps for each logical sub-interval respectively: determine the first row of the sub-interval and the last row of the sub-interval of the logical sub-interval; round down the product of the last row of the sub-interval and the sampling probability to obtain a first sampling number; round down the product of the first row of the sub-interval and the sampling probability to obtain a second sampling number; and obtain the target sampling number of the logical sub-interval based on the difference between the first sampling number and the second sampling number.

[0112] In one embodiment, the sampling unit 505 is used to perform the following steps for each logical sub-interval selected: determine the sampling offset corresponding to the logical sub-interval; determine the sampling starting row corresponding to the logical sub-interval based on the sum of the first row of the sub-interval and the sampling offset; determine the sampling last row corresponding to the logical sub-interval based on the sum of the sampling starting row and the target sampling number corresponding to the logical sub-interval; determine the sampling interval of the logical sub-interval based on the sampling starting row and the sampling last row; and use the data in the sampling interval as the target sampling data corresponding to the logical sub-interval.

[0113] In one embodiment, the sampling unit 505 is used to: perform a hash operation based on the interval number of the logical sub-interval and a specified hash random seed value to obtain a hash value; determine a sampling difference value based on the difference between the sub-interval size corresponding to the logical sub-interval and the target sampling number corresponding to the logical sub-interval; and perform a remainder operation based on the hash value and the sampling difference value to obtain a sampling offset.

[0114] In one embodiment, the target storage area corresponds to an index tree of at least one ordered string table; the index tree contains multiple nodes; the leaf nodes in the index tree are data microblocks; the non-leaf nodes in the index tree are index microblocks; each data microblock is used to store data of each logical row in the target storage area; the index microblock contains index information of the child node, which is used for querying and reading data.

[0115] In one embodiment, the sampling unit 505 is further used to: perform the following steps for each node within the primary key query range in each index tree: determine the current total row scan displacement of at least one index tree; the total row scan displacement is determined based on the number of node rows of each node currently scanned; if the node is a non-filtered node, determine the node row interval of the node based on the total row scan displacement and the number of node rows of the node; if the node row interval does not intersect with all sampling intervals and the primary key value corresponding to the node does not exist in other ordered string tables or memory tables, determine the node as a filter node, otherwise, pre-fetch the node into the memory so that data sampling is performed through the node in the memory.

[0116] In the data sampling method, device, electronic device, and storage medium provided in the embodiments of the present application, a target storage area to be sampled is determined based on a received data sampling instruction; the target storage area is divided to obtain multiple logical sub-intervals; the target sampling quantity corresponding to each logical sub-interval is determined based on the sampling probability and the interval range corresponding to each logical sub-interval; logical sub-intervals with a non-zero target sampling quantity are screened out; and data is sampled for each screened out logical sub-interval to obtain target sampling data corresponding to each logical sub-interval. In this way, the target storage area is divided into intervals, so that it is possible to quickly determine whether each logical sub-interval needs to be sampled, reducing sampling operations and improving efficiency.

[0117] An embodiment of the present application provides an electronic device including a processor and a memory, wherein the memory stores computer instructions for causing the processor to execute a method in any of the above embodiments.

[0118] An embodiment of the present application provides a storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method of any of the above embodiments.

[0119] FIG6 shows a schematic structural diagram of an electronic device 6000. Referring to FIG6, the electronic device 6000 includes a processor 6010 and a memory 6020, and optionally, may further include a power supply 6030, a display unit 6040, and an input unit 6050.

[0120] The processor 6010 is the control center of the electronic device 6000. It uses various interfaces and lines to connect various components and performs various functions of the electronic device 6000 by running or executing software programs and / or data stored in the memory 6020.

[0121] In the embodiment of the present application, the processor 6010 executes the various steps in the above embodiment when calling the computer program stored in the memory 6020.

[0122] Optionally, the processor 6010 may include one or more processing units. Preferably, the processor 6010 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interface, and applications, and the modem processor primarily processes wireless communications. It is understood that the modem processor may not be integrated into the processor 6010. In some embodiments, the processor and memory may be implemented on a single chip. In some embodiments, they may also be implemented on separate chips.

[0123] The memory 6020 may mainly include a program storage area and a data storage area. The program storage area may store an operating system, various applications, etc., and the data storage area may store data created based on the use of the electronic device 6000. In addition, the memory 6020 may include a high-speed random access memory and a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0124] The electronic device 6000 also includes a power supply 6030 (such as a battery) for supplying power to various components. The power supply can be logically connected to the processor 6010 through a power management system, thereby managing functions such as charging, discharging, and power consumption through the power management system.

[0125] The display unit 6040 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device 6000. In the embodiment of the present application, it is mainly used to display the display interface of each application in the electronic device 6000 and objects such as text and pictures displayed on the display interface. The display unit 6040 may include a display panel 6041. The display panel 6041 can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.

[0126] The input unit 6050 can be used to receive information such as numbers or characters input by the user. The input unit 6050 may include a touch panel 6051 and other input devices 6052. The touch panel 6051, also known as a touch screen, can receive user touch operations on or near it (for example, operations performed by the user using a finger, a stylus, or any other suitable object or accessory on or near the touch panel 6051).

[0127] Specifically, the touch panel 6051 can detect user touch operations and the signals generated by the touch operations, convert these signals into touch point coordinates, send them to the processor 6010, and receive and execute commands sent by the processor 6010. In addition, the touch panel 6051 can be implemented using various types, such as resistive, capacitive, infrared, and surface acoustic wave. Other input devices 6052 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, a joystick, etc.

[0128] Of course, the touch panel 6051 can cover the display panel 6041. When the touch panel 6051 detects a touch operation on or near it, it transmits the information to the processor 6010 to determine the type of touch event. The processor 6010 then provides a corresponding visual output on the display panel 6041 based on the type of touch event. Although in Figure 6, the touch panel 6051 and the display panel 6041 are used as two independent components to implement the input and output functions of the electronic device 6000, in some embodiments, the touch panel 6051 and the display panel 6041 can be integrated to implement the input and output functions of the electronic device 6000.

[0129] The electronic device 6000 may further include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity light sensor, etc. Of course, according to the needs of specific applications, the electronic device 6000 may also include other components such as a camera. Since these components are not the key components used in the embodiments of the present application, they are not shown in FIG6 and will not be described in detail.

[0130] Those skilled in the art will understand that FIG6 is merely an example of an electronic device and does not limit the electronic device. The electronic device may include more or fewer components than shown in the figure, or may combine certain components, or may include different components.

[0131] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0132] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the embodiments. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Obvious variations or modifications arising therefrom remain within the scope of protection of this application.

Claims

1. A data sampling method, comprising: Determine the target storage area to be sampled according to the received data sampling instruction; Dividing the target storage area into multiple logical sub-intervals; Determine the target sampling quantity corresponding to each logical subinterval according to the sampling probability and the interval range corresponding to each logical subinterval; Filter out logical subintervals where the target sampling number is not zero; Data sampling is performed on each of the screened logical sub-intervals to obtain target sampling data corresponding to each of the logical sub-intervals.

2. The method according to claim 1, wherein the target storage area includes at least one data record; Each data record is a stored row of stored data or an incremental operation record for the row of stored data; Each row stores data and its corresponding incremental operation record, corresponding to the same logical row; The target storage area is divided into intervals and data is sampled according to logical rows.

3. The method according to claim 2, wherein dividing the target storage area to obtain a plurality of logical sub-intervals comprises: The following steps are executed repeatedly until the end of the interval division condition is reached: Dividing the target storage area into intervals according to subinterval sizes; When it is determined that the size adjustment condition is met, the size of the sub-interval is adjusted.

4. The method according to any one of claims 1 to 3, wherein determining the target number of samples for each logical subinterval comprises: For each logical subinterval, perform the following steps: Determine the first row of the subinterval and the last row of the subinterval of the logical subinterval; Rounding down the product of the last row of the subinterval and the sampling probability to obtain a first sampling quantity; Rounding down the product of the first row of the subinterval and the sampling probability to obtain a second sampling quantity; A target sampling number of the logic subinterval is obtained according to the difference between the first sampling number and the second sampling number.

5. The method according to any one of claims 1 to 3, wherein data sampling is performed on each of the screened logical subintervals to obtain target sampled data corresponding to each of the logical subintervals, comprising: For each logical subinterval that has been filtered out, perform the following steps: Determining a sampling offset corresponding to the logical subinterval; Determine a sampling start row corresponding to the logical subinterval according to the sum of the first row of the subinterval and the sampling offset; Determining the last sampling row corresponding to the logical subinterval according to the sum of the sampling start row and the target number of samples corresponding to the logical subinterval; Determining a sampling interval of the logical subinterval according to the sampling start row and the sampling end row; The data within the sampling interval is used as the target sampling data corresponding to the logical sub-interval.

6. The method according to claim 5, wherein determining the sampling offset corresponding to the logical sub-interval comprises: Performing a hash operation according to the interval number of the logical subinterval and a specified hash random seed value to obtain a hash value; determining a sampling difference value according to a difference between a subinterval size corresponding to the logical subinterval and a target sampling quantity corresponding to the logical subinterval; A modulo operation is performed according to the hash value and the sampling difference to obtain the sampling offset.

7. According to the method of claim 5, the target storage area corresponds to an index tree of at least one ordered string table; the index tree contains multiple nodes; the leaf nodes in the index tree are data microblocks; the non-leaf nodes in the index tree are index microblocks; each data microblock is used to store data of each logical row in the target storage area; the index microblock contains index information of child nodes, which is used for querying and reading data.

8. The method according to claim 7, before sampling data for each of the screened logical subintervals to obtain target sampled data corresponding to each of the logical subintervals, the method further comprises: For each node in the primary key query range in each index tree, perform the following steps: Determining a total row scan displacement of the at least one index tree; the total row scan displacement is determined based on the number of node rows of each node currently scanned; If the node is a non-filtered node, determining the node row interval of the node according to the row scan total displacement and the number of node rows of the node; If the node row interval does not intersect with all sampling intervals and the primary key value corresponding to the node does not exist in other ordered string tables or memory tables, the node is determined to be a filtering node; otherwise, the node is prefetched into the memory so that data sampling is performed through the node in the memory.

9. A data sampling device, comprising: A receiving unit, configured to determine a target storage area to be sampled according to a received data sampling instruction; a partitioning unit, configured to partition the target storage area to obtain a plurality of logical sub-intervals; a determination unit, configured to determine a target sampling quantity corresponding to each logical subinterval according to the sampling probability and the interval range corresponding to each logical subinterval; A screening unit, used for screening out logical subintervals where the target sampling quantity is not zero; The sampling unit is used to perform data sampling on each of the screened logical sub-intervals to obtain target sampling data corresponding to each of the logical sub-intervals.

10. An electronic device comprising: processor; as well as A memory storing computer instructions, wherein the computer instructions are used to enable the processor to execute the method according to any one of claims 1 to 8.

11. A storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 8.

Images