A method for creating a database index, a related device, an equipment and a storage medium
By sorting unordered data partitions in parallel, the ordered data partitions are directly constructed, which solves the problem of low index creation efficiency in the existing technology and achieves efficient index creation and query performance improvement.
Patent Information
- Application Number
- CN202210752984.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-29
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-06-29
AI Technical Summary
The existing database index creation method is inefficient due to single-thread serial execution, and cannot fully utilize multi-threaded parallel processing, resulting in a long index creation time.
Sorting the data in the unordered data partition in parallel, using multi-threaded parallel processing to avoid external sorting, and directly building an ordered data partition to create an index.
Improve the parallelism and efficiency of index creation, reduce index creation time, and improve the performance of database queries.
Smart Images

Figure CN115114293B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of computer technology and database technology, and in particular to a method for creating a database index, related devices, equipment and storage medium. Background Art
[0002] An index is a structure that sorts the values of one or more columns in a database table. One of the main purposes of an index is to speed up the retrieval of data in a table, that is, to help information seekers find record identifiers that meet the restrictions as quickly as possible. Since the amount of data in a database table is often large, it is necessary to establish an efficient index.
[0003] At present, the method of creating an index in a relational database management system (MySQL) mainly includes three stages. In the first stage, a single thread is used to scan the entire primary key index file to generate a secondary index data file. In the second stage, a single thread is used to perform a two-way merge sort until the entire file is sorted. In the third stage, a single thread is used to scan the index data file that has been sorted in the second stage and insert it into the index tree in sequence to obtain an index tree.
[0004] However, the inventors have found that the existing solutions have at least the following problems: in the process of creating a database index, the three stages are all executed serially by a single thread, resulting in low efficiency in creating the index. Summary of the invention
[0005] The embodiment of the present application provides a method for creating a database index, a related device, an apparatus and a storage medium. The present application simultaneously sorts the data in several unordered data partitions in a parallel manner, and since the external data partitions are arranged in order, there is no need for external sorting, thereby improving the efficiency of index creation.
[0006] In view of this, the present application provides, on one hand, a method for creating a database index, comprising:
[0007] Acquire a sampling point data sequence from a target data set, wherein the target data set includes a data set corresponding to a target field, and the sampling point data sequence includes at least one sampling point data in ascending order or descending order;
[0008] Determine at least one quantile data from the sampling point data sequence;
[0009] According to at least one quantile data, the target data set is divided into at least two unordered data partitions;
[0010] Invoke at least two threads to sort the data in at least two unordered data partitions to obtain at least two ordered data partitions, wherein the ordered data partitions have a one-to-one correspondence with the unordered data partitions;
[0011] A database index corresponding to the target field is constructed according to at least two ordered data partitions.
[0012] Another aspect of the present application provides a method for creating a database index, comprising:
[0013] Call N threads to scan the original data set to obtain N data subsets, where each data subset includes a data set corresponding to a target field, and each data subset is obtained by scanning one thread, and N is an integer greater than 1;
[0014] Calling N threads to sort the data in the N data subsets to obtain N first data sequences, wherein each thread in the N threads is used to sort the data in one data subset;
[0015] Calling N / 2 threads to perform two-way merge sorting on the N first data sequences to obtain N / 2 second data sequences until a target data sequence is obtained, wherein each thread of the N / 2 threads is used to sort two first data sequences;
[0016] Build a database index corresponding to the target field based on the target data sequence.
[0017] Another aspect of the present application provides a method for creating a database index, comprising:
[0018] Calling N threads to scan the original data set to obtain N first data sets, wherein each first data set includes a data set corresponding to the target field, and each first data set is obtained by scanning one thread, and N is an integer greater than 1;
[0019] Calling N threads to sort the data in the N first data sets to obtain N first data sequences, wherein each thread in the N threads is used to sort the data in one first data set;
[0020] According to the N first data sequences, determine (N-1) pivots corresponding to each first data sequence, wherein each pivot is used to divide the data in the first data sequence;
[0021] According to (N-1) pivots, divide the N first data sequences into N second data sets;
[0022] Calling N threads to sort the data in the N second data sets to obtain N second data sequences, wherein each thread in the N threads is used to sort the data in one second data set;
[0023] A database index corresponding to the target field is constructed according to the N second data sequences.
[0024] Another aspect of the present application provides an index creation device, comprising:
[0025] An acquisition module, used to acquire a sampling point data sequence from a target data set, wherein the target data set includes a data set corresponding to a target field, and the sampling point data sequence includes at least one sampling point data in ascending order or descending order;
[0026] A determination module, used for determining at least one quantile data from a sampling point data sequence;
[0027] A partitioning module, used for partitioning the target data set into at least two unordered data partitions according to at least one quantile data;
[0028] A sorting module, used for calling at least two threads to sort the data in at least two unordered data partitions to obtain at least two ordered data partitions, wherein the ordered data partitions have a one-to-one correspondence with the unordered data partitions;
[0029] The building module is used to build a database index corresponding to the target field according to at least two ordered data partitions.
[0030] In one possible design, in another implementation of another aspect of the embodiment of the present application,
[0031] The acquisition module is specifically used to sample and process the data corresponding to the target field according to a preset sampling interval when calling S threads to scan the original data set, wherein the original data set includes the data set corresponding to the target field, and S is an integer greater than 1;
[0032] When the scanning of the original data set is completed, a target data set and at least one sampling point data to be sorted are obtained, wherein the target data set includes at least one data subset, and each data subset is obtained by scanning a thread;
[0033] Arrange at least one sampling point data to be sorted in ascending order or descending order to obtain a sampling point data sequence.
[0034] In one possible design, in another implementation of another aspect of the embodiment of the present application,
[0035] The acquisition module is specifically used to sample and process the data corresponding to the target field according to a preset sampling interval when calling the target thread to scan the original data set, wherein the original data set includes the data set corresponding to the target field;
[0036] When the scanning of the original data set is completed, the target data set and at least one sampling point data to be sorted are obtained;
[0037] Arrange at least one sampling point data to be sorted in ascending order or descending order to obtain a sampling point data sequence.
[0038] In one possible design, in another implementation of another aspect of the embodiment of the present application,
[0039] The acquisition module is specifically used to call S threads to scan the original data set to obtain the target data set, wherein the original data set includes the data set corresponding to the target field, the target data set includes at least one data subset, each data subset is obtained by scanning one thread, and S is an integer greater than 1;
[0040] Calling at least two threads to scan the target data set according to a preset sampling interval to obtain at least one sampling point data to be sorted;
[0041] Arrange at least one sampling point data to be sorted in ascending order or descending order to obtain a sampling point data sequence.
[0042] In one possible design, in another implementation of another aspect of the embodiment of the present application,
[0043] The acquisition module is specifically used to call the target thread to scan the original data set to obtain the target data set, wherein the original data set includes the data set corresponding to the target field;
[0044] Calling the target thread, scanning the target data set according to a preset sampling interval, and obtaining at least one sampling point data to be sorted;
[0045] Arrange at least one sampling point data to be sorted in ascending order or descending order to obtain a sampling point data sequence.
[0046] In one possible design, in another implementation of another aspect of the embodiment of the present application,
[0047] A determination module, specifically used to determine the total number of sampling point data according to the sampling point data sequence;
[0048] According to the total number of data partitions to be divided, determine the total number of quantile data;
[0049] The total number of sampling point data and the total number of quantile data are divided to obtain the target quotient value;
[0050] According to integer multiples of the target quotient value, sampling point data are extracted from the sampling point data sequence as quantile data to obtain at least one quantile data.
[0051] In one possible design, in another implementation of another aspect of the embodiment of the present application,
[0052] A partitioning module, specifically used to determine at least two data interval ranges according to at least one quantile data;
[0053] The target thread is called to divide the data corresponding to the target field in the target data set into corresponding data intervals to obtain at least two unordered data partitions.
[0054] In one possible design, in another implementation of another aspect of the embodiment of the present application,
[0055] A partitioning module, specifically used to determine at least two data interval ranges according to at least one quantile data;
[0056] Call N threads to divide the data corresponding to the target field in the N data subsets into corresponding data interval ranges to obtain at least two unordered data partitions, wherein the N data subsets belong to the target data set, and each thread is used to divide the data in a data subset into corresponding data area ranges, and N is an integer greater than 1.
[0057] In one possible design, in another implementation of another aspect of the embodiment of the present application,
[0058] The sorting module is specifically used to call N threads to sort the data in N unordered data partitions to obtain N ordered data partitions, wherein each thread is used to sort the data in one unordered data partition, and N is an integer greater than 1.
[0059] In one possible design, in another implementation of another aspect of the embodiment of the present application,
[0060] A construction module is specifically used to call N threads to perform index construction on N ordered data partitions to obtain N index trees, wherein each thread is used to construct a corresponding index tree according to the ordered data partition, and N is an integer greater than 1;
[0061] A target index tree corresponding to the target field is constructed according to the N index trees, wherein the target index tree belongs to a database index.
[0062] In one possible design, in another implementation of another aspect of the embodiment of the present application,
[0063] A construction module, specifically used to determine the highest tree level number among N index trees;
[0064] According to the highest tree number, add root nodes to the index trees whose tree number is less than the highest tree number among the N index trees, until N index trees with the same tree number are obtained;
[0065] Perform node merging processing on N index trees with the same number of tree layers to obtain the target index tree corresponding to the target field.
[0066] In one possible design, in another implementation of another aspect of the embodiment of the present application,
[0067] The construction module is specifically used to call the target thread, perform index construction on N ordered data partitions, and obtain a target index tree, wherein the target index tree belongs to the database index.
[0068] Another aspect of the present application provides an index creation device, comprising:
[0069] An acquisition module is used to call N threads to scan the original data set to obtain N data subsets, wherein each data subset includes a data set corresponding to a target field, and each data subset is obtained by scanning one thread, and N is an integer greater than 1;
[0070] A sorting module, used for calling N threads to sort the data in the N data subsets to obtain N first data sequences, wherein each thread in the N threads is used for sorting the data in one data subset;
[0071] The sorting module is further used to call N / 2 threads to perform two-way merge sorting on the N first data sequences to obtain N / 2 second data sequences until a target data sequence is obtained, wherein each thread of the N / 2 threads is used to sort two first data sequences;
[0072] The construction module is used to construct a database index corresponding to the target field according to the target data sequence.
[0073] Another aspect of the present application provides an index creation device, comprising:
[0074] An acquisition module, used for calling N threads to scan the original data set to obtain N first data sets, wherein each first data set includes a data set corresponding to the target field, and each first data set is obtained by scanning one thread, and N is an integer greater than 1;
[0075] A sorting module, used for calling N threads to sort the data in the N first data sets to obtain N first data sequences, wherein each thread in the N threads is used for sorting the data in one first data set;
[0076] A determination module, configured to determine, based on the N first data sequences, (N-1) pivots corresponding to each first data sequence, wherein each pivot is used to divide the data in the first data sequence;
[0077] A partitioning module, for partitioning the N first data sequences into N second data sets according to (N-1) pivot axes;
[0078] The sorting module is further used to call N threads to sort the data in the N second data sets to obtain N second data sequences, wherein each thread in the N threads is used to sort the data in one second data set;
[0079] A construction module is used to construct a database index corresponding to the target field according to the N second data sequences.
[0080] On the other hand, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned methods when executing the computer program.
[0081] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the above-mentioned methods are implemented.
[0082] Another aspect of the present application provides a computer program product, including a computer program, which implements the above-mentioned methods when executed by a processor.
[0083] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:
[0084] In an embodiment of the present application, a method for creating a database index is provided. In order to improve the parallelism of external sorting, the data is divided according to the ordered quantile data, so that several externally ordered and internally unordered data partitions can be obtained, and then multiple threads are called to sort the data in the unordered data partitions in parallel, thereby generating multiple ordered data partitions, and thus an ordered data sequence can be obtained to construct a database index. Through the above method, it can be seen that on the one hand, the data in several unordered data partitions are sorted at the same time in a parallel manner, so as to improve the parallelism of index creation and reduce the time required to create the index. On the other hand, since each data partition is ordered externally, there is no need for external sorting, thereby further improving the efficiency of index creation. BRIEF DESCRIPTION OF THE DRAWINGS
[0085] Figure 1 A schematic diagram of the architecture of a database index creation system in an embodiment of the present application;
[0086] Figure 2 A schematic diagram of the structure of a B+ tree in an embodiment of the present application;
[0087] Figure 3 A schematic diagram of a process of creating a database index in an embodiment of the present application;
[0088] Figure 4 A schematic diagram of multi-threaded parallel scanning sampling in an embodiment of the present application;
[0089] Figure 5 A schematic diagram of single-thread serial scanning sampling in an embodiment of the present application;
[0090] Figure 6 A schematic diagram of partitioning according to point data in an embodiment of the present application;
[0091] Figure 7 A schematic diagram of constructing a target index tree in parallel in an embodiment of the present application;
[0092] Figure 8 A schematic diagram of completing the height of the index tree in an embodiment of the present application;
[0093] Fig. 9 A schematic diagram of the height of the compressed index tree in an embodiment of the present application;
[0094] Fig.10 Another flowchart of the method for creating a database index in an embodiment of the present application;
[0095] Fig.11 A schematic diagram of generating a target data sequence based on a two-way merge method in an embodiment of the present application;
[0096] Fig.12 Another flowchart of the method for creating a database index in an embodiment of the present application;
[0097] Fig.13 A schematic diagram of determining a pivot based on a data set in an embodiment of the present application;
[0098] Fig.14 A schematic diagram of an index creation device in an embodiment of the present application;
[0099] Fig.15 Another schematic diagram of the index creation device in the embodiment of the present application;
[0100] Fig.16Another schematic diagram of the index creation device in the embodiment of the present application;
[0101] Fig.17 A structural diagram of a server in an embodiment of the present application. DETAILED DESCRIPTION
[0102] The embodiment of the present application provides a method for creating a database index, a related device, an apparatus and a storage medium. The present application simultaneously sorts the data in several unordered data partitions in a parallel manner, and since the external data partitions are arranged in order, there is no need for external sorting, thereby improving the efficiency of index creation.
[0103] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0104] The development of database technology has provided powerful data organization, data management and data storage capabilities for various computer applications. With the advent of the era of big data and cloud computing, the change in data volume is showing an exponential growth trend. When a database business system is in operation, it is often necessary to create an index for a data table for some reason. In a relational database, an index is a storage structure that sorts the values of one or more columns in a database table. The role of an index is equivalent to the catalog of a book, and the required content can be quickly found according to the page numbers in the catalog. Since this application involves database technology, the database will be introduced below.
[0105] A database can be thought of as an electronic filing cabinet, that is, a place where electronic files are stored, where users can add, query, update, delete, and so on. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application. A database management system (DBMS) is a computer software system designed for managing databases, generally with basic functions such as storage, interception, security, and backup. Database management systems can be classified according to the database model it supports, such as relational, Extensible Markup Language (XML); or according to the type of computer it supports, such as server clusters, mobile phones; or according to the query language used, such as Structured Query Language (SQL), XQuery; or according to the performance impulse focus, such as maximum scale, maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMS can cross categories, for example, supporting multiple query languages at the same time.
[0106] In machine learning involved in artificial intelligence (AI), data can be extracted according to the index for training. Among them, machine learning (ML) is a multi-disciplinary cross-discipline involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and inductive learning.
[0107] In order to improve the efficiency of creating database indexes, this application proposes a database index creation method that can be applied to cloud databases (CDB) to accelerate database operations that require index creation. For example, adding an index, optimizing a data table, altering a data table, etc., especially when the amount of data in the table is relatively large, the method provided by this application can achieve a more obvious improvement effect.
[0108] The method provided in this application is applied to Figure 1The database index creation system shown, as shown in the figure, includes a server and a terminal device, and the client is deployed on the terminal device, wherein the client can be run on the terminal device in the form of a browser, or can be run on the terminal device in the form of an independent application (application, APP), etc., and the specific presentation form of the client is not limited here. The server involved in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal device can be a smart phone, a tablet computer, a laptop, a PDA, a personal computer, a smart TV, a smart watch, a vehicle-mounted device, a wearable device, etc., but is not limited thereto. The terminal device and the server can be directly or indirectly connected by wired or wireless communication, and this application is not limited here. The number of servers and terminal devices is also not limited. The solution provided in this application can be completed independently by a terminal device, can be completed independently by a server, or can be completed by a terminal device and a server in cooperation. This application does not make any specific limitations on this.
[0109] Exemplarily, the server is built on the basis of a database system and has the characteristics of a database system. Its main functions include database management functions (e.g., system configuration and management, data access and update management, data integrity management, and data security management), database query and manipulation functions (e.g., database retrieval and modification), database maintenance functions, and database parallel operation. Users can import data or trigger data instructions through terminal devices.
[0110] Since this application involves some terms related to professional fields, they will be explained below for ease of understanding.
[0111] (1) Relational Database Management System (MySQL): It is one of the open source relational database management systems. "InnoDB" is one of the database engines of MySQL and is now the default storage engine of MySQL. Relational databases store data in different tables, which increases speed and flexibility. The Structured Query Language (SQL) used by MySQL is the most commonly used standardized language for accessing databases.
[0112] (2) B+tree: It is a self-balancing search tree that can keep data stable and orderly. It is usually used in the file system of databases and operating systems. It is also the data structure used by InnoDB table data files. A table is usually composed of multiple B+trees, and one node corresponds to a page on the disk. For ease of understanding, please refer to Figure 2 , Figure 2 This is a structural diagram of the B+ tree in an embodiment of the present application. As shown in the figure, no data is stored on the non-leaf nodes of the B+ tree, only key values are stored. The leaf nodes store not only key values but also data.
[0113] In a T-order B+ tree, except for the root node, each internal node has T / 2 to T child nodes, and all leaf nodes are at the same level of the tree structure, so the tree structure is always height-balanced.
[0114] (3) Index: A structure that sorts the values of one or more columns in a database table. Using an index allows the database system to directly locate the records that meet the conditions without having to scan the entire table. In the InnoDB storage engine, an index corresponds to a B+ tree. Indexes are divided into clustered indexes (i.e., primary key indexes) and non-clustered indexes (i.e., secondary indexes).
[0115] (4) Primary key index (clustered index): The non-leaf nodes of the B+ tree only store the primary key index value, and the leaf nodes store the contents of all columns of a row of data. If not explicitly specified, MySQL will generate a hidden column (row_id) as the primary key column.
[0116] (5) Secondary index: Also known as auxiliary index, it can be divided into unique index and non-unique index according to whether the index key value is unique. The secondary index corresponds to the non-leaf nodes and leaf nodes of the B+ tree, both of which store index data and primary key index data, and does not contain all the data of the row record.
[0117] In combination with the above introduction, the following will introduce a database index creation method in this application, please refer to Figure 3 The method for creating a database index in the embodiment of the present application may be executed by a computer device, which may be a server or a terminal device. The embodiment of the present application includes:
[0118] 110. Acquire a sampling point data sequence from a target data set, wherein the target data set includes a data set corresponding to a target field, and the sampling point data sequence includes at least one sampling point data in ascending order or descending order;
[0119] In one or more embodiments, an original data set is first obtained, and the original data set includes a data set corresponding to at least one field. Based on this, the original data set is scanned according to the requirements. For example, all data sets containing the target field can be scanned to obtain a target data set. For example, a number of sampling point data can be scanned from the original data set at a certain sampling interval, and the sampling point data is the data corresponding to the target field.
[0120] It can be seen that the number of sampling point data should be less than or equal to the number of data in the target data set, and these sampling point data belong to the target data set. Based on this, these sampling point data can be further arranged in ascending or descending order to obtain a sampling point data sequence.
[0121] It should be noted that the “target field” in the present application can be a single field or a field set consisting of multiple fields, which is not limited here.
[0122] 120. Determine at least one quantile data from the sampling point data sequence;
[0123] In one or more embodiments, at least one quantile data is determined from the sampling point data sequence, wherein the quantile data may be sampling point data in the sampling point data sequence, or the quantile data may be determined based on the sampling point data sequence. Taking the sampling point data sequence of "10, 96, 290, 312, 571, 822, 1203" as an example, assuming that one quantile data needs to be extracted, based on this, the following will be introduced in conjunction with an example.
[0124] Exemplarily, the middle sampling point data may be directly extracted from the sampling point data sequence as the quantile data, for example, the sampling point data "312" may be used as the quantile data.
[0125] Exemplarily, the middle sampling point data is first extracted from the sampling point data sequence as the middle data, for example, the middle data "312" is taken as a middle number, and then an interval is constructed according to the range of ±10%, that is, the interval is 280.8 to 343.2. Thus, a value is randomly selected from the interval as the quantile data.
[0126] 130. Divide the target data set into at least two unordered data partitions according to at least one quantile data;
[0127] In one or more embodiments, the target data set is partitioned according to at least one quantile data, thereby obtaining at least two unordered data partitions. Among them, K quantile data can divide the target data set into (K+1) unordered data partitions. Generally, the number of data partitions to be divided can be determined according to the number of available threads. In order to achieve 100% thread utilization, the number of data partitions can be determined as the number of available threads, based on which the number of quantile data can be obtained by subtracting 1 from the number of data partitions.
[0128] 140. Call at least two threads to sort data in at least two unordered data partitions to obtain at least two ordered data partitions, wherein the ordered data partitions have a one-to-one correspondence with the unordered data partitions;
[0129] In one or more embodiments, since the number of available threads is considered when dividing the data partitions, the number of unordered data partitions obtained is equal to the number of available threads. Based on this, a thread is called to sort an unordered data partition respectively. Here, there are N threads, and the N unordered data partitions are sorted in parallel. After the sorting is completed, N ordered data partitions can be obtained.
[0130] 150. Build a database index corresponding to the target field based on at least two ordered data partitions.
[0131] In one or more embodiments, after obtaining at least two ordered data partitions, the data sorting for the target field can be completed, and then a database index corresponding to the target field can be constructed according to the sorted data, wherein the database index is an ordered index structure.
[0132] It should be noted that the database index can be specifically expressed in the form of an index tree. The index tree types involved in this application include but are not limited to B+ trees, B trees, etc., which are not limited here. It can be understood that steps 110 to 150 construct a database index corresponding to a field. In actual applications, a similar method can be used to construct database indexes corresponding to other fields. At the same time, multiple threads can be used to construct database indexes corresponding to different fields in parallel.
[0133] In an embodiment of the present application, a method for creating a database index is provided. Through the above method, on the one hand, the data in several unordered data partitions are sorted in parallel at the same time, thereby improving the parallelism of index creation and reducing the time required for index creation. On the other hand, since each data partition is ordered externally, there is no need for external sorting, thereby further improving the efficiency of index creation.
[0134] Optionally, in the above Figure 3On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, obtaining a sampling point data sequence from a target data set may specifically include:
[0135] When S threads are called to scan the original data set, the data corresponding to the target field is sampled and processed according to a preset sampling interval, wherein the original data set includes the data set corresponding to the target field, and S is an integer greater than 1;
[0136] When the scanning of the original data set is completed, a target data set and at least one sampling point data to be sorted are obtained, wherein the target data set includes at least one data subset, and each data subset is obtained by scanning a thread;
[0137] Arrange at least one sampling point data to be sorted in ascending order or descending order to obtain a sampling point data sequence.
[0138] In one or more embodiments, a method of calling multi-threaded parallel scanning and sampling data is introduced. It can be seen from the aforementioned embodiments that scanning the primary key index can generate an unordered secondary index data file. The parallel scanning stage mainly has two tasks. One task is to scan the primary key record for each index to be created and generate a data file of "<secondary index, primary key index>", where "<secondary index, primary key index>" represents a combination of a secondary index column and a primary key index column. The other task is to perform data sampling. In a database, scanning refers to the process used by the database server to search for each record in a table until all records that meet the given conditions are returned.
[0139] Specifically, for ease of understanding, see Figure 4 , Figure 4 This is a schematic diagram of multi-threaded parallel scanning sampling in an embodiment of the present application. As shown in the figure, taking parallel scanning of three threads (i.e., S equals 3) as an example, based on this, three threads can be called to scan three different indexes in the original data set in parallel, thereby obtaining a data set corresponding to each index. At the same time, the data under each index can also be sampled according to a preset sampling interval to obtain at least one sampling point data to be sorted corresponding to each index. Among them, an index can specify any field as an index column. For example, field A is used as an index column. In this case, field A is the target field. For another example, both field A and field B are used as index columns. In this case, field A and field B together constitute the target field.
[0140] Based on this, after the scanning of the original data set is completed, the data set for S indexes can be obtained. For ease of understanding, please refer to Table 1, which is a schematic diagram of the original data set.
[0141] Table 1
[0142] Logo Name Purchase quantity 0 Xiao Zhao 5000 1 Small Money 5000 2 Xiao Sun 10000 3 Xiao Li 8000 4 Xiao Zhou 6000 5 Xiao Wu 5000 6 Xiao Zheng 12000
[0143] Based on the original data set shown in Table 1, assuming that only the index of the "name" column needs to be constructed, at this time, the target field is "name", it can be seen that the original data set needs to include the data set corresponding to the target field. Therefore, at least one thread can be used to scan, so as to extract all "<name, ID>", thereby generating data containing only "<name, ID>", and then writing it to a file, which is the target data set corresponding to the target field. Optionally, assuming that 4 threads are used to scan the index of the "name" column at the same time, 4 data subsets are obtained, and these 4 data subsets constitute a target data set.
[0144] For ease of understanding, please refer to Table 2, which is a schematic diagram of the target data set.
[0145] Table 2
[0146] Name ID Xiao Li 3 Small Money 1 Xiao Sun 2 Xiao Wu 5 Xiao Zhao 0 Xiao Zheng 6 Xiao Zhou 4
[0147] Similarly, if you need to build an index for the "purchase amount" column, call another thread to perform parallel scanning to extract all "<purchase amount, ID>", thereby generating data containing only "<purchase amount, ID>", and then write it to another file for subsequent processing.
[0148] It should be noted that, for ease of introduction, this application uses the target data set as an example for explanation. It is understandable that data sets constructed based on other indexes are also processed in a similar manner, which will not be elaborated here.
[0149] While scanning the data, the thread can also sample the data under the target field according to a preset sampling interval. One feasible way is to extract data as sampling point data according to a fixed sampling interval, thereby obtaining at least one sampling point data to be sorted. Another way is to extract data as sampling point data according to a random sampling interval, that is, before each sampling, a random value is selected from a reasonable range, and the random value is used as the sampling interval to extract data as sampling point data, thereby obtaining at least one sampling point data to be sorted. Based on this, these sampling point data can be sorted from large to small (i.e., descending order) or from small to large (i.e., ascending order). This application will take the example of sorting the sampling point data from small to large (i.e., ascending order) as an example, but it should not be understood as a limitation of this application. At this point, a sampling point data sequence is obtained.
[0150] It is understandable that, usually, the amount of sampling point data is determined by the size of available memory. For example, if only W-sized memory is available and the size of each record is R, then a total of W / R records are sampled.
[0151] Secondly, in an embodiment of the present application, a method for calling multi-threaded parallel scanning and sampling data is provided. Through the above method, a multi-threaded parallel scanning method is used when scanning the original data set, wherein the core idea is mainly to split the range of the index tree corresponding to the primary key index, and different threads are responsible for scanning the data in these ranges in parallel, thereby achieving the effect of accelerated scanning. At the same time, each index can also be scanned in parallel to obtain multiple data subsets, thereby further improving the efficiency of data scanning.
[0152] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, obtaining a sampling point data sequence from a target data set may specifically include:
[0153] When the target thread is called to scan the original data set, the data corresponding to the target field is sampled and processed according to the preset sampling interval, wherein the original data set includes the data set corresponding to the target field;
[0154] When the scanning of the original data set is completed, the target data set and at least one sampling point data to be sorted are obtained;
[0155] Arrange at least one sampling point data to be sorted in ascending order or descending order to obtain a sampling point data sequence.
[0156] In one or more embodiments, a method of calling a single thread to scan and sample data is introduced. As can be seen from the above embodiments, scanning the primary key index can generate an unordered secondary index data file. The parallel scanning phase mainly has two tasks, one of which is to scan the primary key record for each index to be created. The other task is to perform data sampling.
[0157] Specifically, for ease of understanding, see Figure 5 , Figure 5 This is a schematic diagram of single-thread serial scan sampling in an embodiment of the present application. As shown in the figure, taking a thread (i.e., target thread) scan as an example, based on this, the target thread can be called to scan three different indexes in the original data set in sequence, thereby obtaining a data set corresponding to each index. At the same time, the data under each index can also be sampled according to a preset sampling interval to obtain at least one sampling point data to be sorted corresponding to each index. Among them, an index can specify any field as an index column, which is not limited here.
[0158] Based on this, when the target thread completes scanning the original data set, the target data set corresponding to the target field can be obtained. Optionally, data sets corresponding to other fields can also be obtained. It should be noted that for ease of introduction, this application uses the target data set as an example for illustration. It is understandable that data sets constructed based on other indexes are also processed in a similar manner, which will not be elaborated here.
[0159] While scanning the data, the target thread can also sample the data under the target field according to a preset sampling interval. One feasible way is to extract data as sampling point data according to a fixed sampling interval, and another way is to extract data as sampling point data according to a random sampling interval, thereby obtaining at least one sampling point data to be sorted. Based on this, these sampling point data can be sorted from large to small (i.e., descending order) or from small to large (i.e., ascending order). This application will introduce the sorting of sampling point data from small to large (i.e., ascending order) as an example, but it should not be understood as a limitation of this application. At this point, a sampling point data sequence is obtained.
[0160] Secondly, in the embodiment of the present application, a method of calling single-thread scanning and sampling data is provided. Through the above method, a single-thread serial scanning method is used when scanning the original data set, so that the data scanning work can be completed even when the number of threads is insufficient, thereby improving the flexibility and operability of data scanning.
[0161] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, obtaining a sampling point data sequence from a target data set may specifically include:
[0162] Call S threads to scan the original data set to obtain a target data set, wherein the original data set includes a data set corresponding to the target field, the target data set includes at least one data subset, each data subset is obtained by scanning a thread, and S is an integer greater than 1;
[0163] Calling at least two threads to scan the target data set according to a preset sampling interval to obtain at least one sampling point data to be sorted;
[0164] Arrange at least one sampling point data to be sorted in ascending order or descending order to obtain a sampling point data sequence.
[0165] In one or more embodiments, a method of calling multiple threads to first scan in parallel and then sample data in parallel is introduced. As can be seen from the above embodiments, scanning the primary key index can generate an unordered secondary index data file. The scanning phase mainly has two tasks. First, the primary key record needs to be scanned for each index to be created, and then data sampling needs to be performed.
[0166] Specifically, exemplarily, taking 3 threads (i.e., S is equal to 3) for parallel scanning as an example, based on this, 3 threads can be called to scan three different indexes in the original data set in parallel, thereby obtaining a data set corresponding to each index. Exemplarily, taking 12 threads (i.e., S is equal to 12) for parallel scanning as an example, based on this, 12 threads can be called to scan three different indexes in the original data set in parallel, assuming that each index is scanned in parallel by 4 threads, thereby obtaining a data set corresponding to each index. Among them, taking the target data set as an example, the target data set includes 4 data subsets, each of which is obtained by scanning a thread.
[0167] When the target thread completes scanning the original data set, the target data set corresponding to the target field can be obtained. Optionally, data sets corresponding to other fields can also be obtained. It should be noted that, for ease of introduction, this application uses the target data set as an example for illustration. It is understandable that data sets constructed based on other indexes are also processed in a similar manner, which will not be elaborated here.
[0168] After obtaining the target data set, at least two threads may be called to scan the target data set according to a preset sampling interval. For example, one thread scans backward from the first data in the target data set. Another thread scans forward from the last data in the target data set. When the scanned parts of the two threads overlap, the scanning ends, thereby obtaining at least one sampling point data to be sorted.
[0169] One feasible way is to extract data as sampling point data according to a fixed sampling interval, and another way is to extract data as sampling point data according to a random sampling interval, thereby obtaining at least one sampling point data to be sorted. Based on this, these sampling point data can be sorted from large to small (i.e., descending order) or from small to large (i.e., ascending order). This application will take the sorting of sampling point data from small to large (i.e., ascending order) as an example for introduction, but it should not be understood as a limitation of this application. At this point, a sampling point data sequence is obtained.
[0170] Secondly, in the embodiment of the present application, a method is provided to call multiple threads to scan in parallel first and then sample data in parallel. Through the above method, a multi-threaded parallel scanning method is used when scanning the original data set, wherein the core idea is mainly to split the range of the index tree corresponding to the primary key index, and different threads are responsible for scanning the data in these ranges in parallel, thereby achieving the effect of accelerated scanning. After that, each index is scanned in parallel to obtain multiple data subsets, thereby providing a feasible way to implement the solution.
[0171] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, obtaining a sampling point data sequence from a target data set may specifically include:
[0172] Calling a target thread to scan an original data set to obtain a target data set, wherein the original data set includes a data set corresponding to a target field;
[0173] Calling the target thread, scanning the target data set according to a preset sampling interval, and obtaining at least one sampling point data to be sorted;
[0174] Arrange at least one sampling point data to be sorted in ascending order or descending order to obtain a sampling point data sequence.
[0175] In one or more embodiments, a method of calling a single thread to first scan in parallel and then sample data in parallel is introduced. As can be seen from the above embodiments, scanning the primary key index can generate an unordered secondary index data file. The scanning phase mainly has two tasks. First, the primary key record needs to be scanned for each index to be created, and then data sampling needs to be performed.
[0176] Specifically, exemplarily, taking a thread (i.e., target thread) scanning as an example, based on this, the target thread can be called to scan different indexes in the original data set, thereby obtaining the data set corresponding to each index. Taking the scanning to obtain the target data set as an example, the target thread scans the data corresponding to the target field in the original data set to obtain the target data set. It should be noted that for ease of introduction, this application uses the target data set as an example for explanation. It can be understood that data sets constructed based on other indexes are also processed in a similar manner, which will not be repeated here.
[0177] After obtaining the target data set, the target thread may be called to scan the target data set according to a preset sampling interval. For example, scanning is performed backwards from the first data in the target data set, or scanning is performed forwards from the last data in the target data set. The scanning is completed, thereby obtaining at least one sampling point data to be sorted.
[0178] One feasible way is to extract data as sampling point data according to a fixed sampling interval, and another way is to extract data as sampling point data according to a random sampling interval, thereby obtaining at least one sampling point data to be sorted. Based on this, these sampling point data can be sorted from large to small (i.e., descending order) or from small to large (i.e., ascending order). This application will take the sorting of sampling point data from small to large (i.e., ascending order) as an example for introduction, but it should not be understood as a limitation of this application. At this point, a sampling point data sequence is obtained.
[0179] Secondly, in the embodiment of the present application, a method of calling a single thread to first perform parallel scanning and then perform parallel sampling of data is provided. Through the above method, a single thread serial scanning method is used when scanning the original data set, so that even when the number of threads is insufficient, the data scanning work can be completed, thereby improving the flexibility and operability of data scanning.
[0180] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, determining at least one quantile data from the sampling point data sequence may specifically include:
[0181] According to the sampling point data sequence, determine the total number of sampling point data;
[0182] According to the total number of data partitions to be divided, determine the total number of quantile data;
[0183] The total number of sampling point data and the total number of quantile data are divided to obtain the target quotient value;
[0184] According to integer multiples of the target quotient value, sampling point data are extracted from the sampling point data sequence as quantile data to obtain at least one quantile data.
[0185] In one or more embodiments, a method for extracting quantile data based on a sampling point data sequence is introduced. It can be seen from the above embodiments that K quantile data can divide the target data set into (K+1) unordered data partitions. For each data set corresponding to an index, the corresponding sampling point data sequence can be obtained respectively, and then at least one quantile data is determined according to the sampling point data sequence. The following will take the target data set as an example to introduce the process of determining at least one quantile data from the sampling point data sequence.
[0186] For example, it is assumed that the data obtained after sampling is:
[0187] 290, 571, 822, 312, 10, 1203, 96
[0188] The sampling point data sequence obtained after ascending order is:
[0189] 10, 96, 290, 312, 571, 822, 1203
[0190] Based on this, the total number of sampling point data is determined to be 7 according to the sampling point data sequence. Assuming that the total number of data partitions to be divided is 3, the total number of quantile data is determined to be 2, that is, the total number of quantile data is the total number of data partitions minus 1. Therefore, the total number of sampling point data (for example, 7) and the total number of quantile data (for example, 2) are divided to obtain the target quotient value (for example, 3). Therefore, the sampling point data are extracted from the sampling point data sequence as quantile data according to the integer multiples of the target quotient value. For example, if the target quotient value is 3, the third sampling point data (that is, 290) is extracted from the sampling point data sequence as a quantile data, and the sixth sampling point data (that is, 822) is extracted as another quantile data. It can be considered that these two quantile data represent the global data distribution.
[0191] Secondly, in the embodiment of the present application, a method for extracting quantile data based on a sampling point data sequence is provided. Through the above method, quantile data is extracted from the sampling point data sequence at reasonable intervals, thereby, to a certain extent, the data distribution can be made more uniform to a certain extent, avoiding the imbalance of data volume in each disordered data partition, thereby being more in line with the conditions for parallel processing.
[0192] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, according to at least one quantile data, the target data set is divided into at least two unordered data partitions, which may specifically include:
[0193] Determine at least two data interval ranges according to at least one quantile data;
[0194] The target thread is called to divide the data corresponding to the target field in the target data set into corresponding data intervals to obtain at least two unordered data partitions.
[0195] In one or more embodiments, a method for implementing data partitioning based on a single thread is introduced. From the above embodiments, it can be seen that the target data set can be divided into (K+1) unordered data partitions according to K quantile data, where each unordered data partition corresponds to a data interval range.
[0196] Specifically, combined with the above example, assume that the sampling point data sequence is:
[0197] 10, 96, 290, 312, 571, 822, 1203
[0198] Assuming that the total number of data partitions to be divided is 3, based on this, two quantile data can be extracted from the sampling point data sequence, which are 290 and 822 respectively. Thus, three data interval ranges are obtained. Among them, the first data interval range is less than or equal to 290. The second data interval range is greater than 290 and less than or equal to 822. The third data interval range is greater than 822. Therefore, a thread (i.e., the target thread) can be called to divide the data corresponding to the target field in the target data set. Among them, the target thread can start from the first data in the target data set to divide the data backward, or the target thread can start from the last data in the target data set to divide the data forward. After the data corresponding to the target field in the target data set is divided, the unordered data partition corresponding to each data interval range can be obtained.
[0199] Taking the quantile data as Q1 as an example, illustratively, the first division method is that one data interval range is less than or equal to Q1, and the other data interval range is greater than Q1. Exemplarily, the second division method is that one data interval range is less than Q1, and the other data interval range is greater than or equal to Q1. It can be understood that the present application can adopt the first division method or the second division method, which is not limited here.
[0200] Secondly, in an embodiment of the present application, a method for implementing data partitioning based on a single thread is provided. Through the above method, after the quantile data is generated for each index, the data corresponding to the target field in the target data set is divided into the corresponding data interval range according to these quantile data. Since the sampling point data sequence is ordered, the extracted quantile data is also ordered. Because of this, the unordered data partitions are also ordered. It is relatively simple in the external sorting stage, and multiple threads sort the data in the unordered data partitions completely independently and in parallel. In this way, the multiple ordered data partitions finally generated are also globally ordered, thereby improving the sorting efficiency.
[0201] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, according to at least one quantile data, the target data set is divided into at least two unordered data partitions, which may specifically include:
[0202] Determine at least two data interval ranges according to at least one quantile data;
[0203] Call N threads to divide the data corresponding to the target field in the N data subsets into corresponding data interval ranges to obtain at least two unordered data partitions, wherein the N data subsets belong to the target data set, and each thread is used to divide the data in a data subset into corresponding data area ranges, and N is an integer greater than 1.
[0204] In one or more embodiments, a method for implementing data partitioning based on multithreading is introduced. From the above embodiments, it can be seen that the target data set can be divided into (K+1) unordered data partitions according to K quantile data, wherein each unordered data partition corresponds to a data interval range.
[0205] Specifically, combined with the above example, assume that the sampling point data sequence is:
[0206] 10, 96, 290, 312, 571, 822, 1203
[0207] Assuming that the total number of data partitions to be divided is 3, based on this, two quantile data can be extracted from the sampling point data sequence, which are 290 and 822. Thus, three data interval ranges are obtained. Among them, the first data interval range is less than or equal to 290. The second data interval range is greater than 290 and less than or equal to 822. The third data interval range is greater than 822. Therefore, N threads (for example, 3 threads) can be called to divide the data corresponding to the target field in the target data set.
[0208] For easier understanding, see Figure 6 , Figure 6 This is a schematic diagram of partitioning according to point data in an embodiment of the present application. As shown in the figure, it is assumed that the target data set includes 3 data subsets, wherein each data subset is obtained by scanning by a thread. Therefore, N threads (i.e., 3 threads) can be called to divide the data corresponding to the target fields in the 3 data subsets respectively. Exemplarily, thread 1 divides the data in data subset 1, thereby obtaining data within three data intervals respectively. Thread 2 divides the data in data subset 2, thereby obtaining data within three data intervals respectively. Thread 3 divides the data in data subset 3, thereby obtaining data within three data intervals respectively. Among them, the white area represents the first data interval range, the gray area represents the second data interval range, and the black area represents the third data interval range.
[0209] Based on this, the local partition results obtained by each thread are merged to obtain the global unordered data partition. Figure 6For example, we get 3 unordered data partitions. Among them, all data in unordered data partition 1 is less than the first quantile data. All data in unordered data partition 2 is greater than or equal to the first quantile data, and less than the second quantile data. All data in unordered data partition 3 is greater than the second quantile data.
[0210] It should be noted that the division of the data interval range based on the quantile data is as described in the above embodiment and will not be repeated here.
[0211] Secondly, in an embodiment of the present application, a method for implementing data partitioning based on multi-threading is provided. Through the above method, after generating quantile data for each index, multiple threads can be used in parallel to divide the data corresponding to the target field in the target data set into the corresponding data interval range according to these quantile data, so as to improve the efficiency of index creation. And because the sampling point data sequence is ordered, the extracted quantile data is also ordered. Therefore, the unordered data partitions are also ordered. It is relatively simple in the external sorting stage, thereby improving the sorting efficiency.
[0212] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, calling at least two threads to sort the data in at least two unordered data partitions to obtain at least two ordered data partitions may specifically include:
[0213] N threads are called to sort the data in the N unordered data partitions to obtain N ordered data partitions, wherein each thread is used to sort the data in one unordered data partition, and N is an integer greater than 1.
[0214] In one or more embodiments, a method for implementing parallel sorting based on multi-threading is introduced. It can be seen from the above embodiments that after obtaining N unordered data partitions, the same number of threads can be called to sort the data in the N unordered data partitions.
[0215] Specifically, assuming that N is 3, based on this, after obtaining three unordered data partitions, thread 1 can be called to sort the data corresponding to the target field in unordered data partition 1, and after the sorting is completed, ordered data partition 1 is obtained. Similarly, thread 2 is called to sort the data corresponding to the target field in unordered data partition 2, and after the sorting is completed, ordered data partition 2 is obtained. Thread 3 is called to sort the data corresponding to the target field in unordered data partition 3, and after the sorting is completed, ordered data partition 3 is obtained. It should be noted that thread 1, thread 2 and thread 3 work in parallel, so the data sorting efficiency can be improved.
[0216] It can be seen that after the internal sorting of the data partitions, the ordered data partitions are obtained. And the external sorting has been realized between the data partitions, so the N ordered data partitions generated at last are also globally ordered.
[0217] Secondly, in the embodiment of the present application, a method for implementing parallel sorting based on multithreading is provided. Through the above method, after obtaining at least two unordered data partitions according to the quantile data division, one thread can be used to sort the data in each unordered data partition. Thus, parallel sorting of multiple unordered data partitions is achieved, thereby improving the parallelism of index creation, which is conducive to improving the efficiency of index creation.
[0218] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, constructing a database index corresponding to the target field according to at least two ordered data partitions may specifically include:
[0219] Call N threads to build indexes for N ordered data partitions to obtain N index trees, where each thread is used to build a corresponding index tree according to the ordered data partition, and N is an integer greater than 1;
[0220] A target index tree corresponding to the target field is constructed according to the N index trees, wherein the target index tree belongs to a database index.
[0221] In one or more embodiments, a method of implementing parallel tree building based on multi-threading is introduced. It can be seen from the aforementioned embodiments that N ordered data partitions have been obtained in the previous stage. Based on this, N threads can be called to build the corresponding index trees for these N ordered data partitions. Finally, these index trees are combined into a target index tree, which is the database index for the target field. This application takes the construction of a B+ tree as an example for explanation. It can be understood that in actual applications, it also supports the construction of other ordered data structures.
[0222] Specifically, for ease of understanding, see Figure 7 , Figure 7 This is a schematic diagram of constructing a target index tree in parallel in an embodiment of the present application. As shown in the figure, it is assumed that there are three ordered data partitions, namely ordered data partition 1, ordered data partition 2 and ordered data partition 3. Based on this, thread 1 can be called to construct a corresponding index tree 1 for the data in ordered data partition 1. Thread 2 can be called to construct a corresponding index tree 2 for the data in ordered data partition 2. Thread 3 can be called to construct a corresponding index tree 3 for the data in ordered data partition 3. Index tree 1, index tree 2 and index tree 3 are three subtrees of the target index tree. Therefore, the three index trees are merged to form a target index tree.
[0223] It can be understood that, in the stage of constructing a B+ tree (i.e., the target index tree), N threads are responsible for the N ordered data partitions generated in the sorting stage, and construct sub-B+ trees (i.e., index trees) for them respectively. Since the N ordered data partitions are globally ordered, all sub-B+ trees (i.e., N index trees) are merged to obtain an overall B+ tree (i.e., the target index tree), which is the database index structure that is ultimately desired.
[0224] Secondly, in an embodiment of the present application, a method for creating an index tree in parallel based on multi-threading is provided. Through the above method, multiple threads are used to create an index tree in parallel, thereby improving the parallelism of index creation, which is conducive to improving the efficiency of index creation. At the same time, since the present application can achieve multi-threaded parallelism when scanning the entire primary key index file, it can achieve multi-threaded parallelism when sorting each unordered data partition, and it can also achieve multi-threaded parallelism when building an index tree. Therefore, when it comes to database operations such as index creation and reconstruction, it is able to significantly reduce the time for index creation, thereby reducing the delay of user operations.
[0225] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, constructing a target index tree corresponding to the target field according to N index trees may specifically include:
[0226] Determine the highest tree level among the N index trees;
[0227] According to the highest tree number, add root nodes to the index trees whose tree number is less than the highest tree number among the N index trees, until N index trees with the same tree number are obtained;
[0228] Perform node merging processing on N index trees with the same number of tree layers to obtain the target index tree corresponding to the target field.
[0229] In one or more embodiments, a method for merging multiple index trees is introduced. As can be seen from the above embodiments, after constructing N index trees, these index trees need to be processed according to the characteristics of the target index tree (for example, the characteristic of the B+ tree is that the left and right child nodes can be connected by pointers).
[0230] Specifically, the layers of N index trees can be horizontally strung together first. If the tree heights of the index trees are inconsistent, the tree heights of the index trees with insufficient heights can be filled up according to the highest index tree, and then a new tree root can be generated. Filling up the tree height can be achieved by adding a root node. Finally, the tree height of the root layer of the index tree is compressed. The compressed tree height can determine whether the child nodes under the root node can be merged into the root node. If the amount of data can be stored in one node, it means that it can be merged, that is, the tree height can be compressed.
[0231] For example, the following will be combined with Figure 8 , introduces the method of filling the tree height. Assume that the highest tree level in N index trees is three levels. For ease of understanding, please refer to Figure 8 , Figure 8 This is a schematic diagram of completing the height of the index tree in the embodiment of the present application. As shown in the figure, the index tree on the left has only two layers, so the original root node can be copied and the copied node can be used as the new root node of the index tree. For example, the original root node is "2 5 8", and after completing the tree height, the root node is still "2 58".
[0232] For example, the following will be combined with Fig. 9 , introduces the way to compress tree height. For easier understanding, please refer to Fig. 9 , Fig. 9 This is a schematic diagram of the compressed index tree height in the embodiment of the present application. As shown in the figure, it is assumed that the above figure is an index tree obtained by merging or obtaining two index trees. Based on this, the node "2 6 8" can be merged with the node "11 15", and thus, the leaf node "8" and the leaf node "10 11" can also be merged into one leaf node. Thus, the original three-layer tree height is compressed to a two-layer tree height. Finally, the target index tree corresponding to the target field can be obtained.
[0233] Again, in an embodiment of the present application, a method for merging multiple index trees is provided. Through the above method, on the one hand, the tree height of each index tree can be filled, so that the left and right child nodes can be connected by pointers, thereby meeting the construction requirements of the B+ tree. On the other hand, after filling the tree height, the tree height can also be compressed, thereby achieving the purpose of optimizing the data structure.
[0234] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, constructing a database index corresponding to the target field according to at least two ordered data partitions may specifically include:
[0235] The target thread is called to construct an index for the N ordered data partitions to obtain a target index tree, wherein the target index tree belongs to a database index.
[0236] In one or more embodiments, a method of establishing an index tree based on a single thread is introduced. It can be seen from the above embodiments that N ordered data partitions have been obtained in the previous stage, based on which a thread (i.e., a target thread) can be called to construct a target index tree corresponding to the N ordered data partitions. This application takes the construction of a B+ tree as an example for explanation, and it can be understood that in actual applications, it also supports the construction of other ordered data structures.
[0237] Specifically, assuming that there are three ordered data partitions, namely, ordered data partition 1, ordered data partition 2, and ordered data partition 3, since each ordered data partition also satisfies a certain order, the data of ordered data partition 1, the data of ordered data partition 2, and the data of ordered data partition 3 can be concatenated in the corresponding order to obtain a target data sequence. Therefore, the target thread can be called to construct a corresponding target index tree for the data of the target data sequence.
[0238] For example, if the data in the ordered data partitions satisfy the ascending order rule, then the ordered data partitions should also satisfy the ascending order rule. For example, the data in the ordered data partition 1 are "15, 18, 23, 36, 42", and the data in the ordered data partition 2 are "48, 55, 66, 79, 87, 100". Based on this, the target data sequence is "15, 18, 23, 36, 42, 48, 55, 66, 79, 87, 100".
[0239] For example, if the data in the ordered data partitions satisfy the descending order rule, then the ordered data partitions should also satisfy the descending order rule. For example, the data in the ordered data partition 1 is "100, 87, 79, 66, 55", and the data in the ordered data partition 2 is "48, 42, 36, 23, 18, 15". Based on this, the target data sequence is "100, 87, 79, 66, 55, 48, 42, 36, 23, 18, 15".
[0240] Secondly, in the embodiment of the present application, a method for establishing an index tree based on a single thread is provided. Through the above method, it is supported to use a single thread to build a target index tree corresponding to a target field, thereby increasing the feasibility and operability of the solution.
[0241] In combination with the above introduction, another database index creation method in this application will be introduced below. Fig.10 , the method for creating a database index in the embodiment of the present application includes:
[0242] 210. Call N threads to scan the original data set to obtain N data subsets, wherein each data subset includes a data set corresponding to the target field, and each data subset is obtained by scanning one thread, and N is an integer greater than 1;
[0243] In one or more embodiments, the original data set is obtained, and N threads are called as required to scan the original data set, that is, all data sets containing the target field are scanned, wherein the original data set includes a data set corresponding to at least one field. Assuming that 8 threads are called to scan the original data set, 8 data subsets can be obtained. It should be noted that the "target field" in this application can be a field or a field set consisting of multiple fields, which is not limited here.
[0244] 220. Call N threads to sort the data in the N data subsets to obtain N first data sequences, wherein each thread in the N threads is used to sort the data in one data subset;
[0245] In one or more embodiments, N threads may be called to sort the data in the N data subsets, that is, each thread sorts the data in a data subset in ascending order or descending order, so as to achieve the purpose of parallel sorting. Thus, N first data sequences are obtained.
[0246] 230. Call N / 2 threads to perform two-way merge sorting on the N first data sequences to obtain N / 2 second data sequences until a target data sequence is obtained, wherein each thread of the N / 2 threads is used to sort two first data sequences;
[0247] In one or more embodiments, after obtaining N first data sequences, a two-way merge method may be used to sort the N first data sequences. Since each thread is responsible for merging two of the first data sequences, N / 2 threads may be used here. After sorting, N / 2 second data sequences are obtained. Until a global target data sequence is obtained. It is understandable that if there are only two first data sequences, then calling one thread can obtain the target data sequence.
[0248] Specifically, the following will introduce the process of generating the target data sequence with the help of diagrams. Fig.11 , Fig.11This is a schematic diagram of generating a target data sequence based on a two-way merge method in an embodiment of the present application. As shown in the figure, first, 8 threads are called to sort 8 data subsets, thereby obtaining 8 ordered first data sequences. Based on this, thread 1 is called to perform a two-way merge sort on first data sequence 1 and first data sequence 2 to generate second data sequence 1. Thread 2 is called to perform a two-way merge sort on first data sequence 3 and first data sequence 4 to generate second data sequence 2. Thread 3 is called to perform a two-way merge sort on first data sequence 5 and first data sequence 6 to generate second data sequence 3. Thread 4 is called to perform a two-way merge sort on first data sequence 7 and first data sequence 8 to generate second data sequence 4. Four parallel threads are used in this round of loops.
[0249] Next, in the next loop, thread 1 is called to perform two-way merge sorting on the second data sequence 1 and the second data sequence 2 to generate the third data sequence 1. Thread 2 is called to perform two-way merge sorting on the second data sequence 3 and the second data sequence 4 to generate the third data sequence 2. Two parallel threads are used in this loop.
[0250] Finally, the next loop is entered, and thread 1 is called to perform a two-way merge sort on the third data sequence 1 and the third data sequence 2 to generate the target data sequence. One thread is used in this loop.
[0251] 240. Construct a database index corresponding to the target field according to the target data sequence.
[0252] In one or more embodiments, since an ordered target data sequence has been obtained, a database index corresponding to the target field can be constructed based on the target data sequence. It should be noted that the present application can call a single thread to construct a database index corresponding to the target field, or call multiple threads to construct a database index corresponding to the target field. The database index can be represented as a target index tree.
[0253] Specifically, one way is to call a thread (i.e., the target thread) to construct a target index tree corresponding to the target data sequence. Another way is to divide the target data sequence into N sub-data sequences, where the data in each sub-data sequence is also ordered, and N is an integer greater than 1. In this case, N threads are called to construct the corresponding index trees for the N sub-data sequences, and finally, these index trees are combined into a target index tree.
[0254] In an embodiment of the present application, another method for creating a database index is provided. First, N threads are called to scan the original data set to obtain N data subsets, and then each data subset is sorted respectively, so that several internally ordered data sequences can be obtained. At this point, multiple threads can be called to perform a two-way merge sort on these data sequences, thereby obtaining a globally ordered target data sequence as a basis for constructing a database index. By sorting multiple data sequences in parallel, the parallelism of index creation can be improved and the time required to create the index can be reduced.
[0255] In combination with the above introduction, another database index creation method in this application will be introduced below. Fig.12 , the method for creating a database index in the embodiment of the present application includes:
[0256] 310. Call N threads to scan the original data set to obtain N first data sets, wherein each first data set includes a data set corresponding to the target field, and each first data set is obtained by scanning one thread, and N is an integer greater than 1;
[0257] In one or more embodiments, the original data set is obtained, and N threads are called as required to scan the original data set, that is, all data sets containing the target field are scanned, wherein the original data set includes a data set corresponding to at least one field. Assuming that 4 threads are called to scan the original data set, 4 first data sets can be obtained. It should be noted that the "target field" in this application can be a field or a field set consisting of multiple fields, which is not limited here.
[0258] 320. Call N threads to sort the data in the N first data sets to obtain N first data sequences, wherein each thread in the N threads is used to sort the data in one first data set;
[0259] In one or more embodiments, N threads may be called to sort the data in the N first data sets, that is, each thread sorts the data in one first data set in ascending order or descending order, so as to achieve the purpose of parallel sorting. Thus, N first data sequences are obtained.
[0260] 330. Determine, according to the N first data sequences, (N-1) pivots corresponding to each first data sequence, wherein each pivot is used to divide data in the first data sequence;
[0261] In one or more embodiments, after obtaining N first data sequences, if it is desired to call N threads to perform N-way merge on the N first data sequences (i.e., the N threads are fully utilized), each first data sequence may be further divided into N ranges, so that the data in the first range of each first data sequence is smaller than the data in the second range, and the data in all second ranges is smaller than the data in the third range, and so on. Finally, N threads may be used to perform N-way merge in parallel, and finally a globally ordered target data sequence may be obtained. The position where the range is located is the pivot.
[0262] Specifically, in order to evenly distribute the data in each range, the following example will introduce how to determine the pivot. Fig.13 , Fig.13 This is a schematic diagram of determining a pivot based on a data set in an embodiment of the present application. As shown in the figure, four first data sequences are obtained after sorting. If four threads are used to merge and sort the four first data sequences, each first data sequence needs to be divided into four ranges.
[0263] Based on this, first determine the first pivot of the first data sequence 1. In the initial stage, an equal division method can be used. Assuming that the first data sequence 1 has a total of 10 data, the first data sequence 1 needs to determine a total of 3 pivots, where the position of the first pivot can be selected after the third data. According to the first pivot of the first data sequence 1, the first pivots of the first data sequence 2, the first data sequence 3 and the first data sequence 4 can be found. In this example, 30 is used as the pivot. If the total number of values determined by the first pivots of each first data sequence is greater than the average value that each range should have, then reduce the first pivot in the first data sequence 1, and calculate the first pivots in all the first data sequences again until the difference between the total number of values determined by the first pivot and the average value that each range should have is within a smaller interval (for example, 5), and thus, it is considered that the pivot has been found. It can be understood that a similar method can be used to calculate the positions of the second pivot and the third pivot, which will not be repeated here.
[0264] 340. Divide the N first data sequences into N second data sets according to (N-1) pivots;
[0265] In one or more embodiments, each first data sequence is divided into N second data sets based on the (N-1) pivots corresponding to each first data sequence.
[0266] Specifically, for ease of understanding, please refer again to Fig.13, after determining the pivot corresponding to each first data sequence, four second data sets are obtained according to the pivot position. Among them, the second data set 1 includes "15 20 30 25 510 15 5 10", the second data set 2 includes "40 50 60 70 80 35 55 50 45", the second data set 3 includes "90 100 85 95 105 85 100 90 100", and the second data set 4 includes "205 300 400 120 125 200130 150 200". It can be seen that although the data in each second data set is disordered, the second data sets are ordered, that is, the data in the second data set 1 are all smaller than the data in the second data set 2, the data in the second data set 2 are all smaller than the data in the second data set 3, and the data in the second data set 3 are all smaller than the data in the second data set 4.
[0267] 350. Call N threads to sort the data in the N second data sets to obtain N second data sequences, wherein each thread in the N threads is used to sort the data in one second data set;
[0268] In one or more embodiments, N threads are called to sort the data in N second data sets in parallel, that is, each thread sorts the data in one second data set, thereby obtaining N ordered second data sequences.
[0269] 360. Construct a database index corresponding to the target field according to the N second data sequences.
[0270] In one or more embodiments, N second data sequences are spliced in the order of the second data set to obtain an ordered target data sequence, and therefore, a database index corresponding to the target field can be constructed based on the target data sequence. It should be noted that the present application can call a single thread to construct a database index corresponding to the target field, or call multiple threads to construct a database index corresponding to the target field. The database index can be represented as a target index tree.
[0271] Specifically, one way is to call a thread (ie, target thread) to construct a target index tree corresponding to the target data sequence. Another way is to call N threads to construct index trees corresponding to N second data sequences in parallel, and finally, combine these index trees into a target index tree.
[0272] In an embodiment of the present application, another method for creating a database index is provided. First, N threads are called to scan the original data set to obtain N data subsets, and then each data subset is sorted respectively, so that a number of internally ordered data sequences can be obtained. At this point, the corresponding pivot is determined for each data sequence. Since the pivot partitioning method can accurately know how many data are in each data set, the problem of uneven data distribution can be solved.
[0273] The index creation device in this application is described in detail below. Fig.14 , Fig.14 This is a schematic diagram of an embodiment of an index creation device in an embodiment of the present application. The index creation device 40 includes:
[0274] An acquisition module 410 is used to acquire a sampling point data sequence from a target data set, wherein the target data set includes a data set corresponding to a target field, and the sampling point data sequence includes at least one sampling point data in ascending order or descending order;
[0275] A determination module 420, configured to determine at least one quantile data from a sampling point data sequence;
[0276] A partitioning module 430, configured to partition the target data set into at least two unordered data partitions according to at least one quantile data;
[0277] A sorting module 440 is used to call at least two threads to sort the data in at least two unordered data partitions to obtain at least two ordered data partitions, wherein the ordered data partitions have a one-to-one correspondence with the unordered data partitions;
[0278] The construction module 450 is used to construct a database index corresponding to the target field according to at least two ordered data partitions.
[0279] In an embodiment of the present application, an index creation device is provided. By using the above device, on the one hand, the data in several unordered data partitions are sorted in parallel at the same time, thereby improving the parallelism of index creation and reducing the time required for index creation. On the other hand, since each data partition is ordered externally, there is no need for external sorting, thereby further improving the efficiency of index creation.
[0280] Optionally, in the above Fig.14 On the basis of the corresponding embodiment, in another embodiment of the index creation device 40 provided in the embodiment of the present application,
[0281] The acquisition module 410 is specifically used to sample and process the data corresponding to the target field according to a preset sampling interval when calling S threads to scan the original data set, wherein the original data set includes the data set corresponding to the target field, and S is an integer greater than 1;
[0282] When the scanning of the original data set is completed, a target data set and at least one sampling point data to be sorted are obtained, wherein the target data set includes at least one data subset, and each data subset is obtained by scanning a thread;
[0283] Arrange at least one sampling point data to be sorted in ascending order or descending order to obtain a sampling point data sequence.
[0284] In an embodiment of the present application, an index creation device is provided. Using the above device, a multi-threaded parallel scanning method is used when scanning the original data set, wherein the core idea is mainly to split the range of the index tree corresponding to the primary key index, and different threads are responsible for scanning the data in these ranges in parallel, so as to achieve the effect of accelerated scanning. At the same time, each index can also be scanned in parallel to obtain multiple data subsets, thereby further improving the efficiency of data scanning.
[0285] Optionally, in the above Fig.14 On the basis of the corresponding embodiment, in another embodiment of the index creation device 40 provided in the embodiment of the present application,
[0286] The acquisition module 410 is specifically used to sample the data corresponding to the target field according to a preset sampling interval when calling the target thread to scan the original data set, wherein the original data set includes the data set corresponding to the target field;
[0287] When the scanning of the original data set is completed, the target data set and at least one sampling point data to be sorted are obtained;
[0288] Arrange at least one sampling point data to be sorted in ascending order or descending order to obtain a sampling point data sequence.
[0289] In an embodiment of the present application, an index creation device is provided. With the above device, a single-thread serial scanning method is used when scanning the original data set, so that the data scanning work can be completed even when the number of threads is insufficient, thereby improving the flexibility and operability of data scanning.
[0290] Optionally, in the above Fig.14 On the basis of the corresponding embodiment, in another embodiment of the index creation device 40 provided in the embodiment of the present application,
[0291] The acquisition module 410 is specifically used to call S threads to scan the original data set to obtain the target data set, wherein the original data set includes the data set corresponding to the target field, the target data set includes at least one data subset, each data subset is obtained by scanning one thread, and S is an integer greater than 1;
[0292] Calling at least two threads to scan the target data set according to a preset sampling interval to obtain at least one sampling point data to be sorted;
[0293] Arrange at least one sampling point data to be sorted in ascending order or descending order to obtain a sampling point data sequence.
[0294] In the embodiment of the present application, an index creation device is provided. With the above device, a multi-threaded parallel scanning method is used when scanning the original data set, wherein the core idea is mainly to split the range of the index tree corresponding to the primary key index, and different threads are responsible for scanning the data in these ranges in parallel, so as to achieve the effect of accelerating the scanning. Then, each index is scanned in parallel to obtain multiple data subsets, thereby providing a feasible way to implement the solution.
[0295] Optionally, in the above Fig.14 On the basis of the corresponding embodiment, in another embodiment of the index creation device 40 provided in the embodiment of the present application,
[0296] The acquisition module 410 is specifically used to call the target thread to scan the original data set to obtain the target data set, wherein the original data set includes the data set corresponding to the target field;
[0297] Calling the target thread, scanning the target data set according to a preset sampling interval, and obtaining at least one sampling point data to be sorted;
[0298] Arrange at least one sampling point data to be sorted in ascending order or descending order to obtain a sampling point data sequence.
[0299] In an embodiment of the present application, an index creation device is provided. With the above device, a single-thread serial scanning method is used when scanning the original data set, so that the data scanning work can be completed even when the number of threads is insufficient, thereby improving the flexibility and operability of data scanning.
[0300] Optionally, in the above Fig.14 On the basis of the corresponding embodiment, in another embodiment of the index creation device 40 provided in the embodiment of the present application,
[0301] The determination module 420 is specifically configured to determine the total number of sampling point data according to the sampling point data sequence;
[0302] According to the total number of data partitions to be divided, determine the total number of quantile data;
[0303] The total number of sampling point data and the total number of quantile data are divided to obtain the target quotient value;
[0304] According to integer multiples of the target quotient value, sampling point data are extracted from the sampling point data sequence as quantile data to obtain at least one quantile data.
[0305] In the embodiment of the present application, an index creation device is provided. By using the above device, quantile data is extracted from the sampling point data sequence at reasonable intervals, thereby, to a certain extent, the data distribution can be made more uniform, avoiding the imbalance of data volume in each disordered data partition, thereby being more in line with the conditions for parallel processing.
[0306] Optionally, in the above Fig.14 On the basis of the corresponding embodiment, in another embodiment of the index creation device 40 provided in the embodiment of the present application,
[0307] A partitioning module 430 is specifically configured to determine at least two data interval ranges according to at least one quantile data;
[0308] The target thread is called to divide the data corresponding to the target field in the target data set into corresponding data intervals to obtain at least two unordered data partitions.
[0309] In an embodiment of the present application, an index creation device is provided. Using the above device, after generating quantile data for each index, the data corresponding to the target field in the target data set is divided into corresponding data intervals according to these quantile data. Since the sampling point data sequence is ordered, the extracted quantile data is also ordered. Because of this, the unordered data partitions are also ordered. The external sorting stage is relatively simple, and multiple threads sort the data in the unordered data partitions completely independently and in parallel. In this way, the multiple ordered data partitions finally generated are also globally ordered, thereby improving the sorting efficiency.
[0310] Optionally, in the above Fig.14 On the basis of the corresponding embodiment, in another embodiment of the index creation device 40 provided in the embodiment of the present application,
[0311] A partitioning module 430 is specifically configured to determine at least two data interval ranges according to at least one quantile data;
[0312] Call N threads to divide the data corresponding to the target field in the N data subsets into corresponding data interval ranges to obtain at least two unordered data partitions, wherein the N data subsets belong to the target data set, and each thread is used to divide the data in a data subset into corresponding data area ranges, and N is an integer greater than 1.
[0313] In an embodiment of the present application, an index creation device is provided. Using the above device, after generating quantile data for each index, multiple threads can be used in parallel to divide the data corresponding to the target field in the target data set into corresponding data intervals according to the quantile data, thereby improving the efficiency of index creation. And because the sampling point data sequence is ordered, the extracted quantile data is also ordered. Therefore, the unordered data partitions are also ordered. It is relatively simple in the external sorting stage, thereby improving the sorting efficiency.
[0314] Optionally, in the above Fig.14 On the basis of the corresponding embodiment, in another embodiment of the index creation device 40 provided in the embodiment of the present application,
[0315] The sorting module 440 is specifically used to call N threads to sort the data in the N unordered data partitions to obtain N ordered data partitions, wherein each thread is used to sort the data in one unordered data partition, and N is an integer greater than 1.
[0316] In an embodiment of the present application, an index creation device is provided. Using the above device, after obtaining at least two unordered data partitions according to the quantile data division, a thread can be used to sort the data in each unordered data partition. In this way, multiple unordered data partitions can be sorted in parallel, thereby improving the parallelism of index creation, which is conducive to improving the efficiency of index creation.
[0317] Optionally, in the above Fig.14 On the basis of the corresponding embodiment, in another embodiment of the index creation device 40 provided in the embodiment of the present application,
[0318] The construction module 450 is specifically used to call N threads to perform index construction on the N ordered data partitions to obtain N index trees, wherein each thread is used to construct a corresponding index tree according to the ordered data partition, and N is an integer greater than 1;
[0319] A target index tree corresponding to the target field is constructed according to the N index trees, wherein the target index tree belongs to a database index.
[0320] In an embodiment of the present application, an index creation device is provided. By using the above device, multiple threads are used to create an index tree in parallel, thereby improving the parallelism of index creation, which is beneficial to improving the efficiency of index creation. At the same time, since the present application can achieve multi-threaded parallelism when scanning the entire primary key index file, it can achieve multi-threaded parallelism when sorting each unordered data partition, and it can also achieve multi-threaded parallelism when building an index tree. Therefore, when it comes to database operations such as index creation and reconstruction, it is able to significantly reduce the time for index creation, thereby reducing the delay of user operations.
[0321] Optionally, in the above Fig.14 On the basis of the corresponding embodiment, in another embodiment of the index creation device 40 provided in the embodiment of the present application,
[0322] A construction module 450 is specifically configured to determine the highest tree level number among the N index trees;
[0323] According to the highest tree number, add root nodes to the index trees whose tree number is less than the highest tree number among the N index trees, until N index trees with the same tree number are obtained;
[0324] Perform node merging processing on N index trees with the same number of tree layers to obtain the target index tree corresponding to the target field.
[0325] In an embodiment of the present application, an index creation device is provided. By using the above device, on the one hand, the tree height of each index tree can be filled, so that the left and right child nodes can be connected by pointers, thereby meeting the construction requirements of the B+ tree. On the other hand, after filling the tree height, the tree height can also be compressed, thereby achieving the purpose of optimizing the data structure.
[0326] Optionally, in the above Fig.14 On the basis of the corresponding embodiment, in another embodiment of the index creation device 40 provided in the embodiment of the present application,
[0327] The construction module 450 is specifically used to call the target thread, perform index construction on the N ordered data partitions, and obtain a target index tree, wherein the target index tree belongs to the database index.
[0328] In an embodiment of the present application, an index creation device is provided. The device supports building a target index tree corresponding to a target field in a single thread manner, thereby increasing the feasibility and operability of the solution.
[0329] The index creation device in this application is described in detail below. Fig.15 , Fig.15 This is another embodiment schematic diagram of the index creation device in the embodiment of the present application. The index creation device 50 includes:
[0330] The acquisition module 510 is used to call N threads to scan the original data set to obtain N data subsets, wherein each data subset includes a data set corresponding to the target field, and each data subset is scanned by one thread, and N is an integer greater than 1;
[0331] A sorting module 520 is used to call N threads to sort the data in the N data subsets to obtain N first data sequences, wherein each thread in the N threads is used to sort the data in one data subset;
[0332] The sorting module 520 is further used to call N / 2 threads to perform two-way merge sorting on the N first data sequences to obtain N / 2 second data sequences until a target data sequence is obtained, wherein each thread of the N / 2 threads is used to sort two first data sequences;
[0333] The construction module 530 is used to construct a database index corresponding to the target field according to the target data sequence.
[0334] In an embodiment of the present application, an index creation device is provided. Using the above device, first, N threads are called to scan the original data set to obtain N data subsets, and then each data subset is sorted respectively, so that several internally ordered data sequences can be obtained. At this point, multiple threads can be called to perform two-way merge sorting on these data sequences, thereby obtaining a globally ordered target data sequence as a basis for constructing a database index. By sorting multiple data sequences in parallel, the parallelism of index creation can be improved and the time required to create the index can be reduced.
[0335] The index creation device in this application is described in detail below. Fig.16 , Fig.16 This is another embodiment schematic diagram of the index creation device in the embodiment of the present application. The index creation device 60 includes:
[0336] The acquisition module 610 is used to call N threads to scan the original data set to obtain N first data sets, wherein each first data set includes a data set corresponding to the target field, and each first data set is scanned by one thread, and N is an integer greater than 1;
[0337] A sorting module 620 is used to call N threads to sort the data in the N first data sets to obtain N first data sequences, wherein each thread in the N threads is used to sort the data in one first data set;
[0338] A determination module 630 is used to determine (N-1) pivot axes corresponding to each first data sequence according to the N first data sequences, wherein each pivot axis is used to divide the data in the first data sequence;
[0339] A partitioning module 640, configured to partition the N first data sequences into N second data sets according to (N-1) pivot axes;
[0340] The sorting module 620 is further used to call N threads to sort the data in the N second data sets to obtain N second data sequences, wherein each thread in the N threads is used to sort the data in one second data set;
[0341] The construction module 650 is used to construct a database index corresponding to the target field according to the N second data sequences.
[0342] In an embodiment of the present application, an index creation device is provided. Using the above device, first, N threads are called to scan the original data set to obtain N data subsets, and then each data subset is sorted respectively, so that several internally ordered data sequences can be obtained. At this point, the corresponding pivot is determined for each data sequence. Since the pivot partitioning method can accurately know how many data are in each data set, the problem of uneven data distribution can be solved.
[0343] The present application also provides an index creation device, which is deployed on a server. Fig.17 , Fig.17 : is a schematic diagram of a server structure provided in an embodiment of the present application. The server 700 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 722 (for example, one or more processors) and a memory 732, and one or more storage media 730 (for example, one or more mass storage devices) storing application programs 742 or data 744. Among them, the memory 732 and the storage medium 730 can be short-term storage or permanent storage. The program stored in the storage medium 730 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 722 can be configured to communicate with the storage medium 730 to execute a series of instruction operations in the storage medium 730 on the server 700.
[0344] The server 700 may also include one or more power supplies 726, one or more wired or wireless network interfaces 750, one or more input and output interfaces 758, and / or one or more operating systems 741, such as Windows Server 2000. TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM etc.
[0345] The steps performed by the server in the above embodiment can be based on the Fig.17 The server structure shown.
[0346] A computer device is also provided in an embodiment of the present application, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the methods described in the above embodiments are implemented.
[0347] A computer-readable storage medium is also provided in an embodiment of the present application, on which a computer program is stored. When the computer program is executed by a processor, the steps of the methods described in the above embodiments are implemented.
[0348] A computer program product is also provided in an embodiment of the present application, including a computer program, which, when executed by a processor, implements the steps of the methods described in the above embodiments.
[0349] It can be understood that in the specific implementation of this application, when it comes to data corresponding to the target field, etc., when the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0350] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0351] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0352] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0353] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0354] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store computer programs.
[0355] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for creating a database index. It is characterized in that include: Acquire a sampling point data sequence derived from a target data set, wherein the target data set includes a data set corresponding to a target field, and the sampling point data sequence includes at least one sampling point data in ascending order or descending order; Determine at least one quantile data from the sampling point data sequence; According to the at least one quantile data, the target data set is divided into at least two unordered data partitions; Invoking at least two threads to sort the data in the at least two unordered data partitions to obtain at least two ordered data partitions, wherein the ordered data partitions have a one-to-one correspondence with the unordered data partitions; Constructing a database index corresponding to the target field according to the at least two ordered data partitions; The step of constructing a database index corresponding to the target field according to the at least two ordered data partitions includes: Calling N threads to perform index construction on the N ordered data partitions to obtain N index trees, wherein each thread is used to construct a corresponding index tree according to the ordered data partition, and N is an integer greater than 1; A target index tree corresponding to the target field is constructed according to the N index trees, wherein the target index tree belongs to the database index.
2. The creation method according to claim 1, It is characterized in that The step of obtaining a sampling point data sequence from a target data set includes: When S threads are called to scan the original data set, the data corresponding to the target field is sampled and processed according to a preset sampling interval, wherein the original data set includes the data set corresponding to the target field, and S is an integer greater than 1; When the scanning of the original data set is completed, the target data set and at least one sampling point data to be sorted are obtained, wherein the target data set includes at least one data subset, and each data subset is obtained by scanning a thread; Arrange the at least one sampling point data to be sorted in ascending order or descending order to obtain the sampling point data sequence.
3. The creation method according to claim 1, It is characterized in that The step of obtaining a sampling point data sequence from a target data set includes: When the target thread is called to scan the original data set, the data corresponding to the target field is sampled and processed according to a preset sampling interval, wherein the original data set includes the data set corresponding to the target field; When the scanning of the original data set is completed, the target data set and at least one sampling point data to be sorted are obtained; Arrange the at least one sampling point data to be sorted in ascending order or descending order to obtain the sampling point data sequence.
4. The creation method according to claim 1, It is characterized in that The step of obtaining a sampling point data sequence from a target data set includes: Calling S threads to scan the original data set to obtain the target data set, wherein the original data set includes the data set corresponding to the target field, the target data set includes at least one data subset, each data subset is obtained by scanning one thread, and S is an integer greater than 1; Calling at least two threads to scan the target data set according to a preset sampling interval to obtain at least one sampling point data to be sorted; Arrange the at least one sampling point data to be sorted in ascending order or descending order to obtain the sampling point data sequence.
5. The creation method according to claim 1, It is characterized in that The step of obtaining a sampling point data sequence from a target data set includes: Calling a target thread to scan an original data set to obtain the target data set, wherein the original data set includes a data set corresponding to the target field; Calling the target thread to scan the target data set according to a preset sampling interval to obtain at least one sampling point data to be sorted; Arrange the at least one sampling point data to be sorted in ascending order or descending order to obtain the sampling point data sequence.
6. The creation method according to claim 1, It is characterized in that The step of determining at least one quantile data from the sampling point data sequence comprises: Determine the total number of sampling point data according to the sampling point data sequence; According to the total number of data partitions to be divided, determine the total number of quantile data; The total number of sampling point data and the total number of quantile data are divided to obtain the target quotient value; According to the integer multiples of the target quotient value, sampling point data are extracted from the sampling point data sequence as quantile data to obtain the at least one quantile data.
7. The creation method according to claim 1, It is characterized in that The step of dividing the target data set into at least two unordered data partitions according to the at least one quantile data comprises: Determine at least two data interval ranges according to the at least one quantile data; The target thread is called to divide the data corresponding to the target field in the target data set into corresponding data intervals to obtain the at least two unordered data partitions.
8. The creation method according to claim 1, It is characterized in that The step of dividing the target data set into at least two unordered data partitions according to the at least one quantile data comprises: Determine at least two data interval ranges according to the at least one quantile data; Call N threads to divide the data corresponding to the target field in the N data subsets into corresponding data interval ranges to obtain the at least two unordered data partitions, wherein the N data subsets belong to the target data set, each thread is used to divide the data in a data subset into corresponding data area ranges, and N is an integer greater than 1.
9. The creation method according to any one of claims 1 to 8, It is characterized in that The calling of at least two threads to sort the data in the at least two unordered data partitions to obtain at least two ordered data partitions includes: N threads are called to sort the data in the N unordered data partitions to obtain N ordered data partitions, wherein each thread is used to sort the data in one unordered data partition, and N is an integer greater than 1.
10. The creation method according to claim 1, It is characterized in that The step of constructing a target index tree corresponding to the target field according to the N index trees includes: Determine the highest tree level number among the N index trees; According to the highest tree number, add root nodes to the index trees whose tree number is less than the highest tree number among the N index trees, until N index trees with the same tree number are obtained; Node merging is performed on the N index trees with the same number of tree layers to obtain a target index tree corresponding to the target field.
11. The creation method according to claim 1, It is characterized in that The step of constructing a database index corresponding to the target field according to the at least two ordered data partitions includes: The target thread is called to perform index construction on the N ordered data partitions to obtain a target index tree, wherein the target index tree belongs to the database index.
12. An index creation device, It is characterized in that include: An acquisition module, configured to acquire a sampling point data sequence from a target data set, wherein the target data set includes a data set corresponding to a target field, and the sampling point data sequence includes at least one sampling point data in ascending order or descending order; A determination module, used to determine at least one quantile data from the sampling point data sequence; A partitioning module, configured to partition the target data set into at least two unordered data partitions according to the at least one quantile data; A sorting module, used for calling at least two threads to sort the data in the at least two unordered data partitions to obtain at least two ordered data partitions, wherein the ordered data partitions have a one-to-one correspondence with the unordered data partitions; A construction module, configured to construct a database index corresponding to the target field according to the at least two ordered data partitions; The construction module is specifically used to call N threads to perform index construction on N ordered data partitions to obtain N index trees, wherein each thread is used to construct a corresponding index tree according to the ordered data partition, and N is an integer greater than 1; A target index tree corresponding to the target field is constructed according to the N index trees, wherein the target index tree belongs to the database index.
13. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the creation method according to any one of claims 1 to 11 are implemented.
14. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the creation method according to any one of claims 1 to 11 are implemented.
15. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the creation method described in any one of claims 1 to 11 are implemented.
