Lightweight scanning method of database storage engine and electronic device

Abstract

The present invention provides a lightweight scanning method and electronic device for a database storage engine, which can be applied to the fields of computer technology and database technology. The lightweight scanning method includes: obtaining data distribution information of a target data table to be scanned; determining a scanning strategy based on the data distribution information and the current resource usage information of the database storage engine; when the scanning strategy is determined to be a lightweight scanning strategy, determining the number of scanning threads for parallel scanning and the scanning tasks of each scanning thread based on the data distribution information; calling scanning threads corresponding to the number of scanning threads to perform their respective scanning tasks, each scanning thread performs a scanning task to batch scan and obtain multiple data pages corresponding to it, and fills the corresponding multiple data pages into the private cache of the scanning thread. Therefore, the present invention can at least solve the technical problems of low data table scanning efficiency and poor system performance, and achieve the technical effect of improving scanning efficiency and system performance.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, specifically to the field of database technology, and more specifically to a lightweight scanning method and electronic device for a database storage engine. Background Art

[0002] In the field of modern computer technology, with the acceleration of the information process, the amount of data is growing exponentially. How to efficiently store, manage and query massive data has become an important issue. As the core technology of data management, relational database management system (RDBMS) is widely used in various information systems. However, traditional database storage and query methods often encounter performance bottlenecks when facing high-concurrency, large-scale data query processing. With the continuous expansion of data scale and the increase in concurrent query requirements, it is particularly important to further optimize the storage and query efficiency of the database, especially the efficiency of scanning operations.

[0003] In the big data processing scenario, scanning operation is one of the most common and critical operations in the database query process. Data is stored in the database system in pages as the smallest unit. Traditional scanning technology usually loads data pages from disk into the global cache pool and then processes them page by page and row by row. In high-concurrency scenarios, querying a large number of tables will not only monopolize the global cache pool, but also compete with other table scans for the computer input (Input) / output (Output) system, that is, the I / O system, resulting in a large number of random I / O reads and writes, affecting query efficiency.

[0004] Therefore, how to improve the efficiency of large-scale table scanning operations while ensuring data integrity and accuracy has become an urgent problem to be solved. Summary of the invention

[0005] In view of this, the present invention provides a lightweight scanning method and electronic device for a database storage engine.

[0006] One aspect of the present invention provides a lightweight scanning method for a database storage engine, comprising: obtaining data distribution information of a target data table to be scanned; determining a scanning strategy based on the data distribution information and current resource usage information of the database storage engine; when the scanning strategy is determined to be a lightweight scanning strategy, determining the number of scanning threads for parallel scanning and the scanning tasks of each scanning thread based on the data distribution information; calling at least one scanning thread corresponding to the number of scanning threads to execute respective scanning tasks to scan and obtain the target data table from a database, wherein the database stores the target data table through multiple data pages, each scanning thread obtains multiple data pages corresponding to it by batch scanning from the database by executing the scanning task, and fills the multiple data pages corresponding to each scanning thread into the private cache of the scanning thread.

[0007] Optionally, each scanning thread is associated with a private cache, and the capacity of the private cache limits the number of data pages scanned by the scanning thread from the database; wherein the capacity of the private cache changes dynamically, and the capacity of the private cache is related to the scanning task and / or current resource usage information.

[0008] Optionally, the lightweight scanning method also includes: for each scanning thread, before the scanning thread starts executing the scanning task, determining the target capacity of the private cache according to the scanning task, and modifying the capacity of the private cache from the initial capacity to the target capacity; and after the scanning thread completes the scanning task, restoring the capacity of the private cache from the target capacity to the initial capacity.

[0009] Optionally, the lightweight scanning method also includes: for each scanning thread, when the resource usage information meets the predetermined conditions, increasing the initial capacity of the private cache to the target capacity according to the preset capacity step; when the resource usage information does not meet the predetermined conditions, restoring the capacity of the private cache from the target capacity to the initial capacity.

[0010] Optionally, a scanning strategy is determined based on the data distribution information and the current resource usage information of the database storage engine, including: determining a data distribution calculation result based on the data distribution information; taking a weighted sum of the data distribution calculation result and the resource usage information to obtain a comprehensive score; and when the comprehensive score is greater than a predetermined threshold, determining that the scanning strategy is a lightweight scanning strategy.

[0011] Optionally, the data distribution information includes at least one of the following: the number of rows of the target data table, the number of data pages storing the target data table, and the physical address; based on the data distribution information, determining the data distribution calculation result includes at least one of the following: determining the amount of data in the target data table based on the number of rows and / or the number of pages; determining the distribution value of a continuous segment based on the physical address, wherein a continuous segment represents multiple data pages connected by a physical address, and the distribution value is determined based on the capacity of the continuous segment and / or the number of continuous segments whose capacity is greater than a predetermined capacity.

[0012] Optionally, the number of scanning threads for parallel scanning and the scanning tasks of each scanning thread are determined based on the data distribution information, including: determining the number of scanning threads based on the amount of data; determining at least one scanning thread corresponding to the number of scanning threads in the created scanning threads; determining the scanning tasks of each scanning thread according to the physical address, and binding each scanning task to each scanning thread.

[0013] Optionally, determining the scanning task of each scanning thread according to the physical address includes: dividing a plurality of data pages in the same continuous segment into the scanning tasks of the same scanning thread according to the physical address to obtain the scanning task of each scanning thread.

[0014] Optionally, the lightweight scanning method further includes: when the number of created scanning threads is less than the number of scanning threads, creating at least one scanning thread to determine at least one scanning thread corresponding to the number of scanning threads.

[0015] Another aspect of the present invention provides a lightweight scanning device for a database storage engine, the device comprising: an acquisition module, used to acquire data distribution information of a target data table to be scanned; a first determination module, used to determine a scanning strategy based on the data distribution information and current resource usage information of the database storage engine; a second determination module, used to determine the number of scanning threads for parallel scanning and the scanning tasks of each scanning thread based on the data distribution information when the scanning strategy is determined to be a lightweight scanning strategy; a calling module, used to call at least one scanning thread corresponding to the number of scanning threads to perform respective scanning tasks, so as to scan and obtain the target data table from the database, wherein the database stores the target data table through multiple data pages, and each scanning thread obtains multiple data pages corresponding to it by batch scanning from the database by executing the scanning task, and fills the multiple data pages corresponding to each scanning thread into the private cache of the scanning thread.

[0016] Another aspect of the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0017] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0018] Another aspect of the present invention provides a computer program product, the computer program product comprising computer executable instructions, and the instructions are used to implement the above method when being executed.

[0019] The present invention proposes a lightweight scanning method for a database storage engine, which can achieve at least the following technical effects in high-concurrency and large-data volume processing scenarios:

[0020] On the one hand, by calling at least one scanning thread each having a private cache, each scanning thread can scan the corresponding data page in batches, thereby reducing the number of I / O times and improving the scanning efficiency.

[0021] On the other hand, considering that in high-concurrency, large-data processing scenarios, traditional scanning operations often rely on the global cache pool to store data pages, which results in the monopoly of the global cache pool when querying large tables, seriously affecting scanning efficiency and system performance. The present invention pre-applies a certain amount of cache for each scanning thread as a private cache to store the data pages to be scanned, thereby avoiding the problem of monopolizing the global cache pool and large data tables competing with other tables for I / O, thereby improving scanning efficiency and system performance.

[0022] On the other hand, by determining the scanning strategy through data distribution information and current resource usage information, it is possible to make intelligent decisions on the scanning strategy based on the specific situation, ensuring that the optimal scanning strategy is adopted in the appropriate scenario, improving scanning efficiency and system performance, and meeting user needs in high-concurrency, large-data volume query processing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0024] Figure 1 An exemplary system architecture to which the lightweight scanning method of the database storage engine of the present invention can be applied is shown.

[0025] Figure 2 A flow chart of a lightweight scanning method of a database storage engine according to an embodiment of the present invention is shown.

[0026] Figure 3 An application scenario diagram of calling a scanning thread to implement lightweight scanning according to an embodiment of the present invention is shown.

[0027] Figure 4 A flow chart of a method for determining a scanning strategy according to an embodiment of the present invention is shown.

[0028] Figure 5 A block diagram of a lightweight scanning device for a database storage engine according to an embodiment of the present invention is shown.

[0029] Figure 6 A block diagram of an electronic device suitable for lightweight scanning of a database storage engine according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0030] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0031] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0032] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0033] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0034] Figure 1 An exemplary system architecture of a lightweight scanning method of a database storage engine of the present invention is shown. It should be noted that: Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present invention may be applied, to help those skilled in the art understand the technical content of the present invention, but do not mean that the embodiments of the present invention may not be used in other devices, systems, environments or scenarios.

[0035] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a database 101 and a database storage engine 102 , wherein the database storage engine 102 may also be referred to as a database system.

[0036] The network is a medium for providing a communication link between the database 101 and the database storage engine 102. The network may include various connection types, such as wired and / or wireless communication links, etc. The database storage engine 102 may interact with the database 101 through the network to read data.

[0037] It should be noted that the lightweight scanning method of the database storage engine provided in the embodiment of the present invention can generally be executed by the database storage engine 102. Accordingly, the lightweight scanning device of the database storage engine provided in the embodiment of the present invention can generally be set in the database storage engine 102. The lightweight scanning method of the database storage engine provided in the embodiment of the present invention can also be executed by other servers or other server clusters, and the other servers or server clusters execute different methods from the database storage engine 102 and can communicate with the database 101 and / or the database storage engine 102. Accordingly, the lightweight scanning device of the database storage engine provided in the embodiment of the present invention can also be set in the above-mentioned other servers or other server clusters.

[0038] Figure 2 A flow chart of a lightweight scanning method of a database storage engine according to an embodiment of the present invention is shown.

[0039] like Figure 2 As shown, operations S210 to S240 are included.

[0040] In operation S210, data distribution information of a target data table to be scanned is obtained.

[0041] The database can store multiple data tables in the form of disks, which can also be called database tables. The data table to be scanned is the target data table. Data distribution information is used to characterize the distribution of data in the data table. For example, data distribution information may include: the number of rows, the number of pages, the page capacity (size), the data block distribution, etc.

[0042] The metadata management system (MDS) usually stores the metadata information of all tables in a centralized manner. The metadata information includes data distribution information and table structure information.

[0043] In the embodiment of the present invention, a connection needs to be established between the scanning thread and the metadata management system so as to obtain data distribution information from the metadata relationship system by means of a request.

[0044] In operation S220, a scanning strategy is determined according to the data distribution information and the current resource usage information of the database storage engine.

[0045] The database storage engine may also be referred to as a database system. The current resource usage information of the database storage engine may include current memory usage information and central processing unit (CPU) load information, such as memory usage rate and the number of CPU cores.

[0046] The scanning strategy includes a lightweight scanning strategy and a conventional scanning strategy. In one embodiment, whether the target data table is a large data volume table and the current processing capacity of the database storage engine can be determined based on the data distribution information and the resource usage information, so as to determine the lightweight scanning strategy when the target data table is a large data volume table and the current processing capacity of the database storage engine is strong.

[0047] In operation S230, when it is determined that the scanning strategy is a lightweight scanning strategy, the number of scanning threads for parallel scanning and the scanning task of each scanning thread are determined according to the data distribution information.

[0048] The number of scanning threads represents the number of scanning threads called in parallel so that the above-mentioned second threads can scan simultaneously. Each scanning thread can be bound to a scanning task to ensure that during the scanning process, the loading and processing of all data pages of the scanning thread are performed in the private cache of the current scanning thread.

[0049] It is understandable that after a scanning thread completes a scanning task, it can be unbound from the current scanning task and bound to the next scanning task, so that the scanning thread can load and process all data pages in the next scanning task in the private cache.

[0050] In one embodiment, all data pages of the target data table may be obtained by scanning through one scanning thread; or, all data pages may be obtained by scanning through multiple scanning threads.

[0051] In operation S240, at least one scanning thread corresponding to the number of scanning threads is called to perform respective scanning tasks to scan and obtain a target data table from a database, wherein the database stores the target data table through multiple data pages, and each scanning thread obtains multiple data pages corresponding to it by batch scanning from the database by executing the scanning task, and fills the multiple data pages corresponding to each scanning thread into the private cache of the scanning thread.

[0052] The target data table includes multiple rows. The data of each row can be stored in one data page or multiple data pages. The data of multiple rows can also be stored in one data page.

[0053] The scanning tasks of multiple scanning threads can be the same or different. For example, two scanning threads, such as scanning thread 1 and scanning thread 2, are called at the same time. The scanning task bound to scanning thread 1 is to read the 1st to 10th data pages and fill the 1st to 10th data pages into the private cache of scanning thread 1. The scanning task bound to scanning thread 2 is to read the 11th to 15th data pages and fill the 11th to 15th data pages into the private cache of scanning thread 2. That is, the two scanning threads can scan different numbers of data pages.

[0054] According to an embodiment of the present invention, each scanning thread does not read data pages one by one from the database, but batch reads multiple data pages corresponding to the scanning thread to improve scanning efficiency. For example, scanning thread 1 can batch read 16 data pages through one I / O, rather than reading 16 data pages through multiple I / Os.

[0055] The embodiment of the present invention proposes a lightweight scanning method for a database storage engine, which can achieve at least the following technical effects in a high-concurrency, large-data volume processing scenario:

[0056] On the one hand, by calling at least one scanning thread each having a private cache, each scanning thread can batch scan and obtain corresponding data pages, which can reduce the number of I / O times, reduce the overhead of I / O operations, and improve scanning efficiency.

[0057] On the other hand, considering that in high-concurrency, large-data processing scenarios, traditional scanning operations often rely on the global cache pool to store data pages, which results in the monopoly of the global cache pool when querying a large number of tables, seriously affecting scanning efficiency and system performance. The embodiments of the present invention can avoid the problem of monopolizing the global cache pool and large data tables competing for I / O with other tables by pre-applying a certain amount of cache for each scanning thread as a private cache to store the data pages to be scanned, thereby improving scanning efficiency and system performance.

[0058] On the other hand, by determining the scanning strategy through data distribution information and current resource usage information, it is possible to make intelligent decisions on the scanning strategy based on the specific situation, ensuring that the optimal scanning strategy is adopted in the appropriate scenario, improving scanning efficiency and system performance, and meeting user needs in high-concurrency, large-data volume query processing scenarios.

[0059] In one embodiment, the capacity of the private cache may be fixed, and it is determined based on empirical values ​​or system configuration parameters that the capacity can simultaneously meet most scanning tasks and will not cause resource pressure on other parts of the system.

[0060] In another embodiment, the capacity of the private cache may be dynamically changed.

[0061] According to an embodiment of the present invention, each scanning thread is associated with a private cache, and the capacity of the private cache limits the number of data pages scanned by the scanning thread from the database; wherein the capacity of the private cache changes dynamically, and the capacity of the private cache is related to the scanning task and / or current resource usage information.

[0062] In an embodiment of the present invention, a portion of cache may be applied for each scanning thread in the memory as a private cache of the scanning thread, and the scanning thread and the private cache may be associated in a binding manner.

[0063] The capacity of the private cache is the size of the private cache, which limits the number of data pages that the scanning thread scans in batches. For example, when the capacity of the private cache is 256KB and the size of each data page is 4KB, the scanning thread can only scan 64 data pages in batches.

[0064] In the embodiment of the present invention, the capacity of the private cache is dynamically changing and is not fixed. The scanning thread can be reused by binding multiple scanning tasks, but for each scanning task using the same scanning thread, the capacity of its private cache is dynamically changing. The capacity of the private cache is related to the scanning task and / or the current resource usage information.

[0065] For example, when scanning thread 1 is bound to scanning task 1, the capacity of the private cache limits scanning thread 1 to batch scan 64 data pages; when it is bound to scanning task 2, the capacity of the private cache limits scanning thread 1 to batch scan 128 data pages. Alternatively, when scanning thread 1 is bound to scanning task 1, if the current memory usage is high, the capacity of the private cache limits scanning thread 1 to batch scan 32 data pages; if the current memory usage is low, the capacity of the private cache limits scanning thread 1 to batch scan 64 data pages.

[0066] The embodiments of the present invention dynamically adjust the capacity of the private cache according to the scanning task and / or the current resource usage information of the database storage engine, so as to dynamically adjust the size of the scanning thread batch I / O, thereby improving the scanning efficiency and system performance.

[0067] Figure 3 An application scenario diagram of calling a scanning thread to implement lightweight scanning according to an embodiment of the present invention is shown.

[0068] like Figure 3 As shown, the data of rows 1 to 3 of the target data table in embodiment 300 are stored in multiple data pages, and the scanning thread 1 batch scans the multiple data pages, and fills the scanned multiple data pages into the private cache of the scanning thread 1. In the private cache of the scanning thread 1, the data pages can be processed.

[0069] Similarly, the data of the sixth row of the target data table is stored in multiple data pages. The scanning thread N batch scans the multiple data pages and fills the scanned data pages into the private cache of the scanning thread N for data processing.

[0070] In order to more clearly understand the operation of the scanning thread under the lightweight scanning strategy, the following will take a scanning thread as an example for description.

[0071] Step 1: Batch I / O reads data pages. Under the lightweight scanning strategy, the scanning thread uses batch I / O to read multiple data pages from the disk. For example, scanning thread 1 uses batch I / O to read 512 data pages at a time, reducing the number of disk I / Os and random I / Os, and improving reading efficiency.

[0072] Step 2: Fill the private cache. Fill the batch-read data pages into the private cache of the scanning thread for subsequent filtering and consumption by the SQL (Structured Query Language) layer. Ensure that the management and update operations of the cache pool are carried out efficiently to avoid cache pool overflow or data loss.

[0073] Step 3: SQL layer filtering and consumption. The data pages in the private cache are filtered and consumed by the SQL layer. For example, the SQL layer processes the data pages according to the query conditions and filters out the data that meets the conditions.

[0074] Step 4: Dynamic adjustment: Dynamically adjust the capacity of the private cache according to the progress of the scanning task and the current resource usage information to adjust the size of the batch I / O to ensure the efficient scanning operation.

[0075] Step 5: The scanning task executes the main loop. During the scanning task execution, the main loop continues to perform batch I / O reading, filling the private cache, and SQL layer filtering and consumption operations until the scanning task is completed. For example, the scanning task bound to the scanning thread needs to scan 64 data pages, but the capacity of the scanning thread's private cache is only 32 data pages. Therefore, the scanning thread can execute the scanning task through 64 / 32=2 loops.

[0076] Step 6: Task completion and resource release. When the scanning task is completed, the scanning thread is dispatched by the database system, so that when a new scanning task is started, the original scanning thread can be directly reused. Here, when the next scanning task is started, the newly scanned data page overwrites the already filled data page in the private cache by overwriting, without actively releasing the data in the private cache of each scanning thread.

[0077] Through the above steps 1 to 6, the scanning execution mechanism of the present invention can efficiently fill the data pages in the large data table into the private cache of the scanning thread using batch I / O operations, and provide filtering and consumption for the SQL layer. This mechanism ensures that the scanning operation can be carried out efficiently in high-concurrency, large-data query processing scenarios, improves the overall performance and response speed of the system, and meets the needs of users.

[0078] According to an embodiment of the present invention, the lightweight scanning method also includes: for each scanning thread, before the scanning thread starts to execute the scanning task, determining the target capacity of the private cache according to the scanning task, and modifying the capacity of the private cache from the initial capacity to the target capacity; and after the scanning thread completes the scanning task, restoring the capacity of the private cache from the target capacity to the initial capacity.

[0079] The initial capacity of the private cache may be determined based on empirical values ​​or system configuration parameters to avoid resource pressure on other parts of the system. For example, the initial capacity may be the size of 521 data pages.

[0080] In order to ensure that the scanning thread can meet the current scanning task, the target capacity of the private cache can be determined according to the current scanning task. For example, when the number of data pages required to be scanned by the scanning task is within a first number range, the capacity corresponding to the first number range is determined as the target capacity; when the number of data pages required to be scanned by the scanning task is within a second number range, the capacity corresponding to the second number range is determined as the target capacity.

[0081] It should be noted that the target capacity corresponds to the current scanning task bound to the scanning thread. After the scanning task is completed, the capacity of the private cache is restored from the target capacity to the initial capacity to prevent the private cache from occupying too much memory.

[0082] According to an embodiment of the present invention, the lightweight scanning method also includes: for each scanning thread, when the resource usage information meets the predetermined conditions, the initial capacity of the private cache is increased to the target capacity according to the preset capacity step; when the resource usage information does not meet the predetermined conditions, the capacity of the private cache is restored from the target capacity to the initial capacity.

[0083] Similarly, the target capacity corresponds to the current scanning task bound to the scanning thread. After the scanning task is completed, the capacity of the private cache is restored from the target capacity to the initial capacity to prevent the private cache from occupying too much memory.

[0084] In one embodiment, the resource usage information may be a memory usage rate. When the memory usage rate is less than a first memory threshold, at least one preset capacity step may be added on the basis of the initial capacity to obtain a target capacity. Alternatively, when the memory usage rate is greater than the first memory threshold and less than a second memory threshold, only one preset capacity step is added. Alternatively, when the memory usage rate is greater than the second memory threshold, the target capacity is the initial capacity.

[0085] In another embodiment, the resource usage information may be the number of CPU cores. When the number of CPU cores is greater than 8, at least one preset capacity step is added to the initial capacity to obtain the target capacity.

[0086] It is understandable that the above resource usage information is only an illustrative example, and the present invention may also use other resource usage information to dynamically adjust the capacity of the private cache.

[0087] In yet another embodiment, the target capacity may be determined based on both the scanning task and the resource usage information, which will not be described in detail herein.

[0088] Figure 4 A flow chart of a method for determining a scanning strategy according to an embodiment of the present invention is shown.

[0089] like Figure 4 As shown, operations S421 to S423 are included.

[0090] In operation S421, a data distribution calculation result is determined according to the data distribution information.

[0091] In operation S422, a weighted sum is performed on the data distribution calculation result and the resource usage information to obtain a comprehensive score.

[0092] In operation S423, when the comprehensive score is greater than a predetermined threshold, the scanning strategy is determined to be a lightweight scanning strategy.

[0093] In this embodiment, the data distribution calculation result may be the data volume of the target data table. If the data volume is large, a lightweight scanning strategy may be initiated.

[0094] Since lightweight scanning usually requires additional cache and computing resources, the comprehensive score not only includes the data distribution calculation results, but also needs to evaluate whether there are sufficient resources to support lightweight scanning based on the current resource usage information.

[0095] The weights corresponding to the data distribution calculation results and the resource usage information may be predetermined, and the comprehensive score is calculated by weighted summation. When the comprehensive score is greater than a predetermined threshold, the scanning strategy is determined to be a lightweight scanning strategy.

[0096] In an illustrative embodiment, the data distribution information includes at least one of the following: the number of rows of the target data table, the number of data pages storing the target data table, and the physical address; based on the data distribution information, determining the data distribution calculation result includes at least one of the following: determining the amount of data in the target data table based on the number of rows and / or the number of pages; determining the distribution value of a continuous segment based on the physical address, wherein the continuous segment represents multiple data pages connected by the physical address, and the distribution value is determined based on the capacity of the continuous segment and / or the number of continuous segments whose capacity is greater than a predetermined capacity.

[0097] For example, the result of data distribution calculation can be the data volume. The number of rows or pages can be used as the data volume of the target data table; or, the data volume of the target data table can be determined based on the number of rows and pages, such as data volume = (R / Tr) + (P / Tp), where R is the number of rows, P is the number of pages, Tr and Tp are the row number threshold and page number threshold, respectively, the row number threshold can be 1,000,000 rows, and the page number threshold can be 100,000 pages. R / Tr and R / Tp are used to evaluate the data volume. If both the number of rows and the number of pages exceed the threshold, the score will be greater than 1.

[0098] For another example, the result of data distribution calculation can be a distribution value, which can be the maximum capacity of a continuous segment or the number of continuous segments whose capacity (size) is greater than a predetermined capacity, and the predetermined capacity can be 2MB. Alternatively, the distribution value can be determined by combining the capacity and number of continuous segments. Distribution value = (C / Tc) + (N / Tn), where C is the capacity of the continuous segment, N is the number of continuous segments, and Tc and Tn are the capacity threshold and the number threshold, respectively, such as 2MB and 10 segments. C / Tc + N / Tn is used to evaluate the data disk distribution, and the score is greater than 1 when the capacity and number of continuous segments exceed the threshold.

[0099] For another example, the data distribution calculation result may integrate the data volume and the number of continuous segments, such as data distribution calculation result=(R / Tr)+(P / Tp)+ (C / Tc)+(N / Tn).

[0100] In an exemplary embodiment, a comprehensive score is calculated by combining data volume, data disk distribution and resource usage information. For example, when the resource usage information is system resource availability M (between 0 and 1, the closer to 1, the more sufficient the resource is, and the memory usage and CPU are combined), the comprehensive score is:

[0101] S=Wd×[(R / Tr)+(P / Tp)]+ Wm×[(C / Tc)+(N / Tn)]+ Ws×M

[0102] Among them, Wd, Wm, and Ws represent the weights of data volume, data disk distribution, and resource usage information respectively. The sum of the three weights is 1, that is, Wd+Wm+Ws=1. The meanings of other symbols are as above and will not be repeated here.

[0103] Based on the comprehensive score, if the data volume is large and the system resources are sufficient, when S exceeds a predetermined threshold, such as 20, it can be determined to start a lightweight scanning strategy.

[0104] The lightweight scanning startup judgment mechanism of the present invention can intelligently decide whether to start the lightweight scanning strategy based on data distribution information and resource usage information, ensure the use of the optimal scanning strategy in appropriate scenarios, improve the efficiency of scanning operations and system performance, and meet the needs of users in high concurrency and large data volume query processing scenarios.

[0105] According to an embodiment of the present invention, for operation S230, the number of scanning threads for parallel scanning and the scanning tasks of each scanning thread are determined according to the data distribution information, including: determining the number of scanning threads according to the amount of data; determining at least one scanning thread corresponding to the number of scanning threads in the created scanning threads; determining the scanning tasks of each scanning thread according to the physical address, and binding each scanning task to each scanning thread.

[0106] After deciding to start the lightweight scan strategy, you need to decide whether to enable parallel scanning based on the amount of data in the table and allocate scanning tasks appropriately to make full use of system resources.

[0107] Specifically, the number of scanning threads can be determined according to the amount of data, such as according to a predefined relationship between the amount of data and the number of scanning threads. As described above, the amount of data can be determined according to the number of rows and / or pages. In another embodiment, the number of scanning threads can also be determined based on both the amount of data and the resource usage information.

[0108] For example, when the amount of data is too small, parallel scanning is not enabled, and only one scanning thread is used to perform the scanning task, that is, the number of scanning threads is 1. When the amount of data is large, parallel scanning is enabled, and the number of scanning threads is determined according to the amount of data. Alternatively, it is also possible to determine whether to start parallel scanning according to the amount of data, and to determine the number of scanning threads according to resource usage information. Alternatively, it is also possible to determine whether to start parallel scanning according to the amount of data, and to determine the number of scanning threads according to the amount of data and resource usage information.

[0109] After determining the number of scanning threads, at least one scanning thread to be called can be preferentially determined from the created scanning threads to reuse idle created scanning threads, thereby preventing the scanning threads from applying for too many private caches and affecting system performance.

[0110] According to an embodiment of the present invention, determining the scanning task of each scanning thread according to the physical address includes: dividing multiple data pages in the same continuous segment into the scanning tasks of the same scanning thread according to the physical address to obtain the scanning task of each scanning thread.

[0111] When reading data pages from the disk, the time required to read multiple data pages with relatively scattered physical addresses is longer than the time required to read multiple data pages with relatively concentrated physical addresses. Therefore, when determining the scanning task corresponding to each scanning thread, multiple data pages in the same continuous segment can be divided into the scanning tasks of the same scanning thread according to the page numbers of the data pages, so as to reasonably allocate the scanning tasks and improve the efficiency of reading data pages from the disk.

[0112] For example, when the target data table is stored in the 1st, 2nd, 3rd, 5th, 7th, 8th, 9th, 10th, and 11th data pages respectively, the 1st, 2nd, and 3rd data pages can be used as continuous segment 1, the 5th data page can be used as continuous segment 2, and the 7th, 8th, 9th, 10th, and 11th data pages can be used as continuous segment 3, and assigned to three scanning tasks respectively.

[0113] According to an embodiment of the present invention, when the number of created scanning threads is less than the number of scanning threads, at least one scanning thread is created to determine at least one scanning thread corresponding to the number of scanning threads.

[0114] If the number of created scan threads is less than the number of scan threads, at least one scan thread can be recreated to meet the parallel call requirements. For example, if the number of scan threads is 8 and the number of created scan threads is 6, two more scan threads can be created and private caches can be applied for them. At this time, the lightweight scan strategy can reuse the 6 created scan threads.

[0115] For ease of understanding, the following takes a target data table named LargeTable as an example. The table contains 100 million rows of data, each row of data is 512 bytes, each data page is 4K, and the total data volume is about 51.2GB. The target data table is a large data volume table.

[0116] Step 1: Private cache application. For a newly created scanning thread, when the scanning thread starts, a certain amount of cache is applied in advance as a private cache to store the data pages to be scanned. Assuming that the size of each private cache block is 4MB, the scanning thread applies for 128 such cache blocks, a total of 512MB of private cache.

[0117] It should be noted that when reusing the created scanning thread, there is no need to execute step 1 to reapply for a private cache, and the private cache of the created scanning thread can be reused.

[0118] Step 2: Metadata query. Query the data distribution information of the LargeTable table from the metadata management system, such as the number of rows, the number of pages, the physical address of the data page, etc., to provide a basis for the subsequent scanning strategy formulation. The query results are as follows: The number of rows row_count = 100000000, the number of pages page_count = 12500000, the data page size average_page_size = 4 * 1024, that is, 4KB, and the data volume total_size = 51.2 * 1024 * 1024 * 1024, that is, 51.2GB.

[0119] Step 3: Lightweight scanning strategy judgment mechanism. The scanning thread determines whether to start the lightweight scanning strategy based on the obtained table metadata information. Calculation is based on the above formula 1:

[0120] (R / Tr) = 100000000 / 1000000=100, (P / Tp) = 12500000 / 100000=125, data volume score Sd = Wd×[(R / Tr)+(P / Tp)]=0.4×(100+125)=90.

[0121] (C / Tc)=3 / 2=1.5, (N / Tn)=20 / 10=2, the score of data disk distribution Sm= Wm×[(C / Tc)+(N / Tn)]=0.3×(1.5+2)=0.9.

[0122] Score of resource usage information: Ss= Ws×M=0.3×0.8=0.24.

[0123] The comprehensive score S=Sd+Sm+ Ss=90+0.9+0.24=91.14.

[0124] According to calculation, the comprehensive score greatly exceeds the predetermined threshold of 20, and therefore, the scanning strategy is determined to be a lightweight scanning strategy.

[0125] Step 4: Starting the lightweight scanning strategy and allocating tasks: When it is determined that the lightweight scanning strategy needs to be started, parallel scanning is started according to the data volume, and 8 parallel scanning threads are started according to the resource usage information.

[0126] Step 5: Scan execution. The scanning thread continuously fills the data pages of the LargeTable table stored in the disk into the private cache of the scanning thread through batch I / O, and provides filtering and consumption for the SQL layer, for example, reading 64 data pages (256KB) in batches each time.

[0127] The private cache application in step 1 can avoid the monopoly problem of the global cache pool. The metadata query in step 2 can obtain data distribution information and provide a basis for the formulation of scanning strategies. The lightweight scanning strategy judgment mechanism in step 3 can decide whether to start the lightweight scanning strategy based on data distribution information and resource usage information. The startup and task allocation of the lightweight scanning strategy in step 4 can reasonably allocate tasks and enable parallel scanning when starting the lightweight scanning strategy. The scan execution in step 5 can batch I / O read data pages to the private cache for filtering and consumption by the SQL layer, thereby improving scanning efficiency. Through these steps, the problems of large data tables monopolizing the cache pool and large data tables competing for IO with other tables can be effectively solved, thereby improving scanning efficiency.

[0128] Based on this, the lightweight scanning method of the database storage engine proposed in the present invention improves the performance of the database in high-concurrency and large-scale data table processing scenarios by improving the scanning mechanism and optimizing the data structure. This method can not only effectively reduce the time overhead of the scanning operation, but also significantly reduce the consumption of system resources, thereby improving the overall efficiency and response speed of the database and meeting the needs of users in large data table processing.

[0129] Figure 5 The block diagram of the lightweight scanning device for database storage engine according to the embodiment of the present invention is shown. The lightweight scanning device 500 for database storage engine includes an acquisition module 510 , a first determination module 520 , a second determination module 530 and a calling module 540 .

[0130] The acquisition module 510 is used to acquire data distribution information of the target data table to be scanned.

[0131] The first determination module 520 is used to determine a scanning strategy according to data distribution information and current resource usage information of the database storage engine.

[0132] The second determination module 530 is used to determine the number of scanning threads for parallel scanning and the scanning task of each scanning thread according to the data distribution information when the scanning strategy is determined to be a lightweight scanning strategy.

[0133] The calling module 540 is used to call at least one scanning thread corresponding to the number of scanning threads to perform respective scanning tasks to scan and obtain a target data table from a database, wherein the database stores the target data table through multiple data pages, and each scanning thread obtains multiple data pages corresponding to it from the database in batches by executing the scanning task, and fills the multiple data pages corresponding to each scanning thread into the private cache of the scanning thread.

[0134] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a proper combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be executed.

[0135] For example, any multiple of the acquisition module 510, the first determination module 520, the second determination module 530, and the call module 540 can be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present invention, at least one of the acquisition module 510, the first determination module 520, the second determination module 530, and the call module 540 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 510, the first determination module 520, the second determination module 530 and the calling module 540 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0136] It should be noted that the device part in the embodiment of the present invention corresponds to the method part in the embodiment of the present invention. The description of the device part specifically refers to the method part, which will not be repeated here.

[0137] Figure 6A block diagram of an electronic device suitable for lightweight scanning of a database storage engine according to an embodiment of the present invention is shown. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0138] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0139] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0140] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0141] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0142] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0143] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device.

[0144] For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or the RAM 603 described above and / or one or more memories other than the ROM 602 and the RAM 603 .

[0145] An embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present invention.

[0146] When the computer program is executed by the processor 601, the above functions defined in the system / device of the embodiment of the present invention are executed. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0147] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0148] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways. All these combinations and / or combinations fall within the scope of the present invention.

[0150] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A lightweight scanning method for a database storage engine, characterized in that: The lightweight scanning method comprises: Obtain data distribution information of the target data table to be scanned; Determine a data distribution calculation result according to the data distribution information; Performing a weighted summation on the data distribution calculation result and the current resource usage information of the database storage engine to obtain a comprehensive score; When the comprehensive score is greater than a predetermined threshold, determining the scanning strategy to be a lightweight scanning strategy; When it is determined that the scanning strategy is a lightweight scanning strategy, determining the number of scanning threads for parallel scanning and the scanning task of each scanning thread according to the data distribution information; Call at least one of the scanning threads corresponding to the number of scanning threads to execute the respective scanning tasks to scan and obtain the target data table from the database, wherein the database stores the target data table through multiple data pages, and each of the scanning threads obtains the multiple data pages corresponding to it by batch scanning from the database by executing the scanning tasks, and fills the multiple data pages corresponding to each scanning thread into the private cache of the scanning thread.

2. The lightweight scanning method according to claim 1, characterized in that: Each of the scanning threads is associated with a private cache, and the capacity of the private cache limits the number of data pages scanned by the scanning thread from the database; The capacity of the private cache changes dynamically, and the capacity of the private cache is related to the scanning task and / or the current resource usage information.

3. The lightweight scanning method according to claim 2, characterized in that: The lightweight scanning method further comprises: for each of the scanning threads, Before the scanning thread starts to execute the scanning task, determining a target capacity of the private cache according to the scanning task, and modifying the capacity of the private cache from an initial capacity to the target capacity; and After the scanning thread finishes the scanning task, the capacity of the private cache is restored from the target capacity to the initial capacity.

4. The lightweight scanning method according to claim 2, characterized in that: The lightweight scanning method further comprises: for each of the scanning threads, When the resource usage information satisfies a predetermined condition, increasing the initial capacity of the private cache to a target capacity according to a preset capacity step; When the resource usage information does not satisfy the predetermined condition, the capacity of the private cache is restored from the target capacity to the initial capacity.

5. The lightweight scanning method according to claim 1, characterized in that: The data distribution information includes at least one of the following: the number of rows of the target data table, the number of data pages storing the target data table, and the physical address; Determining the data distribution calculation result according to the data distribution information includes at least one of the following: Determine the data volume of the target data table according to the number of rows and / or pages; According to the physical address, a distribution value of a continuous segment is determined, wherein the continuous segment represents a plurality of data pages connected to the physical address, and the distribution value is determined according to the capacity of the continuous segment and / or the number of continuous segments whose capacity is greater than a predetermined capacity.

6. The lightweight scanning method according to claim 5, characterized in that: The step of determining the number of scanning threads for parallel scanning and the scanning task of each scanning thread according to the data distribution information includes: Determining the number of scanning threads according to the amount of data; Determining at least one of the scan threads corresponding to the number of the scan threads among the created scan threads; According to the physical address, a scanning task of each scanning thread is determined, and each scanning task is bound to each scanning thread.

7. The lightweight scanning method according to claim 6, characterized in that: Determining the scanning task of each scanning thread according to the physical address includes: According to the physical address, a plurality of data pages in the same continuous segment are divided into scanning tasks of the same scanning thread to obtain scanning tasks of each scanning thread.

8. The lightweight scanning method according to claim 6, characterized in that: The lightweight scanning method further includes: In a case where the number of the created scanning threads is less than the number of scanning threads, at least one of the scanning threads is created to determine at least one of the scanning threads corresponding to the number of scanning threads.

9. An electronic device, comprising: one or more processors; a memory for storing one or more programs, It is characterized in that when the one or more programs are executed by the one or more processors, the one or more processors implement the lightweight scanning method described in any one of claims 1 to 8.

Images