Self-adaptive database processing method and device

Abstract

Embodiments of the invention provide an adaptive database processing method and apparatus. The method comprises the steps of obtaining NUMA node information and historical access information of data in a database; wherein the NUMA node information comprises the number of NUMA nodes; according to the historical access information, determining historical access frequency and historical data relevance of the data; determining a data layout strategy according to the historical access frequency and the historical data relevance; dividing data in a database into a plurality of data blocks according to the data layout strategy; wherein the number of the divided data blocks is the same as the number of the NUMA nodes; and distributing the data blocks to the NUMA nodes in a one-to-one correspondence manner. According to the method, the processor architecture and the NUMA characteristics are combined, the layout of the data in the database is optimized, the proper lock strategy is configured based on the optimized data layout, and the database performance can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of database technology, and in particular to an adaptive database processing method and device. Background Art

[0002] In the Advanced RISC Machine (ARM) processor architecture that supports the Non-Uniform Memory Access (NUMA) feature, in the face of high-concurrency scenarios, it is necessary to solve the optimization configuration and locking mechanism of the database. In the related technology, the design and optimization of the database in the X86 processor architecture is relatively mature, but there is no optimization design for the ARM processor architecture that supports the NUMA feature. The traditional database configuration and locking mechanism cannot fully utilize the advantages of the ARM processor architecture and NUMA features, resulting in low database performance and extended response time. Summary of the invention

[0003] In view of this, an object of the embodiments of the present application is to provide an adaptive database processing method and device.

[0004] Based on the above purpose, the embodiment of the present application provides an adaptive database processing method, including:

[0005] Obtaining NUMA node information and historical access information of data in a database; wherein the NUMA node information includes the number of NUMA nodes;

[0006] Determine the historical access frequency of the data and the historical data relevance according to the historical access information;

[0007] Determining a data layout strategy according to the historical access frequency and the historical data relevance;

[0008] According to the data layout strategy, the data in the database is divided into a plurality of data blocks; wherein the number of the divided data blocks is the same as the number of the NUMA nodes;

[0009] Allocate each data block to each NUMA node one by one.

[0010] Optionally, determining a data layout strategy according to the historical access frequency and the historical data relevance includes:

[0011] dividing the data whose historical access frequency reaches a preset access threshold into the same data block; and / or,

[0012] The historical data whose relevance reaches a preset relevance threshold are divided into the same data block.

[0013] Optionally, the method further includes:

[0014] When a preset layout adjustment cycle is reached or a preset layout adjustment condition is met, current access information of data in the database and load information of each NUMA node are obtained;

[0015] Determine the current access frequency of the data and the current data relevance according to the current access information;

[0016] Determining an updated data layout strategy according to the current access frequency and the current data relevance;

[0017] Determining the data blocks to be optimized according to the updated data layout strategy and the current data layout strategy;

[0018] Determine the target NUMA node according to the current access frequency of the data block to be optimized and the load information of each NUMA node;

[0019] The data block to be optimized is migrated from the NUMA node where the data block to be optimized is located to the destination NUMA node.

[0020] Optionally, determining the data block to be optimized according to the updated data layout strategy and the current data layout strategy includes:

[0021] Re-dividing the data in the database into a plurality of data blocks according to the updated data layout strategy;

[0022] Compare the re-divided data blocks with the currently divided data blocks to determine the changed data blocks;

[0023] Determining a target NUMA node according to the current access frequency of the data block to be optimized and the load information of each NUMA node includes:

[0024] According to the load information of each NUMA node, a NUMA node that is adapted to the current access frequency of the data block to be optimized is selected from each NUMA node as the target NUMA node.

[0025] Optionally, the method for determining the layout adjustment condition includes:

[0026] Obtain access information of data in the database and load information of each NUMA node;

[0027] According to the access information, when it is determined that there is data accessed across NUMA nodes, and the amount of data accessed across NUMA nodes reaches a preset access amount threshold, determining that the layout adjustment condition is met; and / or,

[0028] When it is determined according to the load information of each NUMA node that the load of each NUMA node reaches an unbalanced state, it is determined that the layout adjustment condition is met.

[0029] Optionally, after allocating each data block to each NUMA node one by one, the method further includes:

[0030] According to the allocation relationship between each data block and each NUMA node, a corresponding relationship table between the data blocks and the NUMA nodes is established.

[0031] Optionally, the method further includes:

[0032] When a preset lock adjustment period is reached or a preset lock adjustment condition is met, the system performance parameters and access mode information of each data block are obtained;

[0033] Preprocessing the performance parameters and access mode information to obtain preprocessed performance parameters and access mode information;

[0034] Inputting the preprocessed performance parameters and access mode information into a preset lock granularity prediction model, and the lock granularity prediction model outputs a predicted lock granularity level;

[0035] According to the lock granularity level, the lock granularity of each data block is adjusted.

[0036] Optionally, the lock granularity level output by the lock granularity prediction model is a lock granularity level for each data block; and adjusting the lock granularity of each data block according to the lock granularity level includes:

[0037] According to the predicted lock granularity level for each data block, determining data blocks whose current lock granularity level is different from the predicted lock granularity level;

[0038] Adjust the lock granularity level of the data block to the predicted lock granularity level.

[0039] Optionally, the method for determining the lock adjustment condition includes:

[0040] Get the system's operating status parameters;

[0041] Determining whether there is a performance bottleneck and / or concurrency conflict based on the operating status parameters;

[0042] If present, the lock adjustment condition is met.

[0043] The embodiment of the present application also provides an adaptive database processing device, including:

[0044] An acquisition module, used to acquire NUMA node information and historical access information of data in a database; wherein the NUMA node information includes the number of NUMA nodes;

[0045] An analysis module, used to determine the historical access frequency of data and the historical data relevance according to the historical access information;

[0046] A layout module, used to determine a data layout strategy according to the historical access frequency and the historical data relevance;

[0047] A partitioning module, used to partition the data in the database into a plurality of data blocks according to the data layout strategy; wherein the number of the partitioned data blocks is the same as the number of the NUMA nodes;

[0048] The allocation module is used to allocate each data block to each NUMA node one by one.

[0049] From the above, it can be seen that the adaptive database processing method and device provided by the embodiment of the present application determines the data layout strategy according to the historical access frequency and historical data relevance of the data in the database, divides the data in the database into multiple data blocks according to the data layout strategy, and distributes each data block one by one to each NUMA node. The present application combines the processor architecture and NUMA characteristics to optimize the layout of the data in the database, and configures a suitable lock strategy based on the optimized data layout. During the operation of the database, the data layout strategy and lock strategy are dynamically adjusted by comprehensively considering the system performance, data access information, the load status of the NUMA node, etc., so as to improve the database performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 A schematic diagram of a method flow of an embodiment of the present application;

[0052] Figure 2 A schematic diagram of NUMA partitioning according to an embodiment of the present application;

[0053] Figure 3 This is a schematic diagram of the adaptive lock strategy flow in an embodiment of the present application;

[0054] Figure 4 This is a block diagram of the device structure of an embodiment of the present application;

[0055] Figure 5 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0057] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connecting" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to represent relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0058] In the related art, in the ARM processor architecture system supporting NUMA features, there is a server based on ARM architecture, the server is configured with multiple processors and multiple memories supporting NUMA features, or in other words, the server includes multiple NUMA nodes, each NUMA node includes a local processor and a local memory, and the operating system on the server supports NUMA features and can provide NUMA node related information. In this system, database processing is performed according to the traditional database management method, which cannot give full play to the advantages of NUMA features and ARM processor architecture, and the database performance is not high.

[0059] In view of this, an embodiment of the present application provides an adaptive database processing method, which divides the data in the database into multiple data blocks and allocates the data blocks to NUMA nodes accordingly. During operation, the database's data layout strategy and locking strategy are dynamically adjusted based on various indicators such as database access information, NUMA node load information, and system operation status, thereby improving database performance.

[0060] like Figure 1 As shown, the embodiment of the present application provides an adaptive database processing method, including:

[0061] S101: Acquire NUMA node information and historical access information of data in a database; wherein the NUMA node information includes the number of NUMA nodes;

[0062] In this embodiment, query functions, system files, etc. can be used to query and obtain relevant information about NUMA nodes supported by the system. For example, by calling the numa_available() function, it is queried whether NUMA is available, and by calling the numa_max_node() function, the number of NUMA nodes in the system is queried. After identifying the NUMA node, the configuration information of each NUMA node is collected by querying the system file. The configuration information includes but is not limited to processor parameters (identification number, model, processing speed, etc.), memory size, cache hierarchy (including the size and configuration of L1, L2, and L3 caches), physical address range, etc. Optionally, the relevant configuration information of the NUMA node can be obtained by querying the system file ` / sys / devices / system / node / nodeX / meminfo`, etc.

[0063] In some methods, after obtaining the configuration information of the NUMA node, a node check can be performed according to the configuration information, and the available NUMA nodes can be determined according to the check result. Among them, the check method includes determining whether the processor identification numbers of each NUMA node overlap, determining whether the memory size of each NUMA node is within a reasonable range, etc., and taking the NUMA nodes whose processor identification numbers do not overlap and whose memory is within a reasonable range as available nodes. In this embodiment, considering that there may be software and hardware failures or resource allocation errors (for example, multiple processors are artificially assigned to the same NUMA node, or a processor is assigned to multiple nodes at the same time), overlapping processor identification numbers will appear, resulting in the inability to correctly optimize data access and storage; there may also be limitations on the server hardware or the administrator may fail to correctly allocate memory resources to each NUMA node when configuring the server, resulting in the existence of NUMA nodes with insufficient or excessive local memory, which will also affect the optimization of data access and storage. Therefore, after obtaining the configuration information of the NUMA node, it is necessary to check the node according to the configuration information, and perform data layout based on the nodes that pass the check to ensure that the system resources can be correctly and efficiently used for data layout optimization.

[0064] In this embodiment, the historical access information of the data in the database is obtained by querying the database log. The data in the database includes data entities of various data structure types such as data tables, indexes, files, arrays, etc. The access information of the data includes the number of accesses, access time, access mode (sequential access, random access, etc.), etc. This embodiment does not list and explain in detail the type of database, data structure type, log content, etc.

[0065] S102: Determine the historical access frequency of the data and the historical data relevance according to the historical access information;

[0066] In this embodiment, after obtaining the historical access information of various types of data in the database, the historical access frequency of each type of data is determined by counting the historical access times of each type of data. The access frequency of each type of data can be determined by the historical access frequency, which provides a basis for data layout. For example, the historical access times of data tables and indexes within a certain period of time are counted respectively, and the historical access frequencies of data tables and indexes are determined according to the historical access times.

[0067] At the same time, considering that at least two types of related data entities are accessed simultaneously during the database query process, such as accessing a data table and an index simultaneously, data relevance is also a key factor in data layout.

[0068] In some embodiments, in order to determine the correlation between data entities, a preset correlation coefficient calculation method is used to calculate the correlation between two data entities in the database according to the historical access information obtained, and the historical data correlation between two data entities is determined according to the calculation result. Among them, the historical data correlation between two data entities can be determined based on the number of accesses of the two data entities in the same time period. In some embodiments, the Pearson correlation coefficient algorithm can be used to calculate the correlation coefficient of the two data entities according to the mean, standard deviation and covariance of the number of accesses of the two data entities in the same time period. When the calculated correlation coefficient reaches the preset correlation threshold, it can be considered that the two data entities have a strong correlation. For example, when the absolute value of the correlation coefficient is greater than or equal to 0.7, it is considered that the two data entities have a strong correlation. It can be understood that for two data entities with a strong correlation, in the same time period, when the number of accesses of one data entity increases, the number of accesses of the other data entity also increases, and when the number of accesses of one data entity decreases, the number of accesses of the other data entity also decreases.

[0069] In other implementations, the relationship between two data entities can be directly determined by analyzing the structure of the database and the query logic. For example, if there is a foreign key relationship between two data tables, then there may be a relationship between the two in some business logic. In some methods, the association between two data entities can be determined by a foreign key relationship, a JOIN operation in a query, or application logic.

[0070] S103: Determine a data layout strategy based on historical access frequency and historical data relevance;

[0071] In this embodiment, after determining the historical access frequency and historical data relevance of the data in the database, the data layout strategy is determined according to the historical access frequency and historical data relevance. The method for determining the data layout strategy includes dividing the data whose historical access frequency reaches a preset access threshold into the same data block; and / or dividing the data whose historical data relevance reaches a preset relevance threshold into the same data block. That is, data with a higher access frequency is divided into one data block, and data with a higher data relevance is divided into one database, or data with a higher access frequency and data with a higher data relevance are divided into the same data block together, thereby reducing data access across data blocks.

[0072] In some methods, the data volume can also be comprehensively considered to determine the data layout strategy, including dividing the data whose historical access frequency reaches a preset access threshold and whose data volume reaches a preset data volume threshold into the same data block; and / or dividing the data whose historical data relevance reaches a preset relevance threshold and whose data volume reaches a data volume threshold into the same data block. That is, data with a high access frequency and a certain data volume are divided into one data block, and data with a high data relevance and a certain data volume are divided into one data block, or data with a high access frequency and data with a high data relevance are divided into the same data block together, and the data volume in the data block reaches the data volume threshold, so that the data layout is more reasonable.

[0073] S104: Divide the data in the database into a plurality of data blocks according to the data layout strategy; wherein the number of the divided data blocks is the same as the number of NUMA nodes;

[0074] In this embodiment, after determining the data layout strategy, the data in the database is divided into multiple data blocks according to the data layout strategy, and the number of divided data blocks is the same as the number of NUMA nodes. In some methods, after determining the historical access frequency and historical data relevance of the data in the database, a clustering algorithm is used to divide the data whose historical access frequency reaches the access threshold into one category, and the data of this category is divided into the same data block, or the data whose historical data relevance reaches a preset relevance threshold is divided into one category, and the data of this category is divided into the same data block, or the data with a high historical access frequency and the data with a high historical data relevance are divided into one category, and the data of this category is divided into the same data block. In other methods, the data can be divided into multiple data blocks by using a clustering algorithm in combination with the historical access frequency, historical data relevance and data volume of the data.

[0075] S105: Allocate each data block to each NUMA node one by one.

[0076] like Figure 2As shown, in some embodiments, after the data in the database is divided into data blocks with the number of NUMA nodes, each data block is allocated to each NUMA node, that is, one NUMA node is allocated one data block, thereby realizing the logical partitioning of the data in the database. Through the logical partitioning of the data, the data with higher access frequency and / or higher data association is allocated to a NUMA node, and the data blocks are stored in the local memory of a NUMA node. The data can be accessed in the local memory of the same NUMA node, reducing the overhead of accessing data across nodes, improving data access efficiency, reducing system latency, and improving database performance. In some methods, a pre-reading mechanism (for example, the readahead mechanism of the Linux system) can be used to load data blocks into the cache in advance, that is, to place data blocks with higher access frequency on the same NUMA node, reducing the latency of cross-node access, and predicting the data blocks to be accessed by subsequent operations through the historical access information of the data, and using the pre-reading mechanism to pre-read these data blocks and place them in the local memory of the NUMA node where the processor initiating the access is located, thereby improving data processing efficiency.

[0077] In some implementations, during the initialization phase of the database management system, the data in the database is divided into multiple data blocks according to the number of NUMA nodes, the historical access frequency of the data in the database, the historical data relevance, and the data volume, etc.; the partitioning function of the database management system is utilized, the historical access frequency, the historical data relevance, and the data volume of the data blocks, as well as the load information and configuration information of the NUMA nodes are comprehensively considered, and the divided data blocks are allocated to the best NUMA nodes using hash partitioning, range partitioning, or other partitioning methods. For example, data blocks with higher access frequencies are allocated to NUMA nodes with higher processor performance, and data blocks with higher data relevance and larger data volumes are allocated to NUMA nodes with larger memory, etc. In some implementations, the scalability of the system can also be comprehensively considered, and data blocks are allocated to NUMA nodes with larger memory space, which is conducive to data expansion on the node.

[0078] In some embodiments, after allocating data blocks to NUMA nodes, a correspondence table between data blocks and NUMA nodes is established based on the allocation relationship between data blocks and NUMA nodes as a basis for data access and management. On the one hand, when executing a query operation, the NUMA node where the data block is located can be quickly queried based on the correspondence table, thereby improving query and processing efficiency; on the other hand, when the system load changes or the data access mode changes, the allocation strategy of the data block can be dynamically adjusted based on the correspondence table, for example, some data blocks on the NUMA node with a higher load can be migrated to the NUMA node with a lower load to achieve load balance, and the correspondence table can be updated after the data block is migrated. On the third hand, when abnormal situations such as node failure or data loss occur, the corresponding relationship list can be used to quickly restore the storage location and status of the data block to ensure data consistency and integrity. On the fourth hand, based on the analysis and statistics of the corresponding relationship table, the data distribution and usage of each NUMA node can be determined, providing strong data support for future capacity planning and expansion.

[0079] This embodiment provides an adaptive database processing method, which obtains relevant information of available NUMA nodes in an ARM processor architecture system that supports NUMA characteristics, divides the data in the database into multiple data blocks based on the historical access frequency and historical data relevance of the data in the database, allocates the data blocks to appropriate NUMA nodes, and optimizes the data layout based on the NUMA node characteristics, so as to ensure that high-frequency and associated data access is performed in the local memory of the same NUMA node as much as possible, which can significantly reduce the overhead of cross-node data access, improve the local performance and overall performance of data access, improve the database response speed, and improve the database performance, and is particularly suitable for high-concurrency scenarios of databases.

[0080] In some embodiments, the adaptive database processing method further includes:

[0081] When a preset layout adjustment cycle is reached or a preset layout adjustment condition is met, current access information of data in the database and load information of each NUMA node are obtained;

[0082] According to the current access information, determine the current access frequency of the data and the current data relevance;

[0083] Determine updated data layout strategy based on current access frequency and current data relevance;

[0084] Determine the data blocks to be optimized according to the updated data layout strategy and the current data layout strategy;

[0085] Determine the target NUMA node according to the current access frequency of the data block to be optimized and the load information of each NUMA node;

[0086] Migrate the data block to be optimized from the NUMA node where the data block to be optimized is located to the destination NUMA node.

[0087] In this embodiment, after the data in the database is divided into multiple data blocks according to the data layout strategy and allocated to each NUMA node, during the operation of the database, the data layout strategy can be dynamically adjusted when the layout adjustment cycle is reached, or when the layout adjustment conditions are met, to ensure that the database maintains good performance during operation. When adjusting the data layout strategy, the current access information of the data and the load information of the NUMA node are obtained, the current access frequency of the data and the current data relevance are determined according to the current access information, and the updated data layout strategy is determined according to the current access frequency and the current data relevance. According to the updated data layout strategy and the currently executed data layout strategy, the data block to be adjusted and the NUMA node where the data block is located are determined, and then combined with the load information of each NUMA node, a NUMA node suitable for migration is selected, and the data block to be adjusted is migrated to the NUMA node.

[0088] In some methods, the current access information obtained includes the number of accesses to data within a certain period of time. During the operation of the database, the current data relevance is determined by calculating the correlation coefficient between two data entities based on the number of accesses to data within a certain period of time and the access frequency of the statistical data. The load information of the NUMA node obtained includes the local processor occupancy, the local memory occupancy, the number of processes, etc. When the processor occupancy is large, the memory occupancy is large, and / or the number of processes is large, the load of the NUMA node is large.

[0089] In some embodiments, determining the data block to be optimized according to the updated data layout strategy and the current data layout strategy includes:

[0090] According to the updated data layout strategy, the data in the database is re-divided into multiple data blocks;

[0091] Compare the re-divided data blocks with the currently divided data blocks to determine the changed data blocks;

[0092] Determine the target NUMA node based on the current access frequency of the data block to be optimized and the load information of each NUMA node, including:

[0093] According to the load information of each NUMA node, a NUMA node that is adapted to the current access frequency of the data block to be optimized is selected from each NUMA node as the target NUMA node.

[0094] In this embodiment, according to the current access frequency and the current data relevance, an updated data layout strategy is determined, and the data in the database is re-divided into multiple data blocks according to the updated data layout strategy, and the re-divided multiple data blocks are compared with the currently divided multiple data blocks to determine the changed data blocks as the data blocks to be optimized, and the NUMA nodes are reallocated to the data blocks to be optimized. For example, a clustering algorithm is used to divide the data whose current access frequency reaches the access threshold into one category, and the data of this category is divided into the same data block, or the data whose current data relevance reaches the relevance threshold is divided into one category, and the data of this category is divided into the same data block, or the data with a high current access frequency and the data with a high current data relevance are divided into one category, and the data of this category is divided into the same data block. At the same time, according to the load of each NUMA node, a destination NUMA node suitable for the data block to be optimized is selected from them. For example, for a data block with a high access frequency, a NUMA node with a small load is selected from each NUMA node as the destination NUMA node, and the data block to be optimized is migrated to the destination NUMA node. By updating and adjusting the data layout, the adjusted data layout can adapt to the current operating status of the database.

[0095] In some implementations, to ensure data consistency and integrity, after selecting a destination NUMA node that is compatible with the data block to be optimized, determine the correlation between the data block to be optimized and the existing data blocks on the destination NUMA node. If there is data overlap or conflict between the data block to be optimized and the existing data blocks, it is necessary to perform data integration by deduplication, merging, or updating, and save the integrated data block on the destination NUMA node. Before the data block to be optimized is migrated, it is necessary to reserve sufficient storage resources for the data block to be optimized to ensure that the migration process does not significantly affect other applications or data blocks on the destination NUMA node. During the migration of the data block, it is necessary to temporarily lock or adjust access to the relevant data blocks on the destination NUMA node to ensure data consistency and integrity.

[0096] In some embodiments, the method for determining the layout adjustment condition includes:

[0097] Obtain access information of data in the database and load information of each NUMA node;

[0098] When it is determined according to the access information that there is data accessed across NUMA nodes and the amount of data accessed across NUMA nodes reaches a preset access amount threshold, it is determined that the layout adjustment condition is met; and / or,

[0099] When it is determined according to the load information of each NUMA node that the load of each NUMA node reaches an unbalanced state, it is determined that the layout adjustment condition is met.

[0100] In this embodiment, during the operation of the database, by real-time monitoring of the access information of the data in the database and the load information of each NUMA node, it is determined whether the data layout strategy needs to be updated. According to the access information of the data, when it is determined that there is a situation of accessing data across NUMA nodes, and the amount of data accessed across NUMA nodes reaches the access threshold, it is necessary to readjust the data layout to reduce the situation of accessing data across nodes and optimize local performance. Alternatively, when the load of each NUMA node reaches an unbalanced state, it is necessary to readjust the data layout so that each NUMA node reaches a load-balanced state. For example, due to the high frequency of data access on NUMA node 1 and other programs running at the same time, the load is large, while the load of NUMA node 2 is small. Therefore, when adjusting the data layout, the data blocks with higher access frequency can be migrated from NUMA node 1 to NUMA node 2. By adjusting the data layout, the NUMA node load is balanced and the overall system performance is improved.

[0101] In some methods, the process or thread accessing the data can be determined based on the access information, and by detecting the location of the data accessed by the process or thread, it can be determined whether there is data accessed across NUMA nodes. If the data accessed by the process or thread is located on different NUMA nodes, it can be determined that there is data accessed across nodes.

[0102] In some methods, before migrating the data block to be optimized from the NUMA node where it is located to the destination NUMA node, the migration cost of the data block should also be considered. The I / O cost, the occupied processor cost, and the possible lock contention cost of the data block to be optimized are calculated, and the data volume, access frequency, data relevance, and NUMA node load of the data block to be optimized are comprehensively considered to determine the comprehensive migration cost of the data block to be optimized.

[0103] After determining the comprehensive migration cost, select a suitable migration algorithm and use the physical storage management function of the database management system or third-party tools to migrate the data blocks to be optimized from the NUMA node where they are located to the destination NUMA node. The migration of data must ensure the integrity and consistency of the data and minimize the impact on system performance.

[0104] In some methods, the migration algorithms that can be selected include load-based migration and data-association-based migration. Among them, load-based migration is a dynamic resource management strategy that analyzes the real-time load of each NUMA node, identifies nodes with unbalanced loads, and migrates data blocks from high-load nodes to low-load nodes to balance system resources and improve overall performance and resource utilization. Migration based on data association focuses on optimizing the locality of data access. By analyzing the access patterns and transaction dependencies between data blocks, data blocks with strong associations are identified and migrated to the same NUMA node to reduce the delay and overhead of cross-node data access and improve data access efficiency and system performance.

[0105] In some methods, after migrating the data block to be optimized to the destination NUMA node, the effect of data migration can be verified through performance testing, data access information, NUMA node load information, etc., to determine whether the data access efficiency reaches the target and whether the NUAM node achieves load balance.

[0106] In some methods, the layout adjustment cycle can be determined according to application scenarios and performance requirements, for example, the layout adjustment cycle can be daily, weekly or monthly, and the specific time is not limited.

[0107] In some embodiments, the method for determining the layout adjustment condition also includes: obtaining the query type, data access hotspot, and cross-node access ratio of the database; determining whether the data access mode has changed according to the query type, data access hotspot, and cross-node access ratio; when it is determined that the data access mode has changed, determining the amount of system performance change caused by the change in the data access mode; if the amount of system performance change exceeds a preset change threshold, determining that the layout adjustment condition is met. In this embodiment, when the type of data objects in the query database changes, the data access hotspot changes, and / or the ratio of cross-node access to data changes, it can be determined whether the data access mode has changed, for example, the data access mode has changed due to changes in user behavior, changes in business logic, and growth in data volume; when it is determined that the data access mode has changed, the system performance changes caused by the current data access mode can be predicted using a pre-built performance prediction model based on the current data access mode. When it is predicted that the system performance has decreased to a certain extent, an early warning can be issued in a timely manner to prompt that the data layout needs to be adjusted in a timely manner. The specific impact on system performance under different data access modes can also be further analyzed, thereby locating important factors affecting performance.

[0108] like Figure 3 As shown, in some embodiments, the adaptive database processing method further includes:

[0109] S301: When the preset lock adjustment period is reached or the preset lock adjustment condition is met, the system performance parameters and access information of each data block are obtained;

[0110] S302: preprocessing the performance parameter and the access information of each data block to obtain the preprocessed performance parameter and access information;

[0111] S303: inputting the preprocessed performance parameters and access information into a preset lock granularity prediction model, and the lock granularity prediction model outputs a predicted lock granularity level;

[0112] S304: adjusting the lock granularity of each data block according to the lock granularity level.

[0113] In this embodiment, after optimizing the data layout of the database based on the processor architecture and NUMA characteristics, an adaptive lock strategy adjustment method is provided to improve the database performance by configuring and optimizing the lock strategy. After configuring the initial lock strategy when the database is initialized, during the operation of the database, the lock strategy can be dynamically adjusted when the lock adjustment cycle is reached, or when the lock adjustment condition is met, to ensure that the database maintains good performance during operation. When the lock adjustment cycle is reached or the lock adjustment condition is met, the performance parameters of the system and the access information of each database are obtained, and the performance parameters and access information are processed into a data format suitable for the input of the lock granularity prediction model, and then input into the lock granularity prediction model, and the lock granularity prediction model is used to predict the lock granularity level suitable for the optimal performance of the current database, and then the lock granularity of each data block is adjusted according to the predicted lock granularity level. By comprehensively considering the system performance parameters and the access information of the data, the optimal balance between the lock granularity and the performance is determined, and the concurrency performance of the database is improved.

[0114] In some methods, the performance parameters of the system can be obtained through system commands or system files. The performance parameters include processor usage, memory occupancy, I / O waiting time, etc. In the Linux system, they can be obtained by reading files such as / proc / stat and / proc / meminfo. The access information of each data block can be obtained from the database log, including the number of accesses, read / write ratio, and access frequency based on the number of accesses. After obtaining the original performance parameters and access information, the performance parameters and access information are cleaned, denoised, normalized and standardized, and then the processed data are converted into corresponding feature vectors, so that the pre-processed feature vectors are suitable for input lock granularity prediction model processing.

[0115] In some implementations, the output result of the lock granularity prediction model includes a predicted lock granularity level for each data block; according to the lock granularity level, the lock granularity of each data block is adjusted, including: according to the predicted lock granularity level for each data block, determining the data block whose current lock granularity level is different from the predicted lock granularity level, and adjusting the lock granularity level of the data block to the predicted lock granularity level. In this embodiment, considering that the access mode and performance requirements of each data block may be different, it is necessary to formulate appropriate lock granularity strategies for different data blocks. Among them, the lock granularity level includes table-level lock, page-level lock and row-level lock, and each lock granularity level has its own characteristics in terms of concurrency control, resource consumption and performance optimization.

[0116] For example, according to the lock granularity prediction model's prediction result of the lock granularity level for data block A, data block A is suitable for table-level locks to reduce lock management overhead because it is frequently accessed and has few update operations. If the lock granularity level currently configured for data block A is not a table-level lock, the current lock granularity level is adjusted to a table-level lock. According to the lock granularity prediction model's prediction result of the lock granularity level for data block B, data block B is suitable for row-level locks to improve concurrency performance because it has frequent read and write operations and many conflicts. If the lock granularity level currently configured for data block B is not a row-level lock, the current lock granularity level is adjusted to a row-level lock.

[0117] In some implementations, the training method of the lock granularity prediction model includes: obtaining performance parameter samples and data access information samples, preprocessing the performance parameter samples and the access information samples to obtain feature vectors suitable for input into the model, inputting the feature vectors into a machine learning model based on a support vector machine for training, optimizing the prediction performance by adjusting the model parameters during the training process, evaluating the model performance using a validation set, and selecting the best model parameters by cross-validation. During the operation of the database, the lock granularity prediction model can also be updated regularly based on the performance parameters and data access information obtained in real time to improve the accuracy of the prediction results and adapt to the ever-changing business scenarios.

[0118] In some methods, according to business needs, corresponding performance parameters are selected as training samples to train the model, so that the trained lock granularity prediction model can be applied to the corresponding application scenarios. For example, in the billing system of the power industry, it is necessary to meet the business needs of high concurrency, real-time and accuracy. The performance parameters such as transaction volume, response time, number of concurrent users, CPU usage, memory occupancy rate, etc. are selected as training samples for model training to obtain a lock granularity prediction model suitable for the billing system. The lock strategy is dynamically adjusted according to the prediction results of the model to reduce lock conflicts and waiting time, and improve transaction processing speed and system throughput. In the financial report generation system, it is necessary to meet the business needs of large data processing volume, complex report generation, fast query speed, and high resource utilization efficiency. The performance parameters such as query number, query response time, I / O waiting time, disk access frequency, cache hit rate, etc. are selected as training samples for model training to obtain a lock granularity prediction model suitable for the financial report generation system. The lock strategy is dynamically adjusted according to the prediction results of the model to optimize the query plan, reduce unnecessary lock waiting and resource consumption, and improve the query speed and resource utilization efficiency.

[0119] In some implementations, in different application scenarios, users can be supported to configure NUMA node allocation and lock granularity levels. By providing multiple configuration items, users can flexibly configure the allocation relationship between data blocks and NUMA nodes, the lock granularity level of data blocks, and configure data backup and recovery strategies to ensure system reliability.

[0120] In some embodiments, the method for determining the lock adjustment condition includes:

[0121] Get the system's operating parameters and NUMA node load information;

[0122] Determine whether there are performance bottlenecks and / or concurrency conflicts based on operating parameters and NUMA node load information;

[0123] If present, the lock adjustment condition is reached.

[0124] In this embodiment, during the operation of the database, by real-time monitoring the access information and performance parameters of the data in the database, it is determined whether the lock strategy needs to be updated. The operating parameters of the system are obtained, including the concurrency of transactions, the lock holding time, the queue length waiting for resources, etc., and according to the operating parameters and the load information of the NUMA node, it is determined whether there is a performance bottleneck (for example, whether the resource is close to or reaches saturation, whether the response time of the query or transaction is significantly extended, etc.), and potential concurrency conflicts (for example, by monitoring the lock waiting situation to determine whether there is a situation where execution cannot continue due to waiting for the lock, whether there is a deadlock, by determining whether a conflict occurs through the conflict log, whether there is a large number of transaction rollbacks, etc.), if there is, the lock adjustment condition is met, and the current lock strategy needs to be updated. When updating the lock strategy, the current performance parameters of the system and the current access information of each data block are obtained, and the current performance parameters and the current access information are pre-processed and input into the lock granularity prediction model, and the lock granularity prediction model outputs the best lock granularity level suitable for the current system state, and adjusts the lock granularity of each data block according to the predicted lock granularity level.

[0125] In some methods, after dynamically adjusting the lock strategy, the overall performance of the database can be comprehensively evaluated from multiple aspects such as transaction response time, throughput, and deadlock rate, and the lock strategy can be dynamically adjusted continuously based on the evaluation results to continuously optimize the overall performance of the database.

[0126] In some other embodiments, the method for determining the lock adjustment condition further includes:

[0127] Obtain the current performance parameters of the system and the current access information of each data block;

[0128] If the current performance parameter reaches a preset performance parameter threshold, and / or the current access information reaches a preset access threshold, it is determined that the lock adjustment condition is met.

[0129] In this embodiment, the change in the system's performance parameters and the change in the data access mode are comprehensively considered to determine whether the lock strategy needs to be adjusted. If one or more performance parameters reach the corresponding parameter threshold, the lock strategy needs to be readjusted, or if the data access mode has changed significantly, the lock strategy also needs to be readjusted. In addition, when adjusting the lock strategy, it is necessary to adopt an appropriate concurrency control strategy to ensure the consistency and integrity of the data and reduce the occurrence of concurrency conflicts.

[0130] In some methods, the lock adjustment period can be determined according to the application scenario and performance requirements. For example, the lock adjustment period can be daily, weekly or monthly. The specific adjustment period needs to be set based on the overall performance and stability of the system to avoid excessive optimization and bring additional system burden.

[0131] In some embodiments, the database management system initializes the concurrency control strategy in the initialization phase. When concurrent access is made to shared resources, concurrent processing is performed according to the concurrency control strategy and the configured lock strategy. Before accessing shared resources, a transaction selects appropriate locks (including shared locks, exclusive locks, etc.) and concurrency strategies (including optimistic locks, pessimistic locks, etc.) according to the type and access mode of the shared resources. If the requested shared resources have been locked by other transactions, it is necessary to wait for unlocking before accessing or perform other operations such as retrying and rolling back. In concurrent operations, it is necessary to detect whether there is a concurrency conflict. If there is, it is processed according to the concurrency control strategy, such as deadlock detection, transaction rollback, lock upgrade, etc. During the operation of the database, the concurrency control strategy can also be dynamically adjusted according to the load situation, the characteristics of the transaction, the changes in the access mode of the data, etc., such as adjusting the lock granularity, changing the locking mode, optimizing the lock holding time, etc. After the adjustment, the adjustment effect of the concurrency control strategy can be evaluated based on multiple indicators such as the response time, throughput, and deadlock occurrence rate of the transaction, and the concurrency control strategy can be optimized based on the evaluation results to improve the database performance.

[0132] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0133] It should be noted that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0134] like Figure 4 As shown, the embodiment of the present application also provides an adaptive database processing device, including:

[0135] An acquisition module, used to acquire NUMA node information and historical access information of data in a database; wherein the NUMA node information includes the number of NUMA nodes;

[0136] An analysis module, used to determine the historical access frequency of data and the historical data relevance based on the historical access information;

[0137] The layout module is used to determine the data layout strategy based on the historical access frequency and historical data relevance;

[0138] A partitioning module, used to partition the data in the database into a plurality of data blocks according to a data layout strategy; wherein the number of the partitioned data blocks is the same as the number of the NUMA nodes;

[0139] The allocation module is used to allocate each data block to each NUMA node one by one.

[0140] For the convenience of description, the above devices are described in terms of functions divided into various modules. Of course, when implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0141] The apparatus of the above-mentioned embodiment is used to implement the corresponding method in the above-mentioned embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0142] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0143] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0144] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0145] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0146] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0147] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0148] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0149] The electronic device of the above embodiment is used to implement the corresponding method in the above embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0150] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0151] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0152] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present disclosure, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0153] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0154] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present disclosure.

Claims

1. An adaptive database processing method, characterized in that: include: Obtaining NUMA node information and historical access information of data in a database; wherein the NUMA node information includes the number of NUMA nodes; Determine the historical access frequency of the data and the historical data relevance according to the historical access information; Determining a data layout strategy according to the historical access frequency and the historical data relevance; According to the data layout strategy, the data in the database is divided into a plurality of data blocks; wherein the number of the divided data blocks is the same as the number of the NUMA nodes; Allocate each data block to each NUMA node one by one.

2. The method according to claim 1, characterized in that According to the historical access frequency and the historical data relevance, a data layout strategy is determined, including: dividing the data whose historical access frequency reaches a preset access threshold into the same data block; and / or, The historical data whose relevance reaches a preset relevance threshold are divided into the same data block.

3. The method according to claim 1 or 2, characterized in that: Also includes: When a preset layout adjustment cycle is reached or a preset layout adjustment condition is met, current access information of data in the database and load information of each NUMA node are obtained; Determine the current access frequency of the data and the current data relevance according to the current access information; Determining an updated data layout strategy according to the current access frequency and the current data relevance; Determining the data blocks to be optimized according to the updated data layout strategy and the current data layout strategy; Determine the target NUMA node according to the current access frequency of the data block to be optimized and the load information of each NUMA node; The data block to be optimized is migrated from the NUMA node where the data block to be optimized is located to the destination NUMA node.

4. The method according to claim 3, characterized in that Determining the data blocks to be optimized according to the updated data layout strategy and the current data layout strategy includes: Re-dividing the data in the database into a plurality of data blocks according to the updated data layout strategy; Compare the re-divided data blocks with the currently divided data blocks to determine the changed data blocks; Determining a target NUMA node according to the current access frequency of the data block to be optimized and the load information of each NUMA node includes: According to the load information of each NUMA node, a NUMA node that is adapted to the current access frequency of the data block to be optimized is selected from each NUMA node as the target NUMA node.

5. The method according to claim 3, characterized in that: The method for determining the layout adjustment condition includes: Obtain access information of data in the database and load information of each NUMA node; According to the access information, when it is determined that there is data accessed across NUMA nodes, and the amount of data accessed across NUMA nodes reaches a preset access amount threshold, determining that the layout adjustment condition is met; and / or, When it is determined according to the load information of each NUMA node that the load of each NUMA node reaches an unbalanced state, it is determined that the layout adjustment condition is met.

6. The method according to claim 1, characterized in that After allocating each data block to each NUMA node one by one, it also includes: According to the allocation relationship between each data block and each NUMA node, a corresponding relationship table between the data blocks and the NUMA nodes is established.

7. The method according to claim 1, characterized in that Also includes: When a preset lock adjustment period is reached or a preset lock adjustment condition is met, the system performance parameters and access mode information of each data block are obtained; Preprocessing the performance parameters and access mode information to obtain preprocessed performance parameters and access mode information; Inputting the preprocessed performance parameters and access mode information into a preset lock granularity prediction model, and the lock granularity prediction model outputs a predicted lock granularity level; According to the lock granularity level, the lock granularity of each data block is adjusted.

8. The method according to claim 7, characterized in that The lock granularity level output by the lock granularity prediction model is the lock granularity level for each data block; and adjusting the lock granularity of each data block according to the lock granularity level includes: According to the predicted lock granularity level for each data block, determining data blocks whose current lock granularity level is different from the predicted lock granularity level; Adjust the lock granularity level of the data block to the predicted lock granularity level.

9. The method according to claim 7, characterized in that: The method for determining the lock adjustment condition includes: Get the system's operating status parameters; Determining whether there is a performance bottleneck and / or concurrency conflict based on the operating status parameters; If present, the lock adjustment condition is met.

10. An adaptive database processing device, characterized in that: include: An acquisition module, used to acquire NUMA node information and historical access information of data in a database; wherein the NUMA node information includes the number of NUMA nodes; An analysis module, used to determine the historical access frequency of data and the historical data relevance according to the historical access information; A layout module, used to determine a data layout strategy according to the historical access frequency and the historical data relevance; A partitioning module, used to partition the data in the database into a plurality of data blocks according to the data layout strategy; wherein the number of the partitioned data blocks is the same as the number of the NUMA nodes; The allocation module is used to allocate each data block to each NUMA node one by one.

Images