A data placement and thread scheduling method based on non-uniform persistent memory access

By generating a co-occurrence matrix and a relational graph, a NUMA partition mapping table is constructed, which solves the problem of remote access overhead caused by data dispersion in database systems under the NUMA architecture, and improves transaction processing efficiency and system performance.

CN119621282BActive Publication Date: 2025-11-04NORTHEASTERN UNIV CHINA
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411799921.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-12-09
Publication Date
2025-11-04
Estimated Expiration
2044-12-09

AI Technical Summary

Technical Problem

In database systems with NUMA architecture, traditional thread scheduling strategies lack sufficient consideration of the characteristics of database transactions, resulting in the data required for the same transaction being scattered across different NUMA nodes, increasing the overhead of remote access and reducing transaction processing efficiency.

Method used

By generating a co-occurrence matrix to record the access frequency of tables in transactions, a relationship graph and a NUMA partition mapping table are constructed to achieve reasonable data partitioning and thread binding to specific NUMA nodes, ensuring that data of the same transaction resides on the same node as much as possible, and reducing remote access overhead.

Benefits of technology

It achieves transaction-level optimized scheduling, reduces remote access overhead, improves the overall performance and adaptability of the database system, and ensures the rationality of data partitioning and lightweight overhead.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119621282B_ABST
    Figure CN119621282B_ABST
Patent Text Reader

Abstract

The application provides a data placement and thread scheduling method based on non-uniform persistent memory access, and relates to the technical field of databases. The method comprises a thread initialization stage, a transaction planning stage and a transaction execution stage. The frequency of accessing multiple tables in the same transaction is recorded by using a co-occurrence matrix. A relationship graph of the current access table and a NUMA partition mapping table are generated. Data placement and scheduling thread binding to a specific NUMA node are realized. The data contained in the tables frequently accessed together are divided into different NUMA nodes according to the same mapping rule. On the basis of the NUMA-aware data and thread placement strategy, in combination with the data access mode in the transaction processing process, the method realizes the optimized scheduling at the transaction level, reduces the overhead caused by remote access, and through the dynamic adjustment of the transaction co-occurrence matrix, can flexibly cope with the data access changes under different transaction loads, so that the system has higher adaptability and better overall performance.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database, and in particular to a data placement and thread scheduling method based on non-uniform persistent memory access. BACKGROUND

[0002] Data-driven applications are growing, and the performance requirements of database systems are also increasing. Non-uniform memory access (NUMA) is a multi-processor architecture design in which each processor has its own local memory, and accessing local memory is faster than accessing non-local memory (i.e. the memory of other processors or processor groups). This architecture is suitable for applications that require a large amount of memory and high concurrency, and can improve the performance of database systems. Persistent memory (PMEM) is an emerging storage technology that combines the low latency of memory and the persistence of disk, and can retain data after power failure. PMEM has byte-addressable characteristics and can be read and written directly through memory access. It is gradually being applied in database systems. However, in PMEM-based systems, the I / O performance degradation caused by remote access is more serious than in memory (DRAM). Therefore, it is necessary to study the data placement strategy and thread scheduling across NUMA to reduce the overhead of accessing persistent memory across NUMA nodes in database systems.

[0003] Currently, in the operating system under the NUMA architecture, the memory allocation strategies provided include random mapping, interleaved mapping, local mapping, balanced mapping, and hybrid mapping, etc. These strategies can be configured through the numactl tool of Linux. These mapping strategies have different advantages and disadvantages in different scenarios. For example, interleaved mapping considers node load balancing to ensure that continuous address memory access is evenly distributed among nodes, while local mapping maximizes local access but may cause node load imbalance. Hybrid mapping determines different strategies according to the access characteristics of memory pages, such as high balance for memory pages that are frequently accessed by most nodes, and high exclusivity for memory pages that are frequently accessed by only one node. When high balance is selected, interleaved mapping or balanced mapping is selected, and when high exclusivity is selected, hybrid mapping is selected. Thread scheduling strategies mainly include thread migration, thread core binding, and thread NUMA binding. In the database system under the NUMA architecture, data is generally split into different NUMA nodes according to different granularity ranges, and the allocation strategy is similar to the operating system, mainly balancing between load balancing and locality.

[0004] Traditional thread scheduling strategies often lack sufficient consideration of the characteristics of database transactions, especially in the case of processing transactions involving multiple tables. A transaction in a database usually contains multiple SQL statements and accesses multiple tables, and the data in different tables has some relationship and is often accessed together in a transaction. This relationship is called transaction affinity. Simply dividing data by range does not take into account transaction affinity, which will result in the data required by the same transaction being scattered across different NUMA nodes. In this case, even if the thread is bound to a specific NUMA node, remote access of data still incurs high access overhead, reducing the efficiency of transaction processing. Therefore, it is necessary to study the data placement strategy and thread scheduling across NUMA to reduce the overhead of accessing persistent memory across NUMA nodes in a database system. SUMMARY

[0005] The technical problem to be solved by the present application is to address the shortcomings of the prior art. Based on the analysis and summary of the characteristics of database transactions, it is known that multiple tables in a database are usually related to each other through foreign key fields, and the filter conditions in SQL statements are also based on fields. If a suitable field is selected to establish a mapping table for data division, tuples with the same field value are divided into the same NUMA node, lightweight data division can be achieved, and transaction affinity can be effectively supported when processing multiple-table transactions, reducing the overhead of remote access.

[0006] To solve the above technical problems, the technical solution adopted by the present application is: a data placement and thread scheduling method based on non-uniform persistent memory access, involving a thread initialization phase, a transaction planning phase and a transaction execution phase. The co-occurrence matrix is used to record the frequency of accessing multiple tables in the same transaction. By generating the relationship graph RelGraph of the current access table and the NUMA partition mapping table, data placement and thread binding to a specific NUMA node are realized. The data contained in the tables that are often accessed together are divided into different NUMA nodes according to the same mapping rule. The specific method comprises:

[0007] The thread initialization phase allocates thread resources for a non-uniform persistent memory database based on the NUMA architecture, initializes the thread rollback segment according to the NUMA node, and obtains the context resources of the corresponding node according to the NUMA node id bound to the current thread during transaction execution.

[0008] The transaction plan stage parses the id of the current access table and the field id information of the limit clause, the table information manager adds the table id to the access set of the transaction, and updates the field frequency in the table manager through the field id of the limit clause information, imports the access set into the transaction co-occurrence matrix, and records the frequency of accessing multiple tables in the same transaction by using the co-occurrence matrix; the creation / deletion of the table increases / decreases the row and column of the transaction co-occurrence matrix and increases / decreases the mapping of the table id to the matrix index, and the transaction co-occurrence matrix is subjected to a write lock operation during the creation / deletion of the table;

[0009] The co-occurrence matrix is an n*n matrix, where n is the number of current tables; the co-occurrence matrix records the frequency of accessing multiple tables in the same transaction, and each time a set of tables is accessed in a transaction, the co-occurrence count between the corresponding tables is increased, thereby reflecting the access correlation between them;

[0010] The transaction execution stage generates the relation graph RelGraph of the current access table and the NUMA partition mapping table, and according to the NUMA partition mapping table and the field id information of the limit clause parsed in the transaction plan stage, data placement and scheduling thread binding to a specific NUMA node are realized;

[0011] Further, the specific method for generating the relation graph RelGraph of the current access table by the transaction execution stage comprises:

[0012] (1) defining the current access table as CurTable, and other tables frequently accessed together with the current access table CurTable as OtherTable, the build table operation starts the warm-up stage after creating the CurTable, and with the insertion of tuples, the CurTable gradually increases, and when the CurTable reaches a specified threshold size, the warm-up ends;

[0013] (2) determining the high-frequency access OtherTable from the co-occurrence matrix by threshold judgment and topN method, filtering the OtherTable according to the size ratio of the OtherTable to the CurTable, wherein the size ratio of the OtherTable to the CurTable needs to satisfy 1:n, wherein n needs to be greater than a self-defined threshold MinScale, and the table-level filtering is completed;

[0014] (3) filtering the CurTable and the OtherTable according to the field type, and each pair of fields compared between the CurTable and the OtherTable needs to satisfy the matching of the field type;

[0015] (4) Based on the Levenshtein edit distance algorithm, combined with the characteristics of the 1:n relationship field, the field name similarity of the fields of CurTable and the fields of otherTable is sorted, and the fields with a similarity lower than the similarity threshold are filtered out; the Levenshtein edit distance algorithm is used to reduce the weight of the insertion and deletion operation, increase the weight of the replacement operation, sort the field name similarity, record each pair of fields in the sorted order from large to small, generate a field pair set, set a similarity threshold and filter out the fields with a similarity lower than the similarity threshold;

[0016] (5) The field values under the field name of CurTable are extracted from the field pair set, and the field value repetition degree matching is performed, the matched field pair is the field pair with 1:n relationship, the 1:n relationship field pairs contained in all fields of CurTable are determined, and finally the relationship graph RelGraph of CurTable is constructed;

[0017] The relationship graph RelGraph is a star graph centered on the current access table, and the adjacent vertices are tables having transaction affinity relationship with the table, which is called RelTable;

[0018] The transaction affinity relationship is defined as if the tables are frequently accessed together in a transaction and have a 1:n relationship, then they are considered to have transaction affinity;

[0019] Further, the specific method for generating the NUMA partition mapping table of the current access table in the transaction execution stage is: the NUMA scheduler calculates the closure of RelTable according to the relationship graph RelGraph of CurTable, wherein the closure refers to the union set of CurTable itself and RelTable of CurTable; the frequency of the fields in the closure of CurTable in all RelTable closures is counted; the data access frequency of each table contained in the closure of CurTable is obtained through the transaction co-occurrence matrix, and is used as a multiplier weight, and is multiplied by the frequency of the fields in the closure of CurTable in all RelTable closures to obtain the field with the highest product, which is the field for dividing CurTable, and a partition mapping table of data is generated;

[0020] Further, the field for dividing CurTable needs to meet:

[0021] 1) The field is often used as a filter condition of CurTable in SQL;

[0022] 2) The field must correspond to a unique field of a certain OtherTable which is frequently accessed with CurTable in a transaction;

[0023] 3) the unique field corresponding to this field is used as the maximum common field to establish the 1:n relationship in OtherTable which is frequently accessed with CurTable, so that the different tables involved in the whole transaction can be uniformly partitioned;

[0024] Further, the field id information of the restriction clause parsed according to the NUMA partition mapping table and the transaction plan phase is used to implement the specific method of data placement and thread binding to a specific NUMA node, which is that the NUMA scheduler uses the library function numa_alloc_onnode provided by libnuma to directly specify the NUMA node to implement data placement, and the current thread is bound to the CPU set of the specified NUMA node by calling the pthread_setaffinity_np interface function to implement thread binding.

[0025] The CPU set of each NUMA node refers to that the CPU set is initialized when the storage engine is started, and the corresponding bitmask is allocated for each node to represent the available CPU set on the node.

[0026] The beneficial effects generated by the above technical solution are that the data placement and thread scheduling method based on non-uniform persistent memory access provided by the application, on the basis of the NUMA-aware data and thread placement strategy, combines the data access mode in the transaction processing process, adopts the data division method, realizes the optimized scheduling at the transaction level, ensures that the data accessed by the transaction is located in the same NUMA node as much as possible, and reduces the overhead caused by remote access. Through the dynamic adjustment of the transaction co-occurrence matrix, the data access changes under different transaction loads can be flexibly coped with, so that the system has higher adaptability and better overall performance. In addition, the application maintains the table-based relationship graph and the field-granularity division, ensures the rationality of data division and the balance of lightweight overhead, provides a solution for the NUMA-aware scheduling of the persistent memory database, and has important application value. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 The data placement and thread scheduling method based on non-uniform persistent memory access provided by the embodiment is shown in the architecture diagram, wherein the CooccurenceMatrix is a transaction co-occurrence matrix, which is used to obtain a table set frequently accessed together on a transaction, and reflects the workload; the RelGraph is a transaction affinity relationship graph, which reflects the transaction affinity relationship between tables in combination with the workload and the relationship between tables; the NUMA partition mapping table routes the NUMA node according to the field id.

[0028] Figure 2A flowchart of a data placement and thread scheduling method based on non-uniform persistent memory access is provided for the embodiment;

[0029] Figure 3 A thread resource allocation diagram is provided for the embodiment, taking work thread 1 as an example. Thread execution is divided into three time periods, which are bound to node 0, node 1 and node 3 during execution, respectively. When the transaction starts to obtain the context, the node where the thread is located is obtained first, and then AllocUndoContext() allocates resources of the specified node for the thread;

[0030] Figure 4 A transaction planning phase transaction information mining flowchart is provided for the embodiment. The planning phase parses the table id and the field id of the restriction clause. The table id is added to the access set (accessTables) of transaction 1, and the field frequency (Frequency) in the (global) table management is updated through the field id in the restriction clause information;

[0031] Figure 5 A flowchart of a data placement and thread scheduling method after a preheating phase of a table under a typical OLTP load TPCC instance is provided for the embodiment. In the flowchart, O represents the Order table, D represents the District table, W represents the Warehouse table, I represents the Item table, and OL represents the Order_line table. DETAILED DESCRIPTION

[0032] The specific embodiments of the present application will be further described in detail below in conjunction with the drawings and examples. The following examples are used to illustrate the present application, but are not used to limit the scope of the present application.

[0033] In the embodiment, a data placement and thread scheduling method based on non-uniform persistent memory access uses a co-occurrence matrix to record the frequency of accessing multiple tables in the same transaction. By generating a relation graph of the current access table and a NUMA partition mapping table, data placement and scheduling threads are bound to specific NUMA nodes, as shown in Figure 1 , wherein the co-occurrence matrix (CooccurenceMatrix) is used to obtain a set of tables that are frequently accessed together in a transaction, reflecting the workload; the transaction affinity graph (Relation Graph) reflects the transaction affinity relationship between tables in combination with the relationship between the workload and the tables; and the NUMA partition mapping table routes the NUMA node according to the field id.

[0034] The data placement and thread scheduling method based on non-uniform persistent memory access, as shown in Figure 2 , involves a thread initialization phase, a transaction planning phase and a transaction execution phase, and specifically includes:

[0035] The thread initialization stage allocates thread resources for the non-uniform persistent memory database based on the NUMA architecture, initializes the thread rollback segment according to the NUMA node, and acquires the context resources of the corresponding node according to the NUMA node id bound to the current thread during transaction execution;

[0036] The embodiment is based on the persistent memory storage engine nvmdb under the Huawei enterprise-level open source relational database opengauss, and the recovery of the non-uniform persistent memory database does not need to execute the redo committed transaction, but only needs to execute the undo uncommitted transaction. Each thread uses the rollback segment to store the undo log, initializes the thread rollback segment according to the NUMA node, and acquires the context resources of the corresponding node according to the node id bound to the current thread during transaction execution, as shown in Figure 3 ;

[0037] The transaction planning stage parses the id of the current access table and the field id information of the limit clause, constructs a (global) table information manager, and each table id corresponds to a table information (TableInfo). The table id is added to the access set (AccessTables) of the transaction, and the field frequency (Frequency) in the (global) table manager is updated through the field id information of the limit clause. In the commit phase of the transaction, the table access set is imported into the transaction co-occurrence matrix, and the co-occurrence matrix is used to record the frequency of accessing multiple tables in the same transaction, as shown in Figure 4 ; The creation / deletion of a table will increase / decrease the row and column of the transaction co-occurrence matrix and increase / decrease the mapping of the table id to the matrix index, and the transaction co-occurrence matrix will be write-locked during the creation / deletion of the table;

[0038] The co-occurrence matrix is an n*n matrix, where n is the number of current tables. The co-occurrence matrix records the frequency of accessing multiple tables in the same transaction. Each time a transaction accesses a group of tables, the co-occurrence count between the corresponding tables will increase, thereby reflecting their access correlation;

[0039] The transaction execution stage generates the relation graph RelGraph of the current access table and the NUMA partition mapping table, and according to the NUMA partition mapping table and the field id information of the limit clause parsed in the transaction planning stage, realizes data placement and thread binding to a specific NUMA node;

[0040] The specific method for generating the relation graph RelGraph of the transaction execution stage includes:

[0041] (1) define the current access table as CurTable, and other tables frequently accessed with the current access table as OtherTable, the build table operation starts the warm-up phase after creating CurTable, and CurTable gradually increases with the insertion of tuples, and when CurTable reaches the specified threshold size, the warm-up ends;

[0042] (2) determine the high-frequency access OtherTable from the co-occurrence matrix through the threshold judgment and topN method, filter OtherTable according to the size ratio of OtherTable to CurTable, wherein the size ratio of OtherTable to CurTable needs to meet 1:n, wherein n needs to be greater than the self-defined threshold MinScale, and the table-level filtering is completed;

[0043] (3) filter the field types of CurTable and OtherTable, and each pair of fields compared between CurTable and OtherTable needs to meet the field type matching;

[0044] (4) based on the Levenshtein edit distance algorithm, combined with the characteristics of 1:n relationship fields, sort the field name similarity of the fields of CurTable and otherTable, and filter out the fields with similarity lower than the similarity threshold; use the Levenshtein edit distance algorithm to reduce the weight of the insertion and deletion operation, and increase the weight of the replacement operation, sort the field name similarity, record each pair of fields in descending order, generate a field pair set, set a similarity threshold and filter out the fields with similarity lower than the similarity threshold;

[0045] (5) extract the field values under the field name of CurTable from the field pair set, match the field value repetition degree, and the matched field pair is the field pair with 1:n relationship, determine the 1:n relationship field pair contained in all fields of CurTable, and finally construct the relationship graph RelGraph for CurTable;

[0046] The relationship graph RelGraph is a star graph centered on the previously accessed table CurTable, and adjacent vertices are tables having transaction affinity relationship with the table, and adjacent edges are matching fields of the two tables having 1:n relationship, referred to as RelTable. The transaction affinity relationship between CurTable and OtherTable is stored using the relationship graph RelGraph, and through the relationship graph RelGraph, the field most suitable for establishing a mapping table can be calculated, which has certain transaction affinity while ensuring lightweight overhead, and can make tables frequently accessed together divided into different nodes according to the same mapping rule as far as possible; the transaction affinity is defined as if the tables have 1:n relationship and are frequently accessed together in transactions, then the two tables are regarded as having transaction affinity; in the embodiment, the similarity threshold is set to 0.5 to obtain the filtered field pair set;

[0047] The specific method of generating the NUMA partition mapping table of the currently accessed table in the transaction execution stage is as follows: the NUMA scheduler calculates the closure of RelTable according to the relationship graph RelGraph of CurTable, and the closure refers to the union set of CurTable itself and the RelTable of CurTable; the frequency of the fields in the closure of CurTable in all RelTable closures is counted; the data access frequency of each table contained in the closure of CurTable is obtained through the transaction co-occurrence matrix, and is taken as a multiplier weight, and is multiplied by the frequency of the fields in the closure of CurTable in all RelTable closures, to obtain the field with the highest product as the field for dividing CurTable, and generate the partition mapping table of data;

[0048] In the process of selecting the field for dividing CurTable, the field for dividing CurTable needs to meet the following conditions:

[0049] 1) The field is often used as a filter condition of CurTable in SQL;

[0050] 2) The field must correspond to a unique field (unique) of a certain OtherTable frequently accessed with CurTable in transactions;

[0051] 3) The unique field corresponding to the field is the largest common field, and can most establish 1:n relationship in the OtherTable frequently accessed with CurTable, so that different tables involved in the entire transaction can be uniformly partitioned;

[0052] According to the NUMA partition mapping table and the limit clause field id information parsed out in the transaction planning stage, the specific method of data placement and thread binding to a specific NUMA node is as follows: the NUMA scheduler uses the library function numa_alloc_onnode provided by libnuma to directly specify the NUMA node for data placement according to the partition mapping table and the parsed field information, and binds the current thread to the CPU set of the specified NUMA node by calling the pthread_setaffinity_np interface function, so as to realize thread binding. The CPU set of each NUMA node refers to the CPU set that is initialized when the storage engine is started and is allocated to each node with a corresponding bitmask, which represents the available CPU set on the node.

[0053] In this embodiment, the data placement and thread scheduling method based on non-uniform persistent memory access is used to verify the effect of data placement on a typical OLTP load TPCC instance, as shown in the following table: Figure 5 As shown in the table, O represents the Order table, D represents the District table, W represents the Warehouse table, I represents the Item table, and OL represents the Order_line table. Compared with the prior art, the data placement and thread scheduling method based on non-uniform persistent memory access provided by the present application establishes a relationship graph with transaction affinity and a partition mapping table, so that different tables can map data to the same node according to the same rule, thereby improving the effect when performing a cross-table transaction. In the TPCC test result, the data of most tables can be divided according to the Warehouse with the same mapping rule, effectively improving the performance of data access.

[0054] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope defined by the claims of the present application.

Claims

1. A data placement and thread scheduling method based on non-uniform persistent memory access, characterized in that: It involves the thread initialization phase, transaction planning phase, and transaction execution phase. It uses a co-occurrence matrix to record the frequency of accessing multiple tables in the same transaction. By generating a RelGraph of the relationship between the currently accessed tables and a NUMA partition mapping table, it realizes data placement and scheduling threads to be bound to specific NUMA nodes. It divides the data contained in tables that are frequently accessed together into different NUMA nodes according to the same mapping rules. The co-occurrence matrix is ​​an n×n matrix, where n is the number of current tables. The co-occurrence matrix records the frequency of accessing multiple tables in the same transaction. Each time a transaction accesses a set of tables, the co-occurrence count between the corresponding tables increases, thus reflecting their access correlation. The RelGraph is a star graph centered on the currently accessed table, with adjacent vertices being tables that have transactional affinity with this table, referred to as RelTable; The transaction affinity is defined as follows: if tables are frequently accessed together in a transaction and have a 1:n relationship, then they are considered to have transaction affinity.

2. The data placement and thread scheduling method based on non-uniform persistent memory access according to claim 1, characterized in that: The thread initialization phase allocates thread resources to the non-uniform persistent memory database based on the NUMA architecture, initializes the thread rollback segment according to the NUMA node, and obtains the context resources of the corresponding node according to the NUMA node ID bound to the current thread during transaction execution.

3. The data placement and thread scheduling method based on non-uniform persistent memory access according to claim 2, characterized in that: The transaction planning phase parses the ID of the currently accessed table and the field ID information of the constraint clause, constructs a table information manager to add the table ID to the access set of the transaction, updates the field frequency in the table information manager through the field ID information of the constraint clause, imports the access set into the transaction co-occurrence matrix, and uses the co-occurrence matrix to record the frequency of accessing multiple tables in the same transaction. The creation / deletion of a table increases / decreases the number of rows and columns in the transaction co-occurrence matrix and increases / decreases the mapping from table ID to matrix index. During the creation / deletion of a table, a write lock operation is performed on the transaction co-occurrence matrix.

4. The data placement and thread scheduling method based on non-uniform persistent memory access according to claim 3, characterized in that: The transaction execution phase generates a RelGraph of the current access table and a NUMA partition mapping table. Based on the NUMA partition mapping table and the field id information of the constraint clause parsed in the transaction planning phase, data placement and scheduling threads are bound to specific NUMA nodes.

5. The data placement and thread scheduling method based on non-uniform persistent memory access according to claim 4, characterized in that: The specific methods for generating the RelGraph of the currently accessed table during the transaction execution phase include: (1) Define the currently accessed table as CurTable, and other tables accessed frequently along with the currently accessed table CurTable as OtherTable. After the table creation operation creates CurTable, the warm-up phase begins. As tuples are inserted, CurTable gradually increases in size. When CurTable reaches the size of a specified threshold, the warm-up ends. (2) Determine the frequently accessed OtherTable from the co-occurrence matrix by threshold judgment and topN method, and filter OtherTable according to the size ratio of OtherTable to CurTable. The size ratio of OtherTable to CurTable must satisfy 1:n, where n must be greater than the custom threshold MinScale to complete the table-level filtering. (3) Filter the field types of CurTable and OtherTable. Each pair of fields compared between CurTable and OtherTable must meet the field type matching requirement. (4) Based on the Levenshtein edit distance algorithm and combined with the characteristics of 1:n relation fields, sort the fields of CurTable and otherTable by field name similarity and filter out fields with similarity below the similarity threshold; use the Levenshtein edit distance algorithm to reduce the weight of add and delete operations and increase the weight of replace operations, sort the fields by field name similarity, record each pair of sorted fields in descending order, generate a set of field pairs, set a similarity threshold and filter out fields with similarity below the similarity threshold; (5) Extract the field values ​​under the field names of CurTable from the field pair set, perform field value duplication matching, and the field pairs that are successfully matched are the field pairs with 1:n relationship. Determine the 1:n relationship field pairs contained in all fields of CurTable, and finally construct the relationship graph RelGraph for CurTable.

6. The data placement and thread scheduling method based on non-uniform persistent memory access according to claim 5, characterized in that: The specific method for generating the NUMA partition mapping table of the currently accessed table during the transaction execution phase is as follows: The NUMA scheduler calculates the closure of the RelTable based on the RelGraph of the CurTable. The closure refers to the union of the CurTable itself and the RelTable of the CurTable. The frequency of occurrence of fields in the closure of the CurTable in all RelTable closures is counted. The data access frequency of each table contained in the closure of the CurTable is obtained through the transaction co-occurrence matrix. This frequency is used as a multiplier weight and multiplied by the frequency of occurrence of fields in the closure of the CurTable in all RelTable closures. The field with the highest product is the field used as the basis for partitioning the CurTable, and the data partition mapping table is generated.

7. The data placement and thread scheduling method based on non-uniform persistent memory access according to claim 6, characterized in that: The fields used as the basis for partitioning CurTable must satisfy the following: 1) This field is frequently used as a filter condition for CurTable in SQL; 2) This field must correspond to a unique field (unique) of an OtherTable that is frequently accessed together with CurTable in a transaction; 3) The unique field corresponding to this field serves as the largest common field, and can also establish a maximum of 1:n relationships in OtherTable, which is frequently accessed together with CurTable, so that different tables involved in the entire transaction can be uniformly partitioned.

8. The data placement and thread scheduling method based on non-uniform persistent memory access according to claim 7, characterized in that: The specific method for binding data placement and scheduling threads to specific NUMA nodes based on the NUMA partition mapping table and the field id information of the constraint clause parsed during the transaction planning phase is as follows: The NUMA scheduler uses the libnuma library function numa_alloc_onnode to directly specify the NUMA node to implement data placement based on the partition mapping table and the parsed field information. By calling the pthread_setaffinity_np interface function, the current thread is bound to the CPU set of the specified NUMA node to achieve thread binding. The CPU set of a specified NUMA node refers to the set of CPUs that are initialized when the storage engine starts and a corresponding bitmask is assigned to each node, representing the set of CPUs available on that node.

Citation Information

Patent Citations

  • Systems and methods for managing databases

    CN109416682A

  • Distributed transaction processing method and device, equipment, medium and program product

    CN117348980A