NUMA-aware mickey concurrent access control method and system

Abstract

This application provides a NUMA-aware concurrent access control method and system for key cards, including: a parent process creating shared memory partitioned by NUMA nodes and pre-parsed the global lock symbol of the key card library, storing the virtual address of the global lock in the header of the shared memory, and the shared memory being used to store the contention rate, ticket allocator, and current service ticket corresponding to each NUMA node; a child process obtaining the virtual address of its own NUMA node and the global lock, obtaining the state of the global lock through read-only memory mapping, where read-only memory mapping is a mapping method that only reads and does not modify the original memory of the key card library; the child process performing layered avoidance operations according to the state of the NUMA node and the global lock to complete the key card initialization. This application does not modify the source code of the key card library, does not add proxy processes, avoids the thundering herd problem of global lock and inefficiency of cross-node access, reduces high-concurrency initialization latency and lock conflict rate, and ensures system stability.

Description

Technical Field

[0001] This application belongs to the field of computer system concurrency control technology, and in particular relates to a NUMA-aware cryptographic card concurrency access control method and system. Background Technology

[0002] In multi-process database transparent data encryption scenarios, hardware encryption cards are often used to store master keys and perform encryption operations. Each database child process needs to independently execute the encryption card, such as PKCS#11, to initialize and establish an encryption card session. In existing technologies, PKCS#11 encryption card libraries typically use global locks to ensure concurrency safety. When forking child processes during high concurrency, a large number of processes competing for the global lock simultaneously can trigger the thundering herd effect, leading to a cascading drop in initialization performance and causing some processes to fail due to timeouts. Conventional concurrency control schemes either modify the PKCS#11 encryption card library source code to add a queuing mechanism, introducing compatibility and maintenance risks; or introduce a centralized proxy process, increasing additional inter-process communication overhead and single point of failure risks, and requiring the parent process to participate in proxy management, violating the security constraint that the parent process does not hold encryption card sessions or participate in encryption card operations. In addition, existing technologies assume that all processes have the same memory access latency, without considering the NUMA architecture commonly used in modern servers. The latency of cross-node processes accessing the global lock is 3 to 5 times that of processes on the same node. A unified scheduling strategy can cause high-performance processes on the same node to be blocked by low-performance processes on cross-nodes, resulting in a significant decrease in overall efficiency. Summary of the Invention

[0003] This application provides a NUMA-aware concurrent access control method and system for key cards. Without modifying the key card library source code, the database fork process model, or the parent process passing any key card context, it simultaneously solves the thundering herd problem of key card global lock conflicts in high-concurrency fork scenarios and the low efficiency of cross-node access under NUMA architecture.

[0004] This application discloses a NUMA-aware cryptographic card concurrent access control method, including: The parent process creates shared memory partitioned by NUMA nodes and pre-parses the global lock symbol of the MAC library. It stores the virtual address of the global lock in the header of the shared memory. The shared memory is used to store the contention rate, ticket allocator and current service ticket for each NUMA node. The child process obtains the virtual address of its own NUMA node and the global lock, and obtains the state of the global lock through read-only memory mapping, wherein the read-only memory mapping is a mapping method that only reads and does not modify the original memory of the key library; The child process performs a hierarchical avoidance operation based on the state of the NUMA node and the global lock to complete the key card initialization.

[0005] Optionally, the parent process creates shared memory partitioned by NUMA nodes and pre-parses global lock symbols from the MAC library, including: The parent process reads the total number of NUMA nodes on the server, creates an anonymous shared memory of the corresponding size, and sets the permissions of the shared memory to read-only for the child process and read, write, and shareable for the parent process. The parent process loads the key card library and obtains the virtual address of the global lock through the key card library symbol resolution interface; The parent process initializes the contention rate, ticket allocator, and current service ticket for each NUMA node to their initial values.

[0006] Optionally, the child process obtains the virtual address of its own NUMA node and the global lock, including: The child process obtains the NUMA node identifier bound to the current process through the operating system's native API; The child process reads the virtual address of the global lock from the header of the shared memory; The child process maps the memory page containing the global lock into its own address space in a private read-only mode.

[0007] Optionally, the state of the global lock can be obtained through read-only memory mapping, including: The child process uses an atomic load operation with a memory barrier to read the state field of the global lock; The child process determines whether the global lock is held based on the value of the status field; The child process writes the detection results to shared memory and updates the contention rate of the corresponding NUMA node.

[0008] Optionally, the child process performs a layered avoidance operation based on the state of the NUMA node and the global lock, including: The child process reads the contention rate of the corresponding NUMA node and performs hierarchical spin operations based on the contention rate; The child process calculates the backoff base based on the contention rate and performs an adaptive sleep avoidance operation; When the number of avoidance attempts reaches the threshold, the child process acquires execution rights through an atomic Ticket sequence and completes the key card initialization.

[0009] Optionally, hierarchical spin operations are performed based on the competition rate, including: If the child process is on the same NUMA node as the global lock, the child process performs a spin that is inversely proportional to the contention rate, with a fixed spin interval, during which the state of the global lock is repeatedly checked. If the child process is on a different NUMA node than the global lock, the child process will not perform a spin operation and will directly enter an adaptive sleep avoidance operation.

[0010] Optionally, perform adaptive sleep avoidance operations, including: The child process determines the backoff base based on the contention rate of the corresponding NUMA node; the higher the contention rate, the larger the backoff base. The subprocess calculates the exponential backoff time based on the current number of avoidances, and adds random jitter to scatter the wake-up time; The child process adjusts its sleep duration based on the differences between NUMA nodes, and the sleep duration of the cross-node process is a preset multiple of that of the process on the same node.

[0011] Optionally, execution rights can be acquired through an atomic sequence of tickets, including: The child process atomically obtains a unique Ticket sequence number from the Ticket allocator of the corresponding NUMA node; The child process enters a waiting state until the current service ticket of the corresponding NUMA node equals its own ticket number; After the child process completes the key card initialization, it atomically increments the current service ticket of the corresponding NUMA node.

[0012] Optionally, update the contention rate of the corresponding NUMA node, including: The child process retrieves the historical contention rate of the corresponding NUMA node; The subprocess calculates a new contention rate using an exponential moving average algorithm that employs a preset smoothing coefficient. The child process updates the contention rate in shared memory through atomic comparison-swap operations.

[0013] This application also discloses a NUMA-aware concurrent access control system for cryptographic cards, including: The parent process initialization module is used to create shared memory partitioned by NUMA nodes and pre-parse the global lock symbols of the MAC library. The virtual address of the global lock is stored in the header of the shared memory. The shared memory is used to store the contention rate, ticket allocator and current service ticket corresponding to each NUMA node. The lock contention detection module is used to enable the child process to obtain the virtual address of its own NUMA node and the global lock, and to obtain the state of the global lock through read-only memory mapping, wherein the read-only memory mapping is a mapping method that only reads and does not modify the original memory of the key library; The hierarchical avoidance execution module is used to enable the child process to perform hierarchical avoidance operations based on the state of the NUMA node and the global lock, and complete the key card initialization.

[0014] As can be seen from the above technical solution, this application completes global lock symbol resolution and shared memory creation by NUMA nodes in one go through the parent process. All child processes can directly read the global lock address from the shared memory header without repeatedly performing the high-overhead symbol resolution operation. The shared memory structure divided by nodes naturally isolates the scheduling data of different nodes, avoiding cross-node data interference. The child processes use a private read-only mode to map the memory page where the lock is located to read the lock state, without modifying the original memory data of the key card library, ensuring that the running logic of the third-party library is not interfered with, and at the same time, it can obtain the real-time contention state of the lock without intrusion. The child processes combine their own NUMA node information and the real-time state of the global lock to perform layered avoidance, which can limit most of the contention to the same node and reduce the high-latency cache consistency traffic across nodes. In the entire process, the parent process only performs pre-resolution and shared memory creation operations and does not participate in any key card-related operations, which fully complies with security constraints. The various links are naturally connected, and there is no need to modify the database process model and the source code of the third-party library, which can reduce the average latency of key card initialization and lock conflict rate in high-concurrency scenarios and improve the stability of system operation. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a NUMA-aware concurrent access control method for key cards in an embodiment of this application; Figure 2 This is a flowchart illustrating the process of a parent process creating shared memory partitioned by NUMA nodes and pre-parsed the global lock symbol of the PKCS#11 key card library in a NUMA-aware concurrent access control method according to an embodiment of this application. Figure 3 This is a flowchart illustrating how a child process obtains the virtual address of its own NUMA node and the global lock in a NUMA-aware concurrent access control method according to an embodiment of this application. Figure 4 This is a flowchart illustrating a NUMA-aware concurrent access control method for a key card, as described in this application, which obtains the state of a global lock through read-only memory mapping. Figure 5 This is a flowchart illustrating a NUMA-aware concurrent access control method for MAC addresses in this application, showing how a subprocess performs a hierarchical avoidance operation based on the state of the NUMA node and the global lock. Figure 6This is a flowchart illustrating a NUMA-aware SIM card concurrency access control method in this application that performs hierarchical spin operations based on contention rate. Figure 7 This is a flowchart illustrating the adaptive sleep avoidance operation performed by a NUMA-aware SIM card concurrent access control method in an embodiment of this application. Figure 8 This is a flowchart illustrating a NUMA-aware concurrent access control method for acquiring execution rights through an atomic Ticket sequence, as described in an embodiment of this application. Figure 9 This is a flowchart illustrating the update of the contention rate of a NUMA-aware SIM card concurrent access control method in an embodiment of this application. Figure 10 This is a schematic diagram of a NUMA-aware concurrent access control system for cryptographic cards, provided as an embodiment of this application. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not limiting, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without such specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0018] In this embodiment, a typical implementation of multi-process database access hardware key cards in the prior art involves the database parent process loading the key card library's configuration information upon startup, and each child process independently executing the key card initialization process after startup, including loading dynamic libraries, initializing global state, enumerating devices, and establishing sessions. To ensure multi-threaded and multi-process concurrency safety, the key card library adds a global mutex lock before and after global operations such as initialization and device enumeration, allowing only one process to execute these operations at a time. During peak database traffic periods, when dozens or even hundreds of child processes are forked per second, all child processes simultaneously attempt to acquire the global lock, causing a large number of processes to be blocked in a lock-waiting state, triggering a thundering herd effect. The operating system needs to frequently switch process contexts, significantly increasing CPU utilization and raising the average initialization latency from microseconds to milliseconds. Some processes may fail to initialize due to timeouts, thus affecting the database's business processing capabilities. On servers using a NUMA architecture, the global lock is stored in the memory of one of the NUMA nodes. Processes running on other nodes need to access this lock via a cross-node bus, resulting in a latency 3 to 5 times that of accessing the same node. A unified scheduling strategy leads to cross-node processes and processes on the same node competing for locks. Cross-node processes take longer to acquire locks, which can block processes on the same node and further reduce overall initialization efficiency. To address these issues, existing technologies typically employ two solutions. One is to modify the source code of the key card library, replacing the global lock with a fine-grained lock, or adding a queuing mechanism within the library to allow processes to acquire locks sequentially. This solution requires modifying the source code of third-party libraries, and the implementations of key card libraries from different vendors vary significantly. Modification introduces compatibility issues, and subsequent library upgrades require further modifications, resulting in high maintenance costs. The other solution is to introduce a separate proxy process. All key card operations from database child processes are forwarded through the proxy process, which maintains a request queue and executes operations sequentially. This solution introduces additional inter-process communication overhead, as each key card operation requires data transmission between processes, increasing latency several times over. Furthermore, the proxy process becomes a single point of failure; if the proxy process crashes, all database child processes become unable to access the hardware key card. Furthermore, the proxy process needs to be started and managed by the database parent process, and the parent process needs to participate in the key card-related operations, which violates the security constraint that the parent process does not hold the key card session or participate in key card operations in the transparent encryption scenario of the database, and there is a risk of session inheritance and context pollution.

[0019] Based on this, embodiments of this application provide a NUMA-aware cryptographic card concurrent access control method, such as... Figure 1 As shown, it includes: S100: The parent process creates shared memory partitioned by NUMA nodes and pre-parses the global lock symbol of the MAC library. It stores the virtual address of the global lock in the header of the shared memory. The shared memory is used to store the contention rate, Ticket allocator and current service Ticket corresponding to each NUMA node. S200: The child process obtains the virtual address of its own NUMA node and the global lock, and obtains the state of the global lock through read-only memory mapping, wherein the read-only memory mapping is a mapping method that only reads and does not modify the original memory of the key library; S300: The subprocess performs a hierarchical avoidance operation based on the state of the NUMA node and the global lock, and completes the key card initialization.

[0020] This application provides a NUMA-aware concurrent access control method for key cards. The parent process completes global lock symbol resolution and shared memory creation (divided by NUMA nodes) in one step. All child processes can directly read the global lock address from the shared memory header, eliminating the need for repeated, high-overhead symbol resolution operations. The node-divided shared memory structure naturally isolates scheduling data across different nodes, preventing cross-node data interference. Child processes use a private read-only mode to map the memory page containing the lock and read its state, without modifying the original memory data of the key card library. This ensures that the operational logic of third-party libraries remains unaffected and allows for non-intrusive acquisition of the lock's real-time contention status. Child processes combine their own NUMA node information with the real-time global lock state to perform layered avoidance, limiting most contention to the same node and reducing high-latency cache consistency traffic across nodes. Throughout the entire process, the parent process only performs pre-resolution and shared memory creation operations, without participating in any key card-related operations, fully complying with security constraints. Each step is seamlessly integrated, requiring no modification to the database process model or third-party library source code. This reduces the average latency of key card initialization and lock conflict rate in high-concurrency scenarios, improving system stability.

[0021] It should be noted that the core concept of this application is specifically aimed at the key card initialization requirements in high-concurrency fork scenarios of multi-process databases. Under the strict constraints of not modifying the key card library source code, not adding agent processes, and the parent process not holding key card sessions, the parent process pre-parses the global lock symbol of the PKCS#11 key card library, creates shared memory partitioned by NUMA nodes to store the virtual address of the global lock and the data related to the competition and scheduling of each node; the child process obtains the global lock state non-intrusively through private read-only memory mapping without modifying the original memory of the third-party library, and then performs a layered avoidance operation in combination with the real-time state of the global lock of its own NUMA node, and finally completes the key card initialization. Under the premise of fully complying with all constraints, it solves the problems of global lock conflict thundering herd and low efficiency of NUMA cross-node access.

[0022] The embodiments of this application will be described in detail below.

[0023] In this embodiment, the parent process is the database instance master process, responsible for initializing the database instance, deriving child processes, and managing system resources. The parent process does not perform any key card initialization operations, does not hold any key card sessions, and does not pass any key card-related context information; it only performs basic preparatory work related to key card initialization control. Shared memory is a memory sharing mechanism provided by the operating system, allowing multiple processes to map the same physical memory to their respective virtual address spaces to achieve data sharing. Shared memory divided by NUMA nodes refers to a shared memory structure organized according to the number of NUMA nodes on the server. Each NUMA node corresponds to an independent memory area used to store statistical data and scheduling information related to that node. The key card library global lock is a mutex lock used within the key card library to ensure concurrent safety of global operations. It is usually stored as a global variable in the library's data segment, and all processes loading the library share the same lock instance. The key card library symbol resolution interface is an interface provided by the operating system for obtaining the addresses of functions and variables in a dynamic library. Through this interface, the virtual address of the global lock can be obtained at runtime without modifying the library's source code.

[0024] Child processes are processes forked from the database parent process, including business backend processes, startup and recovery processes, WAL writing processes, etc. Each child process needs to independently execute key card initialization to establish a key card session and complete its own business processing. NUMA nodes are computing units in the non-uniform memory access architecture used in modern servers. Each NUMA node contains independent CPU, memory, and I / O controllers. The latency of a CPU accessing local memory within the same node is much lower than the latency of accessing memory on other nodes. Operating system native APIs are interfaces provided by the operating system kernel for obtaining process information. Through this interface, the NUMA node identifier of the currently running process can be obtained. Read-only memory mapping refers to mapping a block of physical memory to the virtual address space of a process in a read-only manner. The process can only read the contents of this memory region and cannot modify it. Any modification operation will trigger copy-on-write, without affecting the contents of the original memory. By obtaining the state of a global lock through read-only memory mapping, the current state of the lock can be read without modifying the original memory of the key card database, determining whether it is held by another process.

[0025] Layered avoidance operation refers to employing different avoidance strategies based on the contention state of the NUMA node where the child process resides and the global lock. Priority is given to ensuring the execution rights of processes within the same node, reducing contention between processes on different nodes. Simultaneously, the avoidance duration is dynamically adjusted according to the contention intensity to avoid the thundering herd effect. Key card initialization refers to the child process executing the C_Initialize function to complete operations such as global state initialization of the key card library, device enumeration, and slot scanning, preparing for subsequent key card session establishment and encryption operations.

[0026] In alternative implementations, such as Figure 2 As shown, the parent process creates shared memory partitioned by NUMA nodes and pre-parses the global lock symbols in the MAC library, including: S110: The parent process reads the total number of NUMA nodes on the server, creates an anonymous shared memory of the corresponding size, and sets the permissions of the shared memory to read-only for the child process and read, write, and shareable for the parent process. S120: The parent process loads the key card library and obtains the virtual address of the global lock through the key card library symbol resolution interface; S130: The parent process initializes the contention rate, ticket allocator, and current service ticket for each NUMA node to their initial values.

[0027] In this embodiment, the parent process can read the total number of NUMA nodes on the server using the NUMA-related APIs provided by the operating system. These APIs can return the number of available NUMA nodes on the server and the CPU and memory information for each node. Anonymous shared memory refers to shared memory that is not associated with any file; it exists only in memory and is automatically released after the process exits, leaving no data on the disk. The permissions for shared memory are set to read, write, and shareable, allowing the parent process to initialize the contents of the shared memory, and all child processes to read and modify the data in the shared memory. The parent process can load the symbol library using the symbol library loading interface provided by the operating system. This interface loads the dynamic library into the process's virtual address space and returns a handle for subsequent symbol resolution. The symbol resolution interface can obtain the virtual address corresponding to a symbol using the handle and symbol name of the dynamic library. For unexported static global lock symbols, the address of the symbol can be obtained by parsing the ELF file format of the dynamic library and traversing the symbol table. The parent process initializes the contention rate, ticket allocator, and current service ticket for each NUMA node to initial values. The initial value of the contention rate is 0, indicating that there is no contention. The initial value of the ticket allocator is 0, which is used to allocate a unique ticket number. The initial value of the current service ticket is 0, which indicates the ticket number that is currently being served.

[0028] It should be noted that after completing the shared memory creation and symbol resolution, the parent process immediately unloads the key card library and releases related resources, ensuring that no PKCS#11 key card library state information remains in the parent process's address space, thus avoiding context pollution during subsequent forking of child processes. The parent process only performs the above operations once at startup and does not participate in any key card-related operations during subsequent operation.

[0029] In alternative implementations, such as Figure 3As shown, the child process obtains the virtual address of its own NUMA node and the global lock, including: S210: The child process obtains the NUMA node identifier bound to the current process through the operating system's native API; S220: The child process reads the virtual address of the global lock from the head of the shared memory; S230: The child process maps the memory page containing the global lock into its own address space in a private read-only mode.

[0030] In this embodiment, the child process obtains the NUMA node identifier bound to the current process through the operating system's native API. This API can return the NUMA node number where the CPU currently running the process resides. The child process reads the virtual address of the global lock from the header of the shared memory. A fixed-size area is reserved at the header of the shared memory to store the address of the global lock. The parent process has already written this address during initialization, and the child process can read it directly without repeating symbol resolution. The child process maps the memory page containing the global lock into its own address space in a private read-only mode. The private read-only mode means that the mapped memory area can only be read by the current process. Any modification operation will trigger copy-on-write, generating a private copy that will not affect the memory content of other processes or the original library. In this way, it can be ensured that the child process's reading of the global lock state will not affect the normal operation of the PKCS#11 key card library, achieving non-intrusive lock state detection.

[0031] For example, the global lock is located at address 0x7f8a9b001234 in the PKCS#11 key library data segment, and the memory page at this address is 4KB in size. The child process calls the operating system's memory mapping interface to map the 4KB memory page starting at 0x7f8a9b001000 into its own virtual address space in private read-only mode. The mapped virtual address is 0x7f8c8d002000. The child process can read the global lock's status field by accessing address 0x7f8c8d002234. Any modification operation to this address will trigger copy-on-write, generating a private memory page for the child process without affecting the original library's memory content.

[0032] In alternative implementations, such as Figure 4 As shown, the state of the global lock is obtained through read-only memory mapping, including: S240: The child process reads the state field of the global lock using an atomic load operation with a memory barrier; S250: The child process determines whether the global lock is held based on the value of the status field; S260: The child process writes the detection result to the shared memory and updates the contention rate of the corresponding NUMA node.

[0033] In this embodiment, the atomic load operation with memory barriers refers to inserting memory barrier instructions when reading memory variables to prevent the CPU from reordering instructions and ensure that the read variable value is the latest. The global lock's status field is the field in a mutex lock used to indicate whether the lock is held. For the POSIX standard `pthread_mutex_t`, this field is usually an integer variable; a value of 0 indicates the lock is not held, and a non-zero value indicates the lock is held. The child process determines whether the global lock is held based on the value of the status field. If the value is 0, PKCS#11 initialization can be executed directly; if the value is non-zero, it indicates contention, and a layered avoidance operation needs to be performed. The child process writes the detection result to shared memory, updating the contention rate of the corresponding NUMA node. A detection result of 1 indicates contention, and 0 indicates no contention. The new contention rate is calculated using an exponential moving average algorithm to reflect the long-term contention intensity of the node.

[0034] Specifically, the atomic load operation uses the C11 standard `atomic_load_acquire` function, which inserts a memory barrier to ensure that all subsequent memory accesses are not reordered before this operation. This ensures that the global lock state read by the child process is up-to-date and will not lead to misjudgments due to CPU instruction reordering. For example, without the memory barrier, the CPU might reorder subsequent PKCS#11 initialization operations before the lock state is read, causing initialization to occur before the lock is released, resulting in data races and program crashes.

[0035] In alternative implementations, such as Figure 5 As shown, the child process performs a layered avoidance operation based on the state of the NUMA node and the global lock, including: S310: The child process reads the contention rate of the corresponding NUMA node and performs a hierarchical spin operation based on the contention rate; S320: The child process calculates the backoff base based on the contention rate and performs an adaptive sleep avoidance operation; S330: When the number of avoidance attempts reaches the threshold, the child process acquires execution rights through an atomic Ticket sequence and completes PKCS#11 initialization.

[0036] In this embodiment, the child process reads the contention rate of the corresponding NUMA node. The contention rate is stored in the shared memory region of the corresponding node and is a floating-point number between 0 and 1; a larger value indicates more intense contention. Layered spin operation refers to using different spin strategies based on the NUMA node the child process resides in. Processes on the same node spin for a short time, while processes on different nodes do not spin. Adaptive sleep avoidance operation refers to dynamically adjusting the sleep duration based on the contention intensity of the node and the current number of avoidance attempts; the more intense the contention, the more avoidance attempts, and the longer the sleep duration. Atomic Ticket sequence refers to each NUMA node maintaining a monotonically increasing Ticket allocator and the current service Ticket. Processes obtain Ticket sequence numbers through atomic operations and acquire execution rights sequentially, ensuring fairness. When the number of avoidance attempts reaches a threshold, the child process acquires execution rights through the atomic Ticket sequence and completes PKCS#11 initialization. The threshold can be set according to actual conditions to avoid long-term process waiting leading to starvation.

[0037] It should be noted that the layered avoidance operation design fully considers the characteristics of the NUMA architecture and the time consumption of PKCS#11 initialization. The average time of PKCS#11 initialization is about 100 microseconds, and the latency of processes on the same node accessing the global lock is about 50 nanoseconds. The overhead of short-term spinning is much lower than the context switching overhead of sleep / wake-up (about 10 microseconds). However, the latency of processes on different nodes accessing the global lock is about 200 nanoseconds. Spinning will only increase CPU usage and cache consistency traffic without any benefit, so it directly enters sleep avoidance mode.

[0038] In the selected implementation method, such as Figure 6 As shown, hierarchical spin operations are performed according to the competition rate, including: S311: If the child process is on the same NUMA node as the global lock, the child process performs a spin that is inversely proportional to the contention rate. The spin interval is fixed, and the state of the global lock is checked repeatedly during the spin. S312: If the child process is on a different NUMA node than the global lock, the child process does not perform spin and directly enters the adaptive sleep avoidance operation.

[0039] In this embodiment, the child process and the global lock residing on the same NUMA node means that the NUMA node where the child process runs and the NUMA node where the global lock is stored are the same node. In this case, the latency of the child process accessing the global lock is low, and the overhead of short-term spinning is lower than the context switching overhead of sleep / wake-up. The child process performs spinning inversely proportional to the contention rate; the higher the contention rate, the fewer the number of spins, avoiding excessive CPU usage caused by a large number of processes spinning simultaneously. Each spin interval is fixed, which can be set to 1 microsecond. During this interval, the state of the global lock is repeatedly checked. If the lock is detected to be released, spinning stops immediately, and PKCS#11 initialization is executed. If the child process and the global lock reside on different NUMA nodes, the latency of the child process accessing the global lock is high. Spinning will only increase CPU usage and cache consistency traffic, without any benefit. Therefore, an adaptive sleep avoidance operation is directly entered.

[0040] Specifically, the formula for calculating the number of spins is: Number of spins = Maximum number of spins - Contention rate × Maximum number of spins. The maximum number of spins can be set to 10. When the contention rate is 0, the number of spins is 10; when the contention rate is 0.5, the number of spins is 5; and when the contention rate is 1, the number of spins is 0. This design can fully utilize the low latency advantage of spins when the contention is low, and reduce invalid CPU usage when the contention is high.

[0041] In the selected implementation method, such as Figure 7 As shown, the adaptive sleep avoidance operation includes: S321: The child process determines the backoff base based on the contention rate of the corresponding NUMA node. The higher the contention rate, the larger the backoff base. S322: The subprocess calculates the exponential backoff time based on the current number of avoidances and adds random jitter to scatter the wake-up time; S323: The child process adjusts its sleep duration based on the differences between NUMA nodes. The sleep duration of a cross-node process is a preset multiple of that of a process on the same node.

[0042] In this embodiment, the subprocess determines the backoff base based on the contention rate of the corresponding NUMA node. The backoff base is the fundamental value for calculating sleep duration; the higher the contention rate, the larger the backoff base and the longer the backoff time. For example, when the contention rate is below 0.4, the backoff base is set to 10 microseconds; when the contention rate is between 0.4 and 0.7, the backoff base is set to 20 microseconds; and when the contention rate is above 0.7, the backoff base is set to 50 microseconds. The subprocess calculates the exponential backoff duration based on the current number of backoffs. Exponential backoff means that the sleep duration increases exponentially with the number of backoffs. For example, when the number of backoffs is 1, the duration is the backoff base multiplied by 2 to the power of 1; when the number of backoffs is 2, the duration is the backoff base multiplied by 2 to the power of 2, and so on. Random jitter is added to disperse the wake-up time. The range of random jitter can be set to ±20% of the backoff base to avoid secondary thundering herd caused by multiple processes sleeping and waking up simultaneously. The child process adjusts its sleep duration based on the differences between NUMA nodes. The sleep duration of the cross-node process is a preset multiple of that of the same-node process. The preset multiple can be set to 3 to prioritize the execution rights of the same-node process and reduce the impact of high-latency cross-node access on overall performance.

[0043] For example, a child process runs on node 1, and the global lock is stored on node 0, making it a cross-node process. The current contention rate on node 1 is 0.6, the backoff base is 20 microseconds, and the current backoff count is 2. The exponential backoff duration is 20 × 2^2 = 80 microseconds. Adding ±20% random jitter, we get 88 microseconds. The sleep duration of the cross-node process is multiplied by 3, resulting in a final sleep time of 264 microseconds. After the sleep ends, the child process checks the state of the global lock again. If the lock has been released, it executes PKCS#11 initialization; if it is still held, the backoff count is incremented by 1, the sleep duration is recalculated, and the backoff continues.

[0044] In the selected implementation method, such as Figure 8 As shown, obtaining execution rights through an atomic Ticket sequence includes: S331: The child process atomically obtains a unique Ticket sequence number from the Ticket allocator of the corresponding NUMA node; S332: The child process enters a waiting state until the current service ticket of the corresponding NUMA node is equal to its own ticket number; S333: After the child process completes the key card initialization, it atomically increments the current service ticket of the corresponding NUMA node.

[0045] In this embodiment, the child process atomically obtains a unique Ticket sequence number from the Ticket allocator of the corresponding NUMA node. This atomic acquisition operation ensures that when multiple processes acquire Tickets simultaneously, there will be no duplication or omission. The child process enters a waiting state until the current service Ticket of the corresponding NUMA node equals its own Ticket sequence number. During the waiting period, the value of the current service Ticket can be checked at fixed intervals to avoid CPU idle time. After completing PKCS#11 initialization, the child process atomically increments the current service Ticket of the corresponding NUMA node, waking up the next waiting process. In this way, a first-in-first-out queuing mechanism can be implemented, ensuring that all processes can eventually acquire the lock and avoiding process starvation.

[0046] Specifically, the `atomic_fetch_add` function is used to atomically retrieve the Ticket sequence number. This function atomically increments the value of the Ticket allocator by 1 and returns the value before the increment. For example, if the current Ticket allocator value is 5, calling `atomic_fetch_add` returns 5, and the Ticket allocator value becomes 6. After the child process obtains the Ticket sequence number 5, it enters a waiting state, checking the value of the current service Ticket every 50 microseconds. When the current service Ticket value becomes 5, the child process acquires execution rights and performs PKCS#11 initialization. After completion, it calls the `atomic_fetch_add` function to increment the current service Ticket value by 1, making it 6, and wakes up the next waiting process.

[0047] In the selected implementation method, such as Figure 9 As shown, update the contention rate of the corresponding NUMA node, including: S261: The child process obtains the historical contention rate of the corresponding NUMA node; S262: The subprocess calculates a new contention rate using an exponential moving average algorithm, which uses a preset smoothing coefficient; S263: The child process updates the contention rate in shared memory through atomic comparison-swap operations.

[0048] In this embodiment, the subprocess obtains the historical contention rate of the corresponding NUMA node, which is stored in the corresponding node's region of shared memory. The subprocess calculates the new contention rate using the exponential moving average algorithm, a commonly used time series smoothing algorithm that can smooth out sudden fluctuations and reflect the long-term trend of data. The smoothing coefficient controls the weight of historical and current data; a larger smoothing coefficient results in a higher weight for current data and a faster response to changes in the algorithm; a smaller smoothing coefficient results in a higher weight for historical data and a more stable algorithm. The smoothing coefficient can be set to 0.25 to balance response speed and stability. The subprocess updates the contention rate in shared memory using atomic comparison-swap operations. Atomic comparison-swap operations ensure that data inconsistency does not occur when multiple processes update the contention rate simultaneously.

[0049] Specifically, the calculation formula for the exponential moving average algorithm is: New contention rate = Smoothing coefficient × Current detection result + (1 - Smoothing coefficient) × Historical contention rate. For example, if the historical contention rate is 0.2, the current detection result is 1, and the smoothing coefficient is 0.25, then the new contention rate is 0.25 × 1 + 0.75 × 0.2 = 0.4. The atomic comparison exchange operation uses the `atomic_compare_exchange_weak` function. This function first compares whether the contention rate in shared memory is equal to the historical contention rate. If they are equal, it updates the historical contention rate to the new contention rate; if they are not equal, it means that other processes have already updated the contention rate, and the historical contention rate needs to be read again and the new contention rate calculated until the update is successful. The specific values ​​in the above examples can be flexibly set by those skilled in the art according to actual needs in practical applications, as long as the inventive purpose of this application is achieved. This application does not limit this.

[0050] This application also provides a NUMA-aware concurrent access control system for cryptographic cards, such as... Figure 10 As shown, it includes: The parent process initialization module 01 is used to create shared memory divided by NUMA nodes and pre-parse the global lock symbol of the cipher library. The virtual address of the global lock is stored in the header of the shared memory. The shared memory is used to store the contention rate, ticket allocator and current service ticket corresponding to each NUMA node. The lock contention detection module 02 is used to enable the child process to obtain the virtual address of its own NUMA node and the global lock, and to obtain the state of the global lock through read-only memory mapping, wherein the read-only memory mapping is a mapping method that only reads and does not modify the original memory of the key library; The layered avoidance execution module 03 is used to enable the subprocess to perform layered avoidance operations based on the state of the NUMA node and the global lock, and complete the key card initialization.

[0051] In this embodiment, the parent process initialization module runs in the database parent process and is responsible for creating shared memory and resolving global lock symbols, providing basic support for the key card initialization control of child processes. The lock contention detection module runs in each database child process and is responsible for obtaining the NUMA node information and global lock status of the child process, providing a basis for hierarchical avoidance. The hierarchical avoidance execution module runs in each database child process and is responsible for executing the corresponding avoidance strategy based on the NUMA node and lock contention status, completing key card initialization. All modules interact with each other through shared memory, eliminating the need for additional inter-process communication and ensuring the system's lightweight and high efficiency.

[0052] It should be noted that all modules of this system are integrated into the database as libraries, requiring no modification to the core database kernel code. Only corresponding calls need to be inserted during database initialization and subprocess startup. This design ensures the system's portability and ease of integration, enabling rapid adaptation to different database versions and PKCS#11 key card libraries from different vendors.

[0053] For example, a database instance is deployed on a 2-way NUMA architecture server. The server has two NUMA nodes, each containing 16 CPU cores and 64GB of memory. The database enables transparent data encryption, using a hardware key card to store the master key. Each business backend process needs to independently execute PKCS#11 initialization to establish a key card session. During peak business periods, the database forks 50 business backend processes per second to process user business requests.

[0054] Specifically, when the database parent process starts, it first reads that the total number of NUMA nodes on the server is 2, and creates an anonymous shared memory block of 24 bytes. The structure of the shared memory is as follows: the first 4 bytes store the virtual address of the global lock, the next 4 bytes store the contention rate of node 0, the next 4 bytes store the ticket allocator of node 0, the next 4 bytes store the current service ticket of node 0, then 4 bytes store the contention rate of node 1, 4 bytes store the ticket allocator of node 1, and 4 bytes store the current service ticket of node 1. The parent process sets the permissions of the shared memory to read, write, and shareable, and initializes all fields to 0. Then, the parent process loads the key card library, obtains the virtual address of the global lock (0x7f8a9b001234) through the key card library symbol resolution interface, and writes this address into the first 4 bytes of the shared memory. After completing these preparations, the parent process unloads the key card library, releases related resources, and begins to spawn child processes to handle business requests.

[0055] The first child process is forked by the parent process and runs on NUMA node 0. After starting, the child process first obtains its own NUMA node identifier (0) using the operating system's native API. Then, it reads the virtual address of the global lock (0x7f8a9b001234) from the first 4 bytes of shared memory. The child process maps the memory page containing the global lock to its own virtual address space in private read-only mode. It then uses an atomic load operation with memory barriers to read the global lock's status field, which is 0, indicating that the lock is not held. The child process writes the result (0) to shared memory, calculates the new contention rate for node 0 using an exponential moving average algorithm (0.25×0 + 0.75×0 = 0), and updates the contention rate of node 0 in shared memory to 0 using an atomic compare-swap operation. Finally, the child process directly executes PKCS#11 initialization and begins processing business requests after completion.

[0056] Subsequently, the parent process forks 49 child processes, with 25 running on node 0 and 24 on node 1. The second child process runs on node 0. Upon startup, it obtains its own node identifier as 0, reads the global lock address, maps the memory page, and atomically reads the lock status field as 1, indicating that the first child process holds the lock. The child process writes the detection result 1 to shared memory, calculates the new contention rate for node 0 as 0.25 × 1 + 0.75 × 0 = 0.25, and updates the contention rate in shared memory. The child process reads the contention rate of node 0 as 0.25 and performs a spin inversely proportional to the contention rate, spinning 10 - 0.25 × 8 = 8 times, with a 1-microsecond interval between each spin. On the third spin, the first child process releases the lock, detects the lock status as 0, stops spinning, and executes PKCS#11 initialization. After completion, it writes the detection result 0 to shared memory and updates the contention rate to 0.25 × 0 + 0.75 × 0.25 = 0.1875.

[0057] The third child process runs on node 1. After starting, it obtains its own node identifier as 1, reads the global lock address, maps the memory page, and atomically reads the lock status field value as 1, indicating that the second child process holds the lock. The child process writes the detection result 1 to shared memory, calculates the new contention rate of node 1 as 0.25×1+0.75×0=0.25, and updates the contention rate in shared memory. Since the child process is on a different node than the global lock, it does not perform spin and directly enters the adaptive sleep avoidance operation. The child process reads the contention rate of node 1 as 0.25, determines the backoff base as 10 microseconds, the current avoidance count is 1, calculates the exponential backoff duration as 10×2^1=20 microseconds, adds ±20% random jitter, and gets 22 microseconds. The sleep duration of the cross-node process is multiplied by 3, resulting in a final sleep of 66 microseconds. After the sleep ends, the child process checks the lock status again. This time, the lock has been released, and PKCS#11 initialization is executed. After completion, the detection result 0 is written to the shared memory, and the contention rate of node 1 is updated to 0.25×0+0.75×0.25=0.1875.

[0058] As business traffic increases, the contention rate of node 0 gradually rises to 0.8. At this point, the read contention rate of the child process running on node 0 is 0.8, and the number of spins executed is 10 - 0.8 × 8 = 3.6, rounded down to 3. If the lock is not acquired after 3 spins, an adaptive sleep avoidance operation is entered. The backoff base is 50 microseconds, the number of avoidances is 1, and the sleep duration is 50 × 2^1 + random jitter = 105 microseconds. If the lock is still not acquired after 10 consecutive avoidances, the child process atomically obtains the Ticket sequence number 5 from the Ticket allocator of node 0, enters a waiting state, and checks the value of the current service Ticket every 50 microseconds. When the current service Ticket of node 0 is incremented to 5, the child process acquires execution rights and executes PKCS#11 initialization. After completion, the current service Ticket is atomically incremented to 6, waking up the next waiting process.

[0059] As business traffic gradually stabilized, the contention rate between node 0 and node 1 gradually decreased to below 0.2. The spin count of child processes returned to 8, the backoff base returned to 10 microseconds, the average PKCS#11 initialization latency of all child processes stabilized at around 120 microseconds, the lock contention rate decreased to below 5%, and no process initialization timeouts occurred, resulting in stable database operation. The specific values ​​mentioned above can be flexibly set by those skilled in the art according to actual needs in practical applications, as long as the inventive objective of this application is achieved; this application does not impose any limitations on this.

[0060] Those skilled in the art will understand that the numerical values ​​in the embodiments of this application are merely examples and do not constitute a limitation of this application. Those skilled in the art can adjust parameters such as the number of spins, the backoff base, the smoothing coefficient, and the maximum number of avoidances according to actual circumstances to achieve optimal performance. Furthermore, the solution of this application is not only applicable to transparent database encryption scenarios, but also to other multi-process application scenarios requiring high-concurrency execution of encryption card initialization, such as distributed storage and cloud computing platforms.

[0061] It should be noted that when the key card library does not export global lock symbols, this solution can automatically degrade to an adaptive avoidance mechanism based on initialization time statistics. The child process records the execution time of the C_Initialize function each time. If the execution time exceeds a preset threshold, it indicates that there is contention, and an avoidance operation is performed. This degradation mechanism ensures that this solution can run normally on key card libraries from different vendors, exhibiting good compatibility. When the server is a non-NUMA architecture, this solution can merge the statistical data of all nodes, degrading to an adaptive avoidance mechanism under a general architecture, ensuring the portability of the solution. When abnormal situations such as key card failure or initialization failure occur, the child process will automatically clean up resources and update the contention rate of the corresponding node, without affecting the normal operation of other processes.

[0062] Furthermore, this solution can be expanded to support scenarios with multiple hardware security cards. When multiple hardware security cards are installed on the server, the parent process can create an independent shared memory region for each card, and the child process can use the corresponding shared memory for contention detection and avoidance operations based on the configured security card index. This expansion can further improve the system's concurrent processing capabilities and meet the needs of high-load scenarios.

[0063] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0064] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A NUMA-aware concurrent access control method for cryptographic cards, characterized in that, include: The parent process creates shared memory partitioned by NUMA nodes and pre-parses the global lock symbol of the MAC library. It stores the virtual address of the global lock in the header of the shared memory. The shared memory is used to store the contention rate, ticket allocator and current service ticket for each NUMA node. The child process obtains the virtual address of its own NUMA node and the global lock, and obtains the state of the global lock through read-only memory mapping, wherein the read-only memory mapping is a mapping method that only reads and does not modify the original memory of the key library; The child process performs a hierarchical avoidance operation based on the state of the NUMA node and the global lock to complete the key card initialization.

2. The method according to claim 1, characterized in that, The parent process creates shared memory partitioned by NUMA nodes and pre-parses global lock symbols from the MAC library, including: The parent process reads the total number of NUMA nodes on the server, creates an anonymous shared memory of the corresponding size, and sets the permissions of the shared memory to read-only for the child process and read, write, and shareable for the parent process. The parent process loads the key card library and obtains the virtual address of the global lock through the key card library symbol resolution interface; The parent process initializes the contention rate, ticket allocator, and current service ticket for each NUMA node to their initial values.

3. The method according to claim 1, characterized in that, The child process obtains the virtual address of its own NUMA node and the global lock, including: The child process obtains the NUMA node identifier bound to the current process through the operating system's native API; The child process reads the virtual address of the global lock from the header of the shared memory; The child process maps the memory page containing the global lock into its own address space in a private read-only mode.

4. The method according to claim 1, characterized in that, The state of the global lock is obtained through read-only memory mapping, including: The child process uses an atomic load operation with a memory barrier to read the state field of the global lock; The child process determines whether the global lock is held based on the value of the status field; The child process writes the detection results to shared memory and updates the contention rate of the corresponding NUMA node.

5. The method according to claim 1, characterized in that, The child process performs a layered avoidance operation based on the state of the NUMA node and the global lock, including: The child process reads the contention rate of the corresponding NUMA node and performs hierarchical spin operations based on the contention rate; The child process calculates the backoff base based on the contention rate and performs an adaptive sleep avoidance operation; When the number of avoidance attempts reaches the threshold, the child process acquires execution rights through an atomic Ticket sequence and completes the key card initialization.

6. The method according to claim 5, characterized in that, Performing hierarchical spin operations based on the competition rate includes: If the child process is on the same NUMA node as the global lock, the child process performs a spin that is inversely proportional to the contention rate, with a fixed spin interval, during which the state of the global lock is repeatedly checked. If the child process is on a different NUMA node than the global lock, the child process will not perform a spin operation and will directly enter an adaptive sleep avoidance operation.

7. The method according to claim 5, characterized in that, Perform adaptive sleep avoidance operations, including: The child process determines the backoff base based on the contention rate of the corresponding NUMA node; The subprocess calculates the exponential backoff time based on the current number of avoidances, and adds random jitter to scatter the wake-up time; The child process adjusts its sleep duration based on the differences between NUMA nodes, and the sleep duration of the cross-node process is a preset multiple of that of the process on the same node.

8. The method according to claim 5, characterized in that, Acquiring execution rights through an atomic ticket sequence includes: The child process atomically obtains a unique Ticket sequence number from the Ticket allocator of the corresponding NUMA node; The child process enters a waiting state until the current service ticket of the corresponding NUMA node equals its own ticket number; After the child process completes the key card initialization, it atomically increments the current service ticket of the corresponding NUMA node.

9. The method according to claim 4, characterized in that, Update the contention rate of the corresponding NUMA node, including: The child process retrieves the historical contention rate of the corresponding NUMA node; The subprocess calculates a new contention rate using an exponential moving average algorithm that employs a preset smoothing coefficient. The child process updates the contention rate in shared memory through atomic comparison-swap operations.

10. A NUMA-aware concurrent access control system for cryptographic cards, characterized in that, include: The parent process initialization module is used to create shared memory partitioned by NUMA nodes and pre-parse the global lock symbols of the MAC library. The virtual address of the global lock is stored in the header of the shared memory. The shared memory is used to store the contention rate, ticket allocator and current service ticket corresponding to each NUMA node. The lock contention detection module is used to enable the child process to obtain the virtual address of its own NUMA node and the global lock, and to obtain the state of the global lock through read-only memory mapping, wherein the read-only memory mapping is a mapping method that only reads and does not modify the original memory of the key library; The hierarchical avoidance execution module is used to enable the child process to perform hierarchical avoidance operations based on the state of the NUMA node and the global lock, and complete the key card initialization.

Images