Method for efficiently reading and writing NVMe hardware mass data

Abstract

The invention provides a method for efficiently reading and writing NVMe (Non-Volatile Memory Express) hardware mass data in order to solve the problems of large data volume, high real-time requirement and high storage cost in the mass data acquisition and storage process of the Internet of Things in the prior art. Comprising the following steps: creating a special channel for a giant page annular lock-free queue, directly storing received data into the giant page annular lock-free queue by a data acquisition program, and then immediately receiving new data; a storage program directly reads data from a giant page annular lock-free queue and stores the data into a hard disk, CPU polling is carried out on the NVMe hard disk in a user mode driving program, a plurality of threads are created on a multi-core device, and each thread is bound to an independent core. According to the method, the rapid read-write capability of mass network data is improved through technologies such as a giant page annular lock-free queue, the problem of performance bottleneck faced by a traditional system is solved, the real-time performance and accuracy of data acquisition are improved, and the storage cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of Internet of Things mass data collection, storage and processing, and in particular to a method for efficiently reading and writing NVMe hardware mass data. Background Art

[0002] With the rapid development of hard disk and flash memory technology, NVME-SSD has gradually entered the core field of storage. With the emergence of NVME-SSD, SSD storage media is replacing mechanical hard disks in traditional data centers. The commercially available NVMe-based hard disks can easily reach a single-disk GB-level read and write bandwidth and a random IOPS of 100,000, which is 5 to 10 times that of SATA solid-state drives. However, due to the limitations of the implementation method and scheduling mechanism of the operating system kernel driver, the performance of general storage software, compared with NVMe, consumes too much time in the entire IO transaction, wasting hard disk performance.

[0003] In order to solve the above problems, the Chinese invention patent with patent number CN112817530A proposes a multi-threaded safe and efficient method for reading and writing ordered data, including the following steps: using a read thread to implement the read operation of the data stored in the container, and using a write thread different from the read thread to implement the write operation of the data stored in the same container, and performing the read operation and write operation on the container at the same time. The present invention is aimed at the situation in the field of data processing where various sortings are performed on the data stored in a container to perform data analysis, and designs a multi-threaded safe and efficient method for reading and writing ordered data. However, frequent interactions between multiple threads and imperfect strategies for sending and receiving massive messages will lead to reduced performance. Summary of the invention

[0004] The present invention mainly solves the problems of large data volume, high real-time requirements and high storage cost faced by the Internet of Things in the process of data collection and storage in the prior art, and provides a method for efficiently reading and writing massive data of NVMe hardware.

[0005] The above technical problems of the present invention are mainly solved by the following technical solutions: A method for efficiently reading and writing massive data on NVMe hardware includes the following steps: S1. Use the user-mode driver framework to create a huge page ring lock-free queue dedicated channel for NVMe hard disk data transmission; S2, the data collection program directly stores the received data into the huge page ring lock-free queue, and then immediately receives new data. The storage program directly reads the data from the huge page ring lock-free queue and stores it into the hard disk; S3, CPU polling of the NVMe hard disk periodically executed in the user-mode driver; S4. Take advantage of the operating system's support for thread affinity and create multiple threads on a multi-core device, with each thread bound to a separate core.

[0006] As a preferred solution, the dedicated channel does not pass through the operating system kernel.

[0007] As a preferred solution, the user-mode driver framework includes UIO, and the UIO includes a user-space driver and a kernel-space driver.

[0008] As a preferred solution, in step S3, the kernel space driver shields the interrupt signal sent by the hardware, and the user space driver acquires data by actively polling.

[0009] As a preferred solution, step S1 specifically includes the following steps: S1.1. Load the UIO driver, obtain the UIO name of the NVMe device under / sys / class / uio, and set the memory mapping address, size, and type; S1.2. Call the uio_register_device function to register and allocate the UIO device in the kernel space. If the registration is successful, the device mapping details will be generated under / sys / class / uio / uioX / maps / mapY / ; S1.3. Call the open function in user space to open the device and create a huge page ring lock-free queue; S1.4. Call the mmap function to map the device memory to the address space of the user space. After the mapping is successful, the address can be directly used to read and write the device memory.

[0010] As a preferred solution, in step S2, the storing does not need to wait for the storage program to read the status. As a preferred solution, in step S2, the reading does not need to wait for the acquisition program to write the status.

[0011] As a preferred solution, in step S3, once an IO operation is queried, the callback function is immediately triggered.

[0012] Therefore, the advantages of the present invention are: 1. Use user-mode driver boxes such as UIO to bypass the kernel and directly operate hardware resources in user space, avoiding a large amount of hard disk idleness and waste of hard disk performance due to processing overhead such as kernel context switching, interrupts, and data copying.

[0013] 2. Directly reading hardware data in user space through continuous polling avoids the time loss caused by context switching caused by hardware interrupts.

[0014] 3. Take advantage of the operating system's support for thread affinity and reduce thread scheduling overhead and save CPU scheduling time by binding each thread to a separate core.

[0015] 4. The huge page ring lock-free queue method is used to enable producers and consumers to access the buffer at the same time, improving data access speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a principle block diagram of the present invention.

[0017] Figure 2 It is a user space driving principle diagram of the present invention.

[0018] Figure 3 It is a huge page ring lock-free queue diagram of the present invention.

[0019] Figure 4 It is a user space polling principle diagram of the present invention.

[0020] Figure 5 It is the CPU affinity principle diagram of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0022] Example: With the rapid development of hard disk and flash memory technology, NVME-SSD has gradually entered the core field of storage. With the emergence of NVME-SSD, SSD storage media is replacing mechanical hard disks in traditional data centers. The commercially available NVMe-based hard disks can easily reach a single-disk GB-level read and write bandwidth and a random IOPS of 100,000, which is 5 to 10 times that of SATA solid-state drives. However, due to the limitations of the implementation method and scheduling mechanism of the operating system kernel driver, the performance of general storage software, compared with NVMe, consumes too much time in the entire IO transaction, wasting hard disk performance.

[0023] From the perspective of application, the processing performance of the system often decreases rapidly with the increase of processing traffic and the increase of processing flow. There are many reasons for the performance degradation. From the perspective of system call and thread switching overhead, the reasons include the following: 1. Multithreaded processing performance is not linearly related to the number of threads. At the same time, the overhead caused by high-frequency switching between threads will lead to reduced processing performance; 2. Hardware interrupt requests will preempt software interrupts with lower priority. Frequently arriving hardware interrupts and soft interrupts mean frequent thread switching, which will lead to a series of CPU performance losses such as running mode switching, context switching, thread scheduler load, cache miss, multi-core cache shared data synchronization, and contention locks; 3. The device driver is located in the kernel state. After receiving the data packet, the device driver will be processed by the kernel protocol stack and then copied to the application layer buffer in the user state. The data copy from the kernel state to the user state is a time-consuming operation. The data copy time will account for more than 50% of the data packet processing process time; 4. A data packet may be interrupted in CPU0, processed in the kernel state in CPU1, and processed in the user state in CPU2. Processing across multiple physical cores (Core) will cause a large number of CPUCache hit misses, resulting in local failure. For NUMA architecture, memory access across NUMA nodes will also occur, which greatly affects CPU performance; 5. Most traditional servers use paged virtual memory, and the memory page defaults to a small 4K page. There will be a large number of page mapping items on servers with large storage space. At the same time, due to the limited TLB cache space, the mapping items of the TLB fast table will eventually change frequently, resulting in a large number of TLB misses.

[0024] Existing solutions generally use DPDK technology to improve the real-time processing capabilities of network devices such as switches, routers, and firewalls, and the data collection capabilities have been significantly improved. However, the traditional IO technology is still used on the data storage side, resulting in data storage efficiency being far lower than the data collection efficiency, resulting in incomplete data. In order to improve data storage efficiency, distributed storage technology is generally used. The distributed storage system makes the system architecture complex and difficult to maintain, which increases operating costs.

[0025] In view of the above situation, the present invention designs a method for efficiently reading and writing massive data of NVMe hardware, aiming to solve the performance bottleneck faced by traditional systems when processing massive data, especially in a high-concurrency environment. How to efficiently read and write data of NVMe hard disks has become an important challenge in the current data storage field. With the rapid development of NVMe hard disk technology, its read and write speed and I / O performance have far exceeded traditional mechanical hard disks and solid-state hard disks based on the SATA protocol. However, due to the scheduling mechanism, context switching, system call overhead and other problems existing in the operating system kernel when processing I / O transactions, the high performance of the hardware cannot be fully utilized, and the overall performance of the system is limited. Therefore, the present invention proposes an efficient method based on user-mode driver framework, user-mode polling mechanism, CPU affinity, huge page ring lock-free queue and other technologies to improve the read and write performance of NVMe hard disks when processing massive network data, including the following steps: S1. Use the user-mode driver framework to create a huge page ring lock-free queue dedicated channel for NVMe hard disk data transmission. The dedicated channel does not pass through the operating system kernel, and enables direct data interaction between the user space and the hardware, avoiding a large amount of hard disk idleness caused by kernel context switching, interrupts, and data copying. Processing overhead, waste of hard disk performance, allowing more clock cycles to be used for actual data storage.

[0026] The user-mode driver framework includes UIO, which includes user-space drivers and kernel-space drivers. The kernel-space UIO core code is responsible for device registration, memory mapping, interrupt management and other operations. The user-space driver is mainly responsible for data reading and writing. The kernel-space driver shields the interrupt signal sent by the hardware, and the user-space driver uses active polling to obtain data.

[0027] Step S1 specifically includes the following steps: S1.1. Load the UIO driver, obtain the UIO name of the NVMe device under / sys / class / uio, and set the memory mapping address, size, and type.

[0028] S1.2. Call the uio_register_device function to register and allocate the UIO device in the kernel space. If the registration is successful, the device mapping details will be generated under / sys / class / uio / uioX / maps / mapY / .

[0029] S1.3. Call the open function in user space to open the device and create a huge page ring lock-free queue.

[0030] S1.4. Call the mmap function to map the device memory to the address space of the user space. After the mapping is successful, the address can be directly used to read and write the device memory.

[0031] The UIO core code in the kernel space is responsible for a series of important device-related operations, including device registration, memory mapping, and interrupt management. Device registration refers to exposing the corresponding hardware device interface to user space programs in the system to ensure that user-mode programs can identify and access these devices. The memory mapping function allows user-mode programs to directly access the memory resources of hardware devices without going through cumbersome system calls, which can greatly reduce data transmission delays. In terms of interrupt management, the UIO core code in the kernel space is also responsible for shielding interrupt signals issued by hardware to prevent frequent interruptions from interrupting system operation and thus reducing system performance.

[0032] Unlike traditional kernel drivers, the user space driver of UIO undertakes most of the specific data reading and writing processing work. Compared with kernel state drivers, the advantage of user space drivers is that they do not need to frequently switch between user state and kernel state, thus reducing the overhead of context switching. This architecture enables user space programs to interact with hardware more efficiently, especially when processing a large number of I / O operations, which can significantly improve performance. The user space driver in the UIO framework uses active polling to obtain data instead of relying on hardware interrupts for data transmission. The kernel space shields the hardware interrupt signal, which means that when the hardware device needs to transmit data, the traditional interrupt mechanism will not be triggered, and the system does not need to consume resources to respond to the interrupt. On the contrary, the user space driver regularly checks the hardware status through active polling to determine whether there is new data to read or write. The advantage of active polling is that it can avoid the system overhead caused by interrupts, while maintaining relatively stable performance in high-load scenarios. Through this mechanism, system resources are more effectively utilized, especially when processing a large number of I / O requests, which can significantly improve the overall throughput and efficiency.

[0033] In traditional I / O operations, data transmission usually needs to pass through the operating system kernel, which means that each data transmission needs to switch between kernel state and user state, involving system calls, context switching, interrupt processing, etc. These operations will significantly increase the system overhead, especially in high concurrency and multi-threaded scenarios. Frequent switching not only consumes a large number of CPU clock cycles, but also causes the hard disk to be idle for a long time, and the high performance of the hard disk cannot be fully utilized. The present invention uses a user state driver framework to enable user space to interact directly with hardware, avoiding the frequent switching between traditional kernel state and user state, thereby greatly reducing the overhead of system calls and context switching. In this way, more CPU clock cycles can be actually used for data storage and processing, improving the performance of the overall system.

[0034] The present invention utilizes a user-mode polling mechanism to further improve the efficiency of I / O operations. In the traditional interrupt-driven I / O model, whenever an I / O event occurs, the hardware will send an interrupt signal to notify the operating system to handle the event. Although this mechanism can ensure the responsiveness of the system, in the case of high concurrency, the frequent triggering of interrupts will cause the system performance to degrade, especially when a large number of I / O requests are concentrated in a short time window, the interrupt storm may exhaust system resources and greatly affect the overall performance of the system. The user-mode polling mechanism bypasses the overhead of interrupt processing by actively querying the hardware status in the user space, ensuring the efficient execution of I / O operations. The polling mechanism can better control the timing of I / O operations, avoid the uncertainty and additional overhead caused by interrupts, and thus improve the performance of the system when processing massive data.

[0035] In order to further improve the performance of the system, the present invention also introduces CPU affinity technology. CPU affinity refers to binding a specific task to a specific CPU core for execution, so as to reduce the performance loss caused by the migration of tasks between different CPU cores. In a multi-core system, frequent migration of tasks between different cores will lead to problems such as cache invalidation and increased cache consistency overhead, thereby affecting the overall performance of the system. By reasonably setting CPU affinity, it can be ensured that a specific thread or task is always executed on the same CPU core, avoiding unnecessary thread migration and cache invalidation problems, and further improving the efficiency of I / O operations.

[0036] In addition, the present invention also designs a giant page-based circular lock-free queue for efficient data transmission. In traditional I / O operations, data transmission usually needs to pass through the buffer of the operating system, involving multiple data copy operations, which not only increases the delay of data transmission, but also consumes a large amount of system resources. The present invention uses the giant page circular lock-free queue technology to design the buffer of data transmission into a lock-free circular queue, avoiding the overhead of multiple data copies and lock competition. The giant page technology reduces the overhead of memory management by merging multiple small pages into a large memory page, while improving the access efficiency of memory. The circular lock-free queue realizes efficient data transmission between multiple threads in a lock-free manner, avoiding the performance bottleneck caused by thread competition in the traditional lock mechanism. Through this design, the system can transmit and process data with higher efficiency, further improving the read and write performance of the hard disk.

[0037] Through the combination of the above-mentioned multiple technologies, the present invention can significantly improve the read and write performance of NVMe hard disks when processing massive network data, and solve the performance bottleneck problem of traditional systems in high concurrency and large data volume scenarios. Due to the introduction of the user-state driver framework, the system no longer needs to frequently switch between kernel state and user state, reducing the overhead of system calls and context switching. The user-state polling mechanism avoids the uncertainty and waste of system resources caused by interrupt processing, and ensures the efficient execution of I / O operations. CPU affinity technology reduces the impact of cache invalidation and cache consistency problems on performance by reducing the migration of tasks between different cores. The huge page ring lock-free queue improves the efficiency of data transmission by reducing the overhead of data copying and lock competition.

[0038] In summary, through the combination of multiple technologies such as user-mode driver framework, user-mode polling mechanism, CPU affinity, huge page ring lock-free queue, etc., the system's read and write performance when processing massive network data can be significantly improved, solving the performance bottleneck problem of traditional systems in high-concurrency environments. By reducing the switching between kernel mode and user mode, reducing the overhead of interrupt processing, optimizing the utilization of CPU resources, and improving the efficiency of data transmission, the present invention not only improves the real-time and accuracy of data acquisition, but also effectively reduces storage costs, providing an efficient and scalable solution for the design of high-performance storage systems.

[0039] S2. The data acquisition program stores the received data directly into the huge page ring lock-free queue without waiting for the storage program to read the status, and then immediately receives new data. The storage program reads the data directly from the huge page ring lock-free queue and stores it into the hard disk without waiting for the acquisition program to write the status.

[0040] S3. The CPU polling of the NVMe hard disk in the user-mode driver is executed periodically. Once the IO operation is queried, the callback function is triggered immediately, instead of relying on the operating system interrupt processing, avoiding the context switching caused by hardware interrupts (theoretically consuming 300 clock cycles) and other losses. The kernel space driver shields the interrupt signal sent by the hardware, and the user space driver uses active polling to obtain data.

[0041] Traditional data acquisition programs usually store the received data in a common queue and then wait for the storage program to read the data. However, this method has some problems. For example, when the data is stored in the queue, it needs to wait for the storage program to read the status, which will cause the data acquisition program to be unable to receive new data in time, thus affecting the system throughput.

[0042] In order to solve this problem, the present invention adopts a huge page ring lock-free queue to realize the communication between the data acquisition program and the storage program. The huge page ring lock-free queue is a high-performance queue implementation method, which can directly store data into the pre-allocated huge page memory without waiting for the storage program to read the status. At the same time, due to the characteristics of the ring queue, the data acquisition program can continuously store new data into the queue without worrying about the queue being full.

[0043] In terms of storage programs, the present invention adopts a method of directly reading data from a huge page ring lock-free queue without waiting for the data acquisition program to write the status, which can minimize the waiting time during data transmission and improve the throughput of the system.

[0044] In addition to the communication method between the data acquisition program and the storage program, the present invention uses the CPU polling technology of the user-mode driver to further optimize the performance of the system. Traditional IO operations usually rely on the interrupt processing mechanism of the operating system, but this method will bring certain performance losses, such as context switching caused by hardware interrupts.

[0045] In order to avoid these performance losses, the present invention adopts the CPU polling technology of the user-mode driver. In this technology, the user-mode driver periodically polls the CPU of the NVMe hard disk, and once an IO operation is queried, the callback function will be triggered immediately instead of relying on the operating system interrupt processing. This can avoid the context switching losses caused by hardware interrupts and improve the performance of the system.

[0046] At the same time, in order to further reduce the performance loss caused by hardware interrupts, the present invention adopts the interrupt shielding technology driven by the kernel space. In this technology, the kernel space driver will shield the interrupt signal sent by the hardware, and the user space driver will use the active polling method to obtain data. This can avoid the loss of context switching caused by hardware interrupts and further improve the performance of the system.

[0047] In general, the present invention constructs a high-performance data acquisition and storage system by adopting technologies such as huge page ring lock-free queue, CPU polling of user-mode driver, and interrupt shielding of kernel space driver. This system can minimize the waiting time in the data transmission process, improve the system throughput, and avoid the performance loss caused by hardware interrupts, thereby achieving efficient data acquisition and storage.

[0048] The UIO user-mode driver framework and user-space polling, the former bypasses the kernel, and the latter actively polls to avoid hardware interrupts, so that the processing of sending and receiving packets can be performed in user mode. It brings zero copy (Zero Copy) and no system call (Systemcall) optimization. At the same time, it also avoids the asynchronous processing of soft interrupts and reduces the consumption caused by context switching.

[0049] The advantage of the UIO user-mode driver framework is that it can move the hardware driver from kernel space to user space, thus avoiding the switch between kernel space and user space. In the traditional kernel driver model, each hardware operation requires a system call, which will bring certain performance overhead. The UIO framework can directly map hardware resources to user space, and user-mode programs can directly access hardware resources, thereby reducing the number of system calls and improving system performance.

[0050] User space polling technology avoids hardware interrupts by actively polling. In traditional network applications, every time a network data packet is received, the hardware will trigger an interrupt, and the kernel will process the soft interrupt, which will bring certain performance overhead. User space polling technology can obtain network data packets by actively polling, thereby avoiding the processing of hardware interrupts and soft interrupts and reducing the consumption of context switching.

[0051] In addition, the UIO user-mode driver framework and user-space polling technology can also bring about zero copy and no system call optimization. In the traditional kernel driver model, each time a network data packet is received, the kernel will copy the data packet to the kernel buffer, and then copy it to the user-space buffer, which will bring certain performance overhead. The UIO framework can directly map hardware resources to user space, and user-mode programs can directly access hardware resources, thus avoiding the overhead of data copying. At the same time, the user-space polling technology can obtain network data packets through active polling, thus avoiding the overhead of system calls.

[0052] S4. Take advantage of the operating system's support for thread affinity and create multiple threads on a multi-core device. Each thread is bound to a separate core to reduce thread scheduling overhead and save CPU scheduling time.

[0053] First, by binding threads to specific cores, contention and switching between threads can be reduced, thereby reducing the overhead of context switching. On multi-core devices, each core has its own cache. If threads frequently switch between different cores, the data in the cache will be frequently invalidated, resulting in a decrease in the cache hit rate, which in turn affects performance.

[0054] Secondly, by binding threads to specific cores, better use of the CPU cache is achieved. When a thread runs on the same core, it can reuse the cache on that core, reducing access to main memory and improving performance.

[0055] Finally, by creating multiple threads, each bound to a separate core, parallel processing can be achieved to fully utilize the computing power of multi-core devices. In some computationally intensive tasks, this approach can significantly improve performance.

[0056] However, when using thread affinity, you also need to pay attention to some issues. For example, if the number of threads exceeds the number of CPU cores, some threads will be scheduled to the same core, resulting in increased contention and switching, which will in turn reduce performance. Therefore, when using thread affinity, you need to determine the appropriate number of threads and binding strategy based on the specific application scenario and hardware configuration.

[0057] In general, by leveraging the operating system's support for thread affinity and creating multiple threads on a multi-core device with each thread bound to a separate core, you can effectively improve application performance, but you need to determine the appropriate strategy based on the specific situation.

[0058] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A method for efficiently reading and writing massive data on NVMe hardware, characterized in that: The following steps are involved: S1. Use the user-mode driver framework to create a huge page ring lock-free queue dedicated channel for NVMe hard disk data transmission; S2, the data acquisition program directly stores the received data into the huge page ring lock-free queue, and then immediately receives new data. The storage program directly reads the data from the huge page ring lock-free queue and stores it into the hard disk; S3, CPU polling of the NVMe hard disk periodically executed in the user-mode driver; S4. By creating multiple threads on a multi-core device, each thread is bound to a separate core.

2. A method for efficiently reading and writing NVMe hardware mass data according to claim 1, characterized in that, The dedicated channel does not pass through the operating system kernel.

3. A method for efficiently reading and writing NVMe hardware mass data according to claim 1, characterized in that, The user-mode driver framework includes UIO, and the UIO includes a user-space driver and a kernel-space driver.

4. A method for efficiently reading and writing NVMe hardware mass data according to claim 3, characterized in that, In step S3, the kernel space driver shields the interrupt signal sent by the hardware.

5. A method for efficiently reading and writing NVMe hardware mass data according to claim 4, characterized in that, The user space driver uses active polling to obtain data.

6. A method for efficiently reading and writing NVMe hardware mass data according to claim 3, characterized in that, The step S1 specifically includes the following steps: S1.

1. Load the UIO driver, obtain the UIO name of the NVMe device under / sys / class / uio, and set the memory mapping address, size, and type; S1.2, call uio_register_device function to register and allocate the UIO device in kernel space; S1.

3. Call the open function in user space to open the device and create a huge page ring lock-free queue; S1.

4. Call the mmap function to map the device memory to the address space of the user space. After the mapping is successful, the address can be directly used to read and write the device memory.

7. A method for efficiently reading and writing NVMe hardware mass data according to claim 1, characterized in that: In step S1.2, if the registration is successful, the device mapping details will be generated under / sys / class / uio / uioX / maps / mapY / .

8. A method for efficiently reading and writing NVMe hardware mass data according to claim 1, characterized in that: In step S2, the storing does not need to wait for the storage program to read the status.

9. A method for efficiently reading and writing NVMe hardware mass data according to claim 1 or 8, characterized in that: In step S2, the reading does not need to wait for the acquisition program to write the status.

10. A method for efficiently reading and writing NVMe hardware mass data according to claim 1, characterized in that: In step S3, once the IO operation is found, the callback function is triggered immediately.

Images