RDMA-based big data transfer and storage system, method, and semiconductor testing equipment

Abstract

This invention provides a large data transfer storage system, method, and semiconductor testing device based on RDMA. The large data transfer storage system includes a hardware layer and a software layer. The hardware layer includes an NVMe SSD storage array, a GPU-based NVMe RAID acceleration card, and an RDMA network card. The software layer includes a multi-threaded transmission module, an adaptive memory pool module, a transmission control module, a data verification and retransmission module, a scheduling and optimization module, and a disk write and storage management module. This invention achieves data pass-through, zero-copy transmission, and RAID acceleration in the hardware link. Multi-threaded parallel RDMA transmission enables multi-channel parallel transmission. Data is directly transmitted from memory to the NVMe SSD storage array. Data reception and disk write threads are decoupled, and asynchronous writing directly accesses the NVMe SSD storage array, avoiding network overload or storage congestion. This results in high data transfer efficiency and reliability, facilitating widespread application in scenarios requiring large-scale image transmission and processing, such as semiconductor testing.

Description

Technical Field

[0001] This invention belongs to the field of big data transmission and storage management technology, specifically relating to a big data transmission and storage system, method and semiconductor testing equipment based on RDMA. Background Technology

[0002] Driven by the digital economy, the scale of data is growing exponentially, and big data transmission has become one of the key technologies for high-performance computing and data-intensive applications such as data centers and semiconductor manufacturing. Its bandwidth, latency, and efficiency directly determine the overall performance of the system. Especially in tasks such as real-time analysis of wafer inspection data in semiconductor manufacturing and distributed node collaborative computing in high-performance computing, it is often necessary to complete the transmission of TB-level or even PB-level data in a short period of time, which places stringent requirements on transmission technology for "high bandwidth, low latency, and high stability".

[0003] The current mainstream TCP / IP transmission schemes have significant bottlenecks: First, bandwidth is limited, and kernel transmission involves multiple data copies and "user mode-kernel mode" context switching, which cannot take full advantage of the high-speed network card performance; second, latency is relatively high, and multiple memory copies and protocol processing reduce real-time performance; third, CPU load is too high, and data reading, writing and copying consume a lot of computing power, affecting task concurrency.

[0004] To address these issues, several new technologies have emerged, one of which is Remote Direct Memory Access (RDMA). RDMA bypasses the operating system kernel to directly access the memory of remote devices, providing a zero-copy, low-latency transmission method that reduces transmission latency to the microsecond level. It also fully utilizes high-speed network bandwidth, becoming a core technology supporting high-performance data transmission and significantly improving data transmission efficiency.

[0005] However, existing RDMA-based data transfer methods still face some technical bottlenecks in large-scale data transfer and big data storage management: First, memory registration and management efficiency is low; "on-demand registration" is time-consuming and increases latency, while "static pre-registration" has poor adaptability and easily leads to idle resources; Second, memory fragmentation is severe; fragmentation generated during long-term transmission (fragmentation rate exceeding 40% in irregular data scenarios) reduces reuse rate and triggers additional allocation and registration overhead; Third, there is insufficient coordination between transmission and storage; the design of linkage with the storage system is ignored, and copying between cache and storage media still relies on the CPU, failing to fully unleash the capabilities of RDMA.

[0006] Therefore, a new high-performance big data transmission device and method based on RDMA is needed to solve the performance bottleneck of existing technologies in big data transmission and provide a data transmission solution with higher bandwidth, lower latency and higher efficiency. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of TCP / IP technology in the transmission of large data throughput in the semiconductor industry, which fails to meet the requirements. At the same time, RDMA has data storage management problems such as high memory registration overhead and low resource reuse rate in large data transmission. Therefore, this invention provides a large data transmission storage system, method and semiconductor testing equipment based on RDMA, which can solve the above problems.

[0008] A large data transfer storage system based on RDMA is disclosed. The large data transfer storage system includes a hardware layer and a software layer. The hardware layer includes an NVMe SSD storage array, an NVMe RAID accelerator card based on a GPU architecture, and an RDMA network card. The software layer includes a multi-threaded transmission module, an adaptive memory pool module, a transmission control module, a data verification and retransmission module, a scheduling and optimization module, and a disk persistence and storage management module.

[0009] Furthermore, the NVMe RAID accelerator card generates RAID checksums through parallel GPU computing, and through the hardware control of the accelerator card, data is directly transmitted between the NVMe SSD storage array and memory via PCIe channel, without the need for the host CPU to participate in data copying.

[0010] Furthermore, the multi-threaded transmission module segments the data to be transmitted into blocks of a preset size, and multiple threads bind to independent queues QP and completion queue CQ to perform parallel RDMA communication transmission.

[0011] Furthermore, the adaptive memory pool module includes multiple pre-configurable and registerable contiguous memory blocks for binding, unbinding, and returning user data addresses before and after transmission.

[0012] Furthermore, the data verification and retransmission module includes a verification unit, a transmission error triggering unit, and a retransmission unit. Each data block is accompanied by a CRC checksum. The transmission error triggering unit operates based on a two-way feedback message confirmation mechanism, and the retransmission unit triggers partial retransmission based on the feedback result.

[0013] Furthermore, the scheduling and optimization module is used to dynamically adjust the number of threads, memory block size, and transmission batches based on real-time load, and combines adaptive rate limiting strategies to prevent network or storage overload.

[0014] Furthermore, the disk write and storage management module decouples the data receiving and disk write threads through a task queue, and bypasses the page cache to directly access the NVMe SSD storage array through asynchronous write.

[0015] Furthermore, the software layer of the big data transmission and storage system also includes a performance monitoring and logging module, which is used to collect and analyze the system's performance indicators and generate monitoring logs to provide a basis for subsequent performance tuning.

[0016] The present invention also provides a large data transfer and storage method based on RDMA, comprising the following steps.

[0017] S1. In the data acquisition stage, the image acquisition unit acquires images of the target workpiece. The acquisition card directly writes the image data from the image acquisition unit into the server memory based on DMA pass-through technology.

[0018] S2, Server Memory Caching Stage: After data enters the server memory, it is cached by the memory pool of the big data transmission and storage system, waiting for the transmission thread or RDMA to schedule and process it.

[0019] S3, Data transmission stage: Data is transmitted to a remote server or storage device via a big data transmission and storage system network.

[0020] S4, Data Write-to-Disk and Storage Stage: The remote server or storage device receives the transmitted data blocks and saves the data blocks with accompanying metadata to the local memory pool. Then, relying on the hardware control of the NVMe RAID acceleration card, the data is directly transmitted from the memory pool to the NVMe SSD storage array through an asynchronous write mechanism to complete the storage.

[0021] The present invention also provides a semiconductor testing device, which performs image acquisition, transmission and storage based on the aforementioned big data transmission and storage system and big data transmission and storage method.

[0022] Compared with existing technologies, the advantages of this invention are as follows: This invention achieves multi-channel parallel transmission through data pass-through, zero-copy transmission, and RAID acceleration in the hardware link, and multi-threaded parallel RDMA transmission, avoiding network overload or storage blockage; data is directly transmitted from the NVMe SSD storage array to the GPU through the PCIe channel, and the data receiving and disk writing threads are decoupled to directly access the NVMe SSD storage array through asynchronous writing, which improves data transmission efficiency, has high reliability, and is easy to promote and apply in scenarios that require large-scale image transmission and processing, such as semiconductor testing. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the hardware architecture of the large data transmission and storage system of the present invention;

[0024] Figure 2 A schematic diagram of the software architecture of a big data transmission and storage system;

[0025] Figure 3 This is a flowchart illustrating a method for storing large data transfer data. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0028] A large data transfer storage system based on RDMA, comprising a hardware layer and a software layer.

[0029] See Figure 1 The hardware layer includes NVMe SSD storage arrays, GPU-based NVMe RAID accelerator cards, and RDMA network cards. Specifically, the hardware layer achieves high-speed parallel read / write and integrated access to large-scale data by building a high-performance storage array composed of multiple NVMe SSDs and cooperating with GPU-based NVMe RAID accelerator cards. The RAID accelerator card is responsible for virtualizing multiple NVMe devices into logical volumes, providing a unified high-bandwidth data access interface for the upper-layer RDMA channel, thereby eliminating the performance bottleneck of traditional CPU software RAID.

[0030] The NVMe RAID acceleration card used in the entire system is based on GPU architecture. It achieves a breakthrough in IO performance through hardware-level parallel computing reconstruction and IO process optimization, as detailed below.

[0031] GPU-driven parallel computing accelerates RAID checksum generation: Traditional RAID cards rely on CPUs or dedicated ASIC chips to generate checksums, resulting in computational bottlenecks. This system uses a RAID card integrated with an NVIDIA A2000 GPU, completely offloading the RAID checksum generation task to the GPU for parallel execution. For write operations, data striping decomposes into parallel computing units, allowing the GPU to process checksum generation for multiple stripes simultaneously, achieving speed improvements several times over compared to traditional solutions. For read operations, the GPU can schedule multiple SSD I / O requests in parallel, fully unleashing the concurrent performance of NVMe SSDs.

[0032] Zero CPU dependency and direct hardware connection in the IO process: The NVMe RAID accelerator card used in this system adopts a fully hardware-based IO path design, completely bypassing the intervention of the host CPU and system memory. Relying on the hardware control of the accelerator card, data is directly transmitted between the NVMe SSD storage array and memory via PCIe channels, without the need for additional memory transfer, eliminating the traditional bottleneck of "CPU participation in data copying". At the hardware level, PCIe bridging and high-speed interconnect protocols are integrated, and the distribution, scheduling, and verification of IO commands are all completed autonomously by the GPU and RAID controller chip.

[0033] By combining GPU parallel computing with hardware I / O, the NVMe RAID accelerator card used in this system achieves a performance revolution over traditional RAID solutions. Especially in I / O-intensive scenarios such as AI training, high-performance computing, and cloud computing, it can increase data throughput and response speed by several times to tens of times.

[0034] In a specific example, at the hardware layer, the data storage server uses a high-performance computing platform as the basic hardware environment. In one example, the server configuration is as follows.

[0035] Central Processing Unit: Equipped with a dual-processor multi-core processor, it performs tasks such as data segmentation, transmission scheduling, and system control, and has high-concurrency multi-threaded processing capabilities.

[0036] Memory: The server is configured with 256GB of DDR4 memory for system operation, cache management, and data buffering during RDMA transfers. A portion of the memory space is managed by a memory pool module for dynamically allocating RDMA-registerable memory blocks to reduce the overhead of frequent memory allocation and registration during large-scale image transfers.

[0037] Storage System: The server features 24 built-in NVMe SSDs, each with a capacity of 8TB, for a total capacity of approximately 192TB. The SSDs are aggregated into a single logical volume using a GPU-based NVMe RAID accelerator card. The RAID accelerator card leverages the GPU's parallel computing capabilities to implement RAID encoding and decoding logic at the hardware level, significantly reducing the computational load of traditional CPU software RAID. Simultaneously, it utilizes multi-channel NVMe I / O paths to achieve high-bandwidth parallel access and fault-tolerant redundancy to the storage array. When working in conjunction with the SSDs, the local I / O performance can stably reach approximately 50GB / s, providing robust support for high-performance data transfer.

[0038] Network interface: The server is equipped with multiple dual-port 100Gbps RDMA network cards, supporting the RoCEv2 (RDMA over Converged Ethernet) protocol.

[0039] This network card provides high-bandwidth, low-latency data transmission capabilities for the RDMA channel, enabling zero-copy communication with remote clients.

[0040] System interconnection and expansion bus: The server internally connects the RAID acceleration card and network interface card via a high-speed PCIe 4.0 bus, ensuring high bandwidth and low latency of the data path.

[0041] Through the above hardware architecture, this invention constructs a high-performance data transmission server with ultra-high concurrency read / write capabilities, low-latency transmission paths, and large-capacity storage resources, providing strong hardware support for subsequent software layer modules.

[0042] See Figure 2 The software layer includes a multi-threaded transmission module, an adaptive memory pool module, a transmission control module, a data verification and retransmission module, a scheduling and optimization module, and a disk persistence and storage management module. The software layer runs on top of the hardware layer and is built using a modular design. The NVMe RAID accelerator card generates RAID checksums through GPU parallel computing, and through the hardware control of this accelerator card, data is directly transmitted between the NVMe SSD storage array and memory via PCIe channels, without requiring host CPU involvement in data copying.

[0043] The multi-threaded transmission module divides the data to be transmitted into segments according to a preset block size, and multiple threads bind to independent queues QP and completion queue CQ to perform parallel RDMA communication transmission.

[0044] Specifically, the multi-threaded transmission module is responsible for dividing the massive amounts of data to be transmitted (such as image files or detection result data ranging from hundreds of megabytes to tens of gigabytes) into blocks and transmitting them in parallel through a multi-threaded mechanism.

[0045] Thread partitioning and binding: At startup, the system dynamically creates multiple transmission threads based on the number of CPU cores and network bandwidth. Each thread independently maintains a set of RDMA communication resources, including queue pairs (QP) and completion queues (CQ), to ensure that multi-channel parallel communication does not interfere with each other.

[0046] Task scheduling strategy: Data shards are allocated to each thread for execution by the task scheduling module according to a load balancing strategy. Adaptive scheduling based on block size, channel bandwidth, or real-time latency is supported to avoid congestion on some channels. During transmission, network bandwidth and storage speed are matched in real-time to prevent network overload or storage blockage.

[0047] Parallel transmission: Each thread concurrently executes RDMA operation requests (such as IBV_WR_RDMA_WRITE or IBV_WR_RDMA_READ), continuously sending data to the remote node in a pipeline manner across multiple QPs, thereby making full use of network bandwidth and achieving near-line-speed transmission performance.

[0048] The adaptive memory pool module includes multiple pre-configurable and registerable contiguous memory blocks for binding, unbinding, and returning user data addresses before and after transmission.

[0049] Specifically, the adaptive memory pool module is used to manage the registerable memory regions during RDMA data transfer, reducing the system overhead caused by frequent registration.

[0050] Dynamic allocation and binding: During the initialization phase, the system creates a memory pool based on available memory and load prediction, and divides it into multiple contiguous memory blocks of fixed size (e.g., 40MB or 64MB). Before each transfer, the thread obtains a free memory block from the memory pool and dynamically registers it by calling ibv_reg_mr when needed, returning the memory address and access key (RKey).

[0051] Unbinding and Reuse: After the transmission is completed, the thread triggers the unbinding operation through the CQE notification mechanism, updates the status of the memory block to free, and returns it to the memory pool to achieve fast reuse.

[0052] State machine management mechanism: Each memory block maintains an independent state flag (idle, in use, locked) to ensure data consistency and safe release during concurrent access by multiple threads. Through this mechanism, this invention significantly reduces memory registration latency and improves resource reuse efficiency while maintaining flexibility.

[0053] The data verification and retransmission module includes a verification unit, a transmission error triggering unit, and a retransmission unit. Each data block is accompanied by a CRC checksum. The transmission error triggering unit operates based on a two-way feedback message confirmation mechanism, and the retransmission unit triggers partial retransmission based on the feedback result. This module operates using a verification mechanism, a retransmission mechanism, and a feedback mechanism.

[0054] Verification mechanism: Each data block is accompanied by a CRC checksum, and the receiving end returns the result after completing the verification. Batch verification can be used for large files to reduce CPU overhead.

[0055] Retransmission mechanism: Network transmission errors (packet loss, connection anomalies) automatically trigger partial retransmissions, supporting timed retries and retry limit settings. Data integrity errors (CRC check failures) trigger block-level retransmissions to ensure data consistency. It also supports mixed handling of network and data errors to guarantee high reliability.

[0056] Feedback mechanism: The receiving end returns ACK / NACK by data block or batch, and the sending end intelligently schedules retransmission based on the feedback to avoid global blocking.

[0057] This module works closely with multi-threaded transmission and memory pools to ensure data reliability under high concurrency.

[0058] The scheduling and optimization module dynamically adjusts the number of threads, memory block size, and transmission batches based on real-time load, and incorporates adaptive rate limiting strategies to prevent network or storage overload. Specifically, the scheduling and optimization module includes: a task scheduling unit responsible for scheduling tasks based on priority and dependencies; a resource management unit that monitors resource status and dynamically allocates resources; a data flow control unit that coordinates data flow between different nodes; an optimization engine that applies algorithms to improve processing efficiency; and a monitoring and feedback unit that collects performance data to support adaptive adjustments.

[0059] The scheduling and optimization module is responsible for the overall system performance monitoring and adaptive resource tuning. It dynamically adjusts the number of threads and the size of transmission batches based on real-time transmission rate, CPU utilization, and memory usage. When the system detects memory pool resource shortages or a sudden increase in transmission load, it automatically expands or reclaims some memory blocks to improve system resilience.

[0060] The disk write and storage management module decouples data reception and disk write threads through a task queue, and bypasses the page cache to directly access the NVMe SSD storage array through asynchronous write.

[0061] Specifically, the disk write and storage management module executes data reception and disk writing logic on remote nodes: the reception and disk write threads are decoupled through a task queue; asynchronous write using io_uring or O_DIRECT directly accesses the NVMe array, bypassing the page cache; 4KB page alignment optimization is performed before writing to reduce write amplification; the RAID GPU card automatically completes parity generation and striped writing at the underlying level to accelerate redundant disk writing; after the data is written to disk, the memory block is immediately returned to the memory pool, forming an efficient cyclic pipeline.

[0062] Furthermore, the software layer of the big data transmission and storage system also includes a performance monitoring and logging module, which is used to collect and analyze the system's performance indicators and generate monitoring logs to provide a basis for subsequent performance tuning.

[0063] System performance metrics include RDMA latency, memory registration time, bandwidth utilization, and NVMe I / O rate.

[0064] Furthermore, the software layer of the big data transmission and storage system also includes a transmission control module, which is responsible for the low-level operation control of RDMA communication, constructs and submits RDMA operation requests (WR, Work Request), including the local buffer address, remote address and operation type.

[0065] Specifically, the transmission control module manages the establishment of the QP (queue pair) state machine and the initialization and disconnection of the RTS (Ready To Send). The transmission control module asynchronously detects data transmission completion events through the CQ (Completion Queue), employing a Poll + Notify pattern to reduce polling overhead. When the CQE (Completion Event) returns successfully, it triggers subsequent memory reclamation and task scheduling updates. This module interacts directly with the RDMA hardware and is crucial for achieving low-latency data transmission.

[0066] Of course, according to their functions, the software layer includes the application layer, control layer, data transmission layer, and data processing layer.

[0067] The application layer includes business call interfaces and a performance monitoring module, which mainly monitors memory usage and network speed.

[0068] The control layer includes multi-threaded transmission, memory pool, and task scheduling. This part is mainly used for thread partitioning and binding, task scheduling, dynamic memory allocation, memory reuse, and adjusting transmission according to actual conditions.

[0069] The data transmission layer includes a transmission control module, verification and retransmission, and an adaptive rate control mechanism. This part is mainly used for RDMA session establishment, WR construction and reuse, QP state management, CQ event processing, and data verification, and operates based on the retransmission mechanism and the adaptive rate control mechanism.

[0070] The data processing layer includes data segmentation, data writing to disk, and storage management. This part is mainly used for decoupling transmission / writing tasks, asynchronous disk writing, 4K memory alignment, and RAID striping writing.

[0071] The overall principle and operation steps of a big data transmission and storage system.

[0072] System principle: The NVMe RAID acceleration card used in the system is based on GPU architecture and achieves a breakthrough in IO performance through hardware-level parallel computing reconstruction and IO process optimization.

[0073] GPU-driven parallel computing accelerates RAID checksum generation. Traditional RAID cards rely on CPUs or dedicated ASIC chips to generate checksums, resulting in computational bottlenecks. This system uses RAID cards with integrated GPUs, such as the NVIDIA A2000, which completely offload the RAID checksum generation task to the GPU for parallel execution. For write operations, data striping is broken down into parallel computing units, allowing the GPU to process checksum generation for multiple stripes simultaneously, resulting in speed improvements several times over compared to traditional solutions. For read operations, the GPU can schedule multiple SSD I / O requests in parallel, fully leveraging the concurrent performance of NVMe SSDs.

[0074] The system features a "zero CPU dependency" and direct hardware-level connection for its IO process. The NVMe RAID accelerator card used in this system employs a fully hardware-based IO path design, completely bypassing the intervention of the host CPU and system memory. Relying on the accelerator card's hardware control, data is directly transmitted between the NVMe SSD storage array and memory via PCIe channels, eliminating the need for additional memory intermediaries and removing the traditional bottleneck of "CPU involvement in data copying." At the hardware level, PCIe bridging and high-speed interconnect protocols are integrated, and the distribution, scheduling, and verification of IO instructions are all autonomously handled by the GPU and RAID controller chip. Through GPU parallel computing and hardware-based IO, the NVMe RAID accelerator card used in this system achieves a performance revolution compared to traditional RAID solutions, especially in IO-intensive scenarios such as AI training, high-performance computing, and cloud computing, where it can increase data throughput and response speed by several times to tens of times.

[0075] The system workflow is divided into system preparatory work, system initialization stage, data preparation and allocation stage, RDMA transmission stage, transmission completion and memory reclamation stage, and disk write and storage management stage, as detailed below.

[0076] System preparatory work: Load and initialize multiple NVMe SSD disks through a GPU-based RAID acceleration card to build a high-performance NVMe RAID array; aggregate multiple NVMe devices by the RAID control logic to form a unified logical volume interface, providing high-bandwidth, low-latency storage access capabilities; connect and configure a 100G high-speed network card, and enable RoCE (RDMA over Converged Ethernet) or InfiniBand RDMA protocol.

[0077] System initialization phase: The system creates an independent RDMA communication context for each worker thread, including QP (Queue Pair), CQ (Completion Queue), and MR (Memory Region) references; thread-to-thread resource contention is avoided through lock separation mechanism and thread-local queues, improving concurrency performance; during the initialization phase, the system creates a memory pool based on available memory resources and expected load; the memory pool consists of several configurable contiguous memory blocks.

[0078] Data preparation and allocation phase: After the upper layer application submits the task to be transmitted, the data is fragmented according to the preset block size (such as 256MB, 512MB, etc.); the task scheduling module allocates the data to the multi-threaded channel according to the load; each thread obtains an available memory block from the memory pool and dynamically binds the data buffer to MR before sending.

[0079] RDMA transmission phase: The transmission control module constructs an RDMA request (such as RDMA_WRITE or RDMA_READ) and submits the WR to the corresponding QP; multi-threaded parallel execution of transmission operations; CQ asynchronously detects completion events through a Poll+Notify mechanism to improve concurrency efficiency. Adaptive rate control mechanism; Data verification and retransmission module: The receiver performs CRC verification on each data block; successful verification returns ACK, and failed verification returns NACK; network anomalies or packet loss will also trigger NACK; the sender triggers partial retransmission (data block level or batch level) based on ACK / NACK feedback, supporting timed retries and number of attempts; it also supports mixed handling of network errors and data errors to ensure high reliability.

[0080] Transmission Completion and Memory Reclamation Phase: After the transmission is completed, CQE notifies the binding module to perform MR unbinding operation; the memory block status is updated to free and returned to the memory pool.

[0081] Disk write and storage management phase: Data reception and disk write threads are decoupled through task queues; 4KB page alignment optimization is performed before writing; asynchronous writing bypasses page cache and directly accesses the NVMe SSD array.

[0082] Based on the above system operation process, the present invention extracts the following large data transmission and storage method. Specifically, a large data transmission and storage method based on RDMA is described below. Figure 3 The big data transmission and storage method includes the following steps.

[0083] S1. In the data acquisition stage, the image acquisition unit acquires images of the target workpiece. The acquisition card directly writes the image data from the image acquisition unit into the server memory based on DMA pass-through technology.

[0084] After this stage is completed, raw detection image data blocks are generated in real time in the server memory; on the semiconductor detection production line, the image data is acquired in real time by a high-speed TDI camera.

[0085] The capture card and camera are connected via a 100Gbps high-speed fiber optic link. The capture card is directly mounted on the server motherboard via a PCIe slot, sharing high-bandwidth interconnect with the system bus.

[0086] The capture card incorporates an FPGA logic unit to receive the raw pixel stream from the camera and perform initial buffering and frame synchronization. Simultaneously, the capture card supports DMA (Direct Memory Access) pass-through technology, mapping its local cache space to the same address space as the server system memory. Through this mapping mechanism, image data can be directly written from the capture card to the server memory without requiring a CPU copy process, thus achieving "zero-copy" high-speed transmission from the camera to the server memory.

[0087] Once this stage is completed, raw detection image data blocks (single image size is approximately hundreds of MB to tens of GB) are generated in real time in the server's memory.

[0088] S2, Server Memory Caching Stage: After data enters the server memory, it is cached by the memory pool of the big data transmission and storage system, waiting for the transmission thread or RDMA to schedule and process it.

[0089] S3, Data transmission stage: Data is transmitted to a remote server or storage device via a big data transmission and storage system network.

[0090] In some real-time monitoring or distributed storage scenarios, data also needs to be transmitted over the network to a remote server or storage system. This stage is completed by the RDMA transmission module, and the specific process is as follows.

[0091] S31. Task scheduling and allocation: The system divides the data blocks into segments (e.g., each segment is 256MB to 512MB) according to the size of the image to be transmitted, and allocates them to different transmission threads.

[0092] S32, RDMA Session Establishment: Each thread maintains an independent QP and CQ, and establishes an RDMA communication channel with the remote node.

[0093] S33. Memory Registration and Binding: When a thread is ready to send data, it obtains a free block from the memory pool and calls the memory registration interface to register the memory region. The system saves the corresponding local address and RKey information for use by RDMA operations.

[0094] S34. Data transmission execution: The RDMA transmission module constructs a Work Request and submits it to the corresponding QP; the remote node executes the IBV_WR_RDMA_READ or IBV_WR_RDMA_WRITE operation to realize direct remote reading and writing of data.

[0095] S35. Adaptive Rate Control: The system dynamically adjusts data block allocation and transmission size by monitoring memory pool usage, network transmission rate, and storage I / O load in real time, thereby achieving automatic matching between network transmission rate and storage I / O rate.

[0096] S36. Verification and Retransmission: Each data block is accompanied by a CRC checksum. After the receiving end completes the verification, it returns ACK / NACK.

[0097] Network transmission errors (packet loss, connection anomalies) automatically trigger partial retransmissions, supporting time-limited retries and attempt limits. Data integrity errors (CRC check failures) trigger block-level retransmissions. The sending end intelligently schedules retransmissions based on feedback, ensuring high reliability and avoiding global blocking.

[0098] S37. Complete Detection and Memory Reclamation: CQ asynchronously detects the transmission completion event through the Poll / Notify mechanism. After the transmission is completed, it performs a memory unbinding operation, updates the memory block status to free, and returns it to the memory pool for reuse.

[0099] Through the above process, a high-bandwidth, low-latency, zero-copy transmission link is achieved from server memory to remote storage nodes. In a 100Gbps network environment, the single-channel transmission rate can reach near line speed (approximately 11-12GB / s), and the end-to-end latency is as low as microseconds, meeting the real-time and reliability requirements of semiconductor testing scenarios.

[0100] S4, Data Write-to-Disk and Storage Stage: The remote server or storage device receives the transmitted data blocks and saves the data blocks with accompanying metadata to the local memory pool. Then, relying on the hardware control of the NVMe RAID acceleration card, the data is directly transmitted from the memory pool to the NVMe SSD storage array through an asynchronous write mechanism to complete the storage.

[0101] Specifically, after the RDMA remote transmission is completed, the remote node (server or storage system) receives the data block from the acquisition end and saves it in the local memory pool. Each data block is accompanied by metadata (such as sequence number, size, check information, etc.).

[0102] The system maintains a thread-safe task queue, enqueuing data blocks to be written to disk in order. Through queue management, the transmission thread and the disk writing thread are decoupled, improving the overall throughput.

[0103] The system starts multiple disk write threads. Each thread retrieves data blocks from the task queue for writing operations. For RAID arrays composed of NVMe, io_uring or asynchronous Direct I / O (O_DIRECT) is used to bypass the kernel page cache, reduce CPU usage, and improve write speed. Data blocks can be memory aligned (e.g., 4K alignment) before writing to match the disk page size and reduce write amplification.

[0104] If a write failure or verification anomaly is detected during the disk writing process, the system can trigger a partial retransmission to ensure end-to-end data consistency.

[0105] After the data is written to disk, the disk writing thread marks the corresponding memory block as free and returns it to the memory pool. The RDMA transfer module can immediately reuse these memory blocks, supporting continuous and high-speed data transmission and realizing an end-to-end "zero-copy" pipeline.

[0106] RAID cards use GPUs to perform write consolidation and verification, accelerating data write speeds and ensuring the reliability of redundant storage.

[0107] This method enables a high-speed data path across the entire chain, from camera acquisition to memory buffering to RAID storage.

[0108] A semiconductor testing device is provided, which is based on the aforementioned big data transmission and storage system and employs the aforementioned big data transmission and storage method for high-speed image acquisition, transmission and storage.

[0109] In summary, compared with existing TCP / IP and RDMA data transmission schemes, the present invention has the following technical highlights and beneficial effects.

[0110] Maximizing Hardware Capabilities: This invention achieves a maximum overall transmission and write performance of 50 GB / s (depending on NVMe array configuration and number of links) by implementing data pass-through, zero-copy transmission, and RAID acceleration in the hardware link. Compared to NFS RDMA solutions, this system directly calls the RDMA interface in user space, eliminating the need for NFS underlying protocol encapsulation and parsing, resulting in a shorter transmission path and lower latency. Under the same hardware environment, the end-to-end transmission rate of this invention is at least three times that of the NFS RDMA solution, with bandwidth utilization exceeding 90%, effectively leveraging the peak performance of the underlying RDMA and NVMe devices.

[0111] Higher system resource utilization and lower CPU usage: In traditional TCP / IP transmission, data needs to undergo multiple layers of protocol encapsulation, multiple copies between kernel and user modes, and context switching, resulting in significant CPU resource consumption. This invention, based on RDMA's zero-copy mechanism, allows data to be transmitted directly in user mode without CPU intervention. Under the same bandwidth conditions, the CPU usage of RDMA transmission is typically only 10%–20% of that of the TCP / IP scheme, significantly improving overall system resource utilization.

[0112] This not only enables the server to maintain stable throughput under high bandwidth, but also frees up more computing resources for other parallel tasks (such as image processing). Especially in the semiconductor testing industry, the CPU often needs to undertake complex image analysis and equipment control tasks. The low CPU usage feature of this invention can effectively alleviate the problem of computing resource shortage and achieve efficient parallelism of transmission and computing.

[0113] An adaptive memory pool mechanism improves memory registration efficiency: By decoupling memory allocation from the registration process, this invention maintains flexibility while avoiding repetitive memory requests and system page table operations, significantly reducing CPU involvement and registration overhead. Compared to traditional on-demand allocation and frequent registration schemes, this invention achieves higher transmission efficiency and lower end-to-end latency, resulting in a significant reduction in transmission latency compared to traditional on-demand registration schemes.

[0114] State machine-based memory block management enhances concurrent scheduling capabilities: each memory block maintains an independent state flag (idle, in use, locked), supporting multi-threaded parallel access and safe release of references. This mechanism enables dynamic reuse and precise lifecycle control of memory blocks, avoiding memory fragmentation and improving the stability of continuous big data transmission.

[0115] Multi-threaded parallel RDMA transmission mechanism: By fragmenting large-scale data and processing it in parallel by multiple RDMA threads, each thread is bound to an independent QP (Queue Pair), achieving multi-channel parallel transmission. Under 100G network bandwidth, it can fully leverage the hardware's potential, increasing system throughput by approximately 3 to 5 times. Adaptive rate control, combined with rate limiting strategies and load monitoring, achieves end-to-end balancing, ensuring continuous high-bandwidth, low-latency data flow and avoiding network overload or storage congestion.

[0116] NVMe RAID storage acceleration architecture for optimized end-to-end performance: This invention uses an NVMe RAID acceleration card with an integrated GPU architecture to assemble multiple NVMe SSDs into a high-speed logical volume. This architecture achieves I / O parallelization and cache prefetching in both the write and read phases, accelerating data write-to-disk and loading speeds, further improving end-to-end transmission efficiency.

[0117] Low latency and high bandwidth end-to-end optimization: This invention optimizes each stage of data caching, registration, transmission and storage, reducing CPU context switching and kernel-mode memory copying. In a 100Gbps network environment, it achieves microsecond-level transmission latency and near-line-speed data throughput, significantly improving the system's real-time response capability and data consistency.

[0118] Suitable for high real-time scenarios: This solution is particularly suitable for large data transmission applications that require low latency, high reliability, and high concurrency, such as semiconductor testing, large-scale image acquisition, high-performance computing (HPC), and cloud storage systems.

[0119] The aforementioned semiconductor inspection equipment includes devices for bright / dark field inspection of patterned and patternless wafers, overlay measurement, and alignment. It is also used in other fields, such as scanning electron microscopy, where high-speed image and large-scale data transmission is required.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large data transfer and storage system based on RDMA, characterized in that: Big data transmission and storage systems consist of a hardware layer and a software layer: The hardware layer includes NVMe SSD storage arrays, GPU-based NVMe RAID accelerator cards, and RDMA network cards; The software layer includes a multi-threaded transmission module, an adaptive memory pool module, a transmission control module, a data verification and retransmission module, a scheduling and optimization module, and a disk persistence and storage management module. The NVMe RAID accelerator card generates RAID checksums through GPU parallel computing, and through the hardware control of the accelerator card, data is directly transmitted between the NVMe SSD storage array and memory via PCIe channel without the host CPU participating in data copying.

2. The large data transmission and storage system according to claim 1, characterized in that: The multi-threaded transmission module divides the data to be transmitted into segments according to a preset block size, and multiple threads bind to independent queues QP and completion queue CQ to perform parallel RDMA communication transmission.

3. The large data transmission and storage system according to claim 1, characterized in that: The adaptive memory pool module includes multiple pre-configurable and registerable contiguous memory blocks for binding, unbinding, and returning user data addresses before and after transmission.

4. The large data transmission and storage system according to claim 1, characterized in that: The data verification and retransmission module includes a verification unit, a transmission error triggering unit, and a retransmission unit. Each data block is accompanied by a CRC check code. The transmission error triggering unit operates based on a two-way feedback message confirmation mechanism, and the retransmission unit triggers partial retransmission based on the feedback result.

5. The large data transmission and storage system according to claim 1, characterized in that: The scheduling and optimization module is used to dynamically adjust the number of threads, memory block size and transmission batches according to the real-time load, and combine adaptive rate limiting strategies to prevent network or storage overload.

6. The large data transmission and storage system according to claim 1, characterized in that: The disk write and storage management module decouples the data receiving and disk write threads through a task queue, and bypasses the page cache to directly access the NVMeSSD storage array through asynchronous write.

7. The large data transmission and storage system according to claim 1, characterized in that: The software layer of the big data transmission and storage system also includes a performance monitoring and logging module, which is used to collect and analyze the system's performance indicators and generate monitoring logs to provide a basis for subsequent performance tuning.

8. A large data transfer and storage method based on RDMA, characterized in that, The big data transmission method includes the following steps: S1. In the data acquisition stage, the image acquisition unit acquires images of the target workpiece, and the acquisition card directly writes the image data of the image acquisition unit into the server memory based on DMA pass-through technology. S2, Server Memory Caching Stage: After data enters the server memory, the memory pool of the big data transmission and storage system allocates data caching, waiting for the transmission thread or RDMA scheduling to process. S3, Data transmission stage: Data is transmitted to a remote server or storage device via a big data transmission and storage system network. S4, Data Write-to-Disk and Storage Stage: The remote server or storage device receives the transmitted data blocks and saves the data blocks with accompanying metadata to the local memory pool. Then, relying on the hardware control of the NVMe RAID acceleration card, the data is directly transmitted from the memory pool to the NVMe SSD storage array through an asynchronous write mechanism to complete the storage.

9. A semiconductor testing device, characterized in that: The semiconductor testing equipment is based on the large data transmission and storage system according to any one of claims 1-7, and uses the large data transmission and storage method according to claim 8 to perform image acquisition, transmission and storage.

Images