A high-speed streaming system and method based on PCIe bus

Through a high-speed flow disk system based on PCIe bus, efficient data transmission is achieved using DMA method and RAID controller, which solves the problem of low data acquisition and transmission speed in the prior art, and realizes efficient data acquisition and storage.

CN114116572BActive Publication Date: 2024-11-26QUEENTEST
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111083767.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-14
Publication Date
2024-11-26
Estimated Expiration
2041-09-14

AI Technical Summary

Technical Problem

In the prior art, data acquisition and transmission speeds are relatively low, making it difficult to meet the needs of high-speed data processing.

Method used

Using a high-speed flow disk system based on PCIe bus, the data is written directly into the memory of the host computer through the acquisition card in DMA mode, and then the CPU transports the blocks of the data to the hardware cache of the RAID controller, and finally stores it in the disk array, supporting Efficient data transmission and storage.

Benefits of technology

It realizes efficient data acquisition and transmission. The continuous acquisition storage speed of the acquisition card can reach 2.5GByte/s, supports 1-32TByte data storage capacity, reduces CPU interrupt load, allows hardware communication at different speeds, and data can be sized according to the user specified. Split into multiple files.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114116572B_ABST
    Figure CN114116572B_ABST
Patent Text Reader

Abstract

The present invention discloses a high-speed disk streaming system and method based on PCIe bus, which relates to the technical field of data acquisition and transmission. The system includes an acquisition card, a PCIe bus, a host computer, a RAID controller and a disk array; the acquisition card is used to acquire data, and the acquired data is directly written into the memory of the host computer in a DMA mode via the PCIe bus, and then the CPU of the host computer moves the block data to the hardware cache controlled by the RAID controller, and finally stores it in the disk array. It allows hardware devices of different speeds to communicate without relying on a large number of interrupt loads of the CPU, and has a continuous data recording function; the acquisition card can continuously acquire and store at a speed of up to 2.5GByte / s; the data storage capacity of the host computer supports 1‑32TByte; it supports 1‑4 channels and 1‑3 acquisition cards; the acquisition card can be controlled by QTCapture Lab software to realize continuous acquisition and storage; the data can be divided into multiple files according to the size specified by the user; the user specifies the file name and storage directory; each data file has a matching configuration file.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition and transmission, and in particular to a high-speed disk streaming system and method based on a PCIe bus. Background Art

[0002] Data acquisition (DAQ) refers to the automatic collection of non-electrical or electrical signals from analog and digital units such as sensors and other devices under test, and sending them to the host computer for analysis and processing. The data acquisition system is a flexible, user-defined measurement system that combines measurement software and hardware products based on computers or other dedicated test platforms.

[0003] In the prior art, the data collection and transmission speed needs to be improved. Summary of the invention

[0004] In order to overcome the above problems or at least partially solve the above problems, an embodiment of the present invention provides a high-speed disk streaming system and method based on a PCIe bus to improve data acquisition and transmission speed.

[0005] The embodiment of the present invention is achieved as follows:

[0006] In a first aspect, an embodiment of the present invention provides a high-speed disk streaming system based on a PCIe bus, which includes an acquisition card, a PCIe bus, a host computer, a RAID controller and a disk array; the acquisition card is used to acquire data, and the acquired data is directly written into the memory of the host computer in a DMA manner via the PCIe bus, and then the CPU of the host computer moves the block data to the hardware cache controlled by the RAID controller, and finally stores it in the disk array.

[0007] Exemplarily, the configuration of the host computer is: memory capacity 16GB, DDR4; support 4 PCIe Gen2 X8 slots; support all slots to transmit data to PC memory in parallel; system disk SSD 256GB, DVD R / W optical drive; using Supermicro server (Super Micro) motherboard, integrated graphics card; Xeon Quad Core 2.4GHz processor, pre-installed genuine Windows 7 64bit Professional Edition operating system (SP1); power supply 1280W.

[0008] Such a system allows hardware devices of different speeds to communicate without relying on a large number of CPU interrupt loads, and has a continuous data recording function; the acquisition card can continuously acquire and store data at a speed of up to 2.5GByte / s; the host computer's data storage capacity supports 1-32TByte; it supports 1-4 channels and 1-3 acquisition cards; the acquisition card can be controlled by QTCapture Lab software to achieve continuous acquisition and storage; data can be divided into multiple files of user-specified size; the user specifies the file name and storage directory; each data file has a matching configuration file.

[0009] Based on the first aspect, in some embodiments of the present invention, the acquisition card is provided with: an infinite point single trigger module, which is used to virtualize the onboard memory of the host computer into a large-capacity FIFO memory, allowing the data collected by the acquisition card to be buffered by the FIFO memory and then continuously transmitted to the host memory or hard disk through the PCIe bus.

[0010] Based on the first aspect, in some embodiments of the present invention, the acquisition card is further provided with: an infinite point multiple trigger module, which is used to divide the storage space of the host computer into multiple memory sub-segments and can receive continuous trigger operations.

[0011] Based on the first aspect, in some embodiments of the present invention, the infinite point single trigger module further includes: for presetting the capacity of the FIFO memory and monitoring the status of the FIFO memory in real time, and automatically starting DMA operation to perform data transfer.

[0012] Based on the first aspect, in some embodiments of the present invention, the infinite point multiple triggering module further includes: a module for automatically storing data collected before and after each triggering into a corresponding memory subsegment.

[0013] In the second aspect, an embodiment of the present invention provides a high-speed disk streaming method based on a PCIe bus, comprising the following steps: collecting data; writing the collected data directly to the memory of a host computer in DMA mode via the PCIe bus; and then the CPU of the host computer moves the block data to the hardware cache controlled by the RAID controller, and finally stores it in the disk array.

[0014] Based on the second aspect, in some embodiments of the present invention, the above-mentioned data collection step includes: virtualizing the onboard memory of the above-mentioned host computer into a large-capacity FIFO memory, allowing the data collected by the above-mentioned acquisition card to be buffered by the FIFO memory and then continuously transmitted to the host memory or hard disk through the PCIe bus.

[0015] Based on the second aspect, in some embodiments of the present invention, the step of collecting data further includes: dividing the storage space of the host computer into a plurality of memory sub-segments that can receive continuous trigger operations.

[0016] Based on the second aspect, in some embodiments of the present invention, the above-mentioned step of collecting data also includes: pre-setting the capacity of the above-mentioned FIFO memory and monitoring the status of the FIFO memory in real time, and automatically starting DMA operation to perform data transfer.

[0017] Based on the second aspect, in some embodiments of the present invention, the step of collecting data further includes: automatically storing the data collected before and after each trigger into a corresponding memory subsegment.

[0018] The embodiments of the present invention have at least the following advantages or beneficial effects:

[0019] In the first aspect, the embodiment of the present invention provides a high-speed streaming system based on PCIe bus, which includes an acquisition card, PCIe bus, host computer, RAID controller and disk array; the acquisition card is used to acquire data, and the acquired data is directly written into the memory of the host computer in DMA mode via the PCIe bus, and then the CPU of the host computer moves the block data to the hardware cache controlled by the RAID controller, and finally stores it in the disk array. The configuration of the host computer is: memory capacity 16GB, DDR4; support 4 PCIe Gen2 X8 slots; support all slots to transmit data to PC memory in parallel; system disk SSD 256GB, DVD R / W optical drive; using Supermicro server (Super Micro) motherboard, integrated graphics card; Xeon QuadCore 2.4GHz processor, pre-installed genuine Windows7 64bit Professional Edition operating system (SP1); power supply 1280W.

[0020] Such a system allows hardware devices of different speeds to communicate without relying on a large number of CPU interrupt loads, and has a continuous data recording function; the acquisition card can continuously acquire and store data at a speed of up to 2.5GByte / s; the host computer's data storage capacity supports 1-32TByte; it supports 1-4 channels and 1-3 acquisition cards; the acquisition card can be controlled by QTCapture Lab software to achieve continuous acquisition and storage; data can be divided into multiple files of user-specified size; the user specifies the file name and storage directory; each data file has a matching configuration file.

[0021] In the second aspect, an embodiment of the present invention provides a high-speed disk streaming method based on a PCIe bus, comprising the following steps: collecting data; writing the collected data directly to the memory of a host computer in DMA mode via the PCIe bus; and then the CPU of the host computer moves the block data to the hardware cache controlled by the RAID controller, and finally stores it in the disk array.

[0022] This method allows hardware devices of different speeds to communicate without relying on a large number of CPU interrupt loads, and has a continuous data recording function; the acquisition card can continuously acquire and store data at a speed of up to 2.5GByte / s; the host computer's data storage capacity supports 1-32TByte; it supports 1-4 channels and 1-3 acquisition cards; the acquisition card can be controlled by QTCapture Lab software to achieve continuous acquisition and storage; data can be divided into multiple files according to the size specified by the user; the user specifies the file name and storage directory; each data file has a matching configuration file. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 It is a structural block diagram of an embodiment of a high-speed disk streaming system based on a PCIe bus of the present invention;

[0025] Figure 2 It is a schematic diagram of unlimited-point single-shot acquisition by an acquisition card in one embodiment of a high-speed streaming system based on a PCIe bus of the present invention;

[0026] Figure 3 It is a schematic diagram of unlimited points and multiple acquisitions by an acquisition card in one embodiment of a high-speed streaming system based on a PCIe bus of the present invention;

[0027] Figure 4 The present invention is a flowchart of a high-speed disk streaming method based on PCIe bus.

[0028] Icons: 1. Capture card; 2. Host computer; 3. RAID controller; 4. Disk array. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0030] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0032] In the embodiments provided in the present application, it should be understood that the disclosed systems and methods can also be implemented in other ways. The system embodiments are merely schematic, for example, the block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems and computer program products according to the multiple embodiments of the present application. In this regard, each box in the block diagram can represent a module, a program segment or a part of a code, and a part of a module, a program segment or a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0033] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0034] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions to enable a computer device, which can be a personal computer, a server, or a network device, etc., to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0035] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "setting" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0036] Example 1

[0037] After long-term research and practice, the inventor of the present application found that in the prior art, the data collection and transmission speed needs to be improved. In view of this, an embodiment of the present invention provides a high-speed disk streaming system based on a PCIe bus.

[0038] Please refer to Figure 1 , Figure 1 The structure diagram of a high-speed disk streaming system based on PCIe bus of the present invention is shown in FIG. The system includes: an acquisition card 1, a PCIe bus, a host computer 2, a RAID controller 3 and a disk array 4.

[0039] Exemplarily, the acquisition card 1 can be a QT11xx series acquisition card 1; exemplary, the configuration of the host computer 2 is: memory capacity 16GB, DDR4; support 4 PCIe Gen2 X8 slots; support all slots to transmit data to the PC memory in parallel; system disk SSD 256GB, DVD R / W optical drive; using Supermicro server (Super Micro) motherboard, integrated graphics card; Xeon Quad Core 2.4GHz processor, pre-installed genuine Windows7 64bit Professional Edition operating system (SP1); power supply 1280W.

[0040] The acquisition card 1 is used to acquire data. The acquired data is directly written into the memory of the host computer 2 in DMA mode via the PCIe bus. The CPU of the host computer 2 then moves the data in blocks to the hardware cache controlled by the RAID controller 3 and finally stores it into the disk array 4.

[0041] Such a system allows hardware devices of different speeds to communicate without relying on a large number of interrupt loads on the CPU, and has a continuous data recording function; the continuous acquisition and storage speed of acquisition card 1 can reach 2.5GByte / s; the data storage capacity of host computer 2 supports 1-32TByte; supports 1-4 channels and 1-3 acquisition cards 1; the acquisition card 1 can be controlled by QTCapture Lab software to achieve continuous acquisition and storage; data can be divided into multiple files according to the size specified by the user; the user specifies the file name and storage directory; each data file has a matching configuration file.

[0042] Example 2

[0043] This embodiment is further improved on the basis of Embodiment 1. The difference between this embodiment and Embodiment 1 is that:

[0044] Please refer to Figure 2 , Figure 2 The schematic diagram of the unlimited point single-shot acquisition of the acquisition card 1 in the high-speed streaming system based on PCIe bus of the present invention is shown in FIG. In the embodiment of the present invention, an unlimited point single-shot acquisition mode is provided.

[0045] The acquisition card 1 is provided with: an infinite point single trigger module, which is used to virtualize the onboard memory of the host computer 2 into a large-capacity FIFO memory, allowing the data collected by the acquisition card 1 to be buffered by the FIFO memory and continuously transmitted to the host memory or hard disk through the PCIe bus. The capacity of the FIFO memory is preset and the status of the FIFO memory is monitored in real time, and the DMA operation is automatically started to carry out data transfer. After the infinite point single acquisition of the acquisition card 1 is completed, the length of the collected data is allowed to be infinite, and the restriction condition is the memory capacity or hard disk capacity of the host.

[0046] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the unlimited point multiple acquisition of the acquisition card 1 in the high-speed streaming system based on the PCIe bus of the present invention. It provides an unlimited point multiple acquisition method.

[0047] The acquisition card 1 is also provided with: an infinite point multiple trigger module, which is used to divide the storage space of the host computer 2 into multiple memory sub-segments, which can receive continuous trigger operations and automatically store the data collected before and after each trigger into the corresponding memory sub-segment.

[0048] Infinite point single acquisition and infinite multiple acquisition can be applied to different application scenarios, and the internal working process will be different. Infinite point single acquisition and infinite multiple acquisition are two acquisition modules. Users can freely set the acquisition data length before triggering and the acquisition data length after triggering.

[0049] Example 3

[0050] This embodiment is a further improvement on the basis of Embodiment 1. The difference between this embodiment and Embodiment 1 and Embodiment 2 is that:

[0051] Please refer to Figure 4 The embodiment of the present invention provides a high-speed disk streaming method based on a PCIe bus, comprising the following steps:

[0052] S1: Collect data;

[0053] The data acquisition step includes: virtualizing the onboard memory of the host computer 2 as a large-capacity FIFO memory, allowing the data collected by the acquisition card 1 to be buffered by the FIFO memory and then continuously transmitted to the host memory or hard disk through the PCIe bus. The storage space of the host computer 2 is divided into multiple memory sub-segments, which can receive continuous trigger operations.

[0054] The step of collecting data also includes: presetting the capacity of the FIFO memory and monitoring the status of the FIFO memory in real time, and automatically starting the DMA operation to carry out data transfer, which is used to automatically store the data collected before and after each trigger into the corresponding memory sub-segment.

[0055] Infinite point single acquisition and infinite multiple acquisition can be applied to different application scenarios, and the internal working process will be different. Infinite point single acquisition and infinite multiple acquisition are two acquisition modules. Users can freely set the acquisition data length before triggering and the acquisition data length after triggering.

[0056] S2: Write the collected data directly to the memory of host computer 2 via PCIe bus in DMA mode;

[0057] DMA transfer copies data from one address space to another. When the CPU initiates the transfer, the transfer itself is performed and completed by the DMA controller 3RAID. A typical example is moving a block of external memory to a faster memory area inside the chip. Operations like this do not delay the processor's work, but can be rescheduled to handle other tasks. It should be noted that when implementing DMA transfer, the DMA controller 3RAID directly controls the bus, so there is a problem of bus control transfer. That is, before DMA transfer, the CPU must hand over the bus control to the DMA controller 3RAID, and after the DMA transfer is completed, the DMA controller 3RAID should immediately hand over the bus control back to the CPU.

[0058] S3: The CPU of the host computer 2 then moves the data blocks to the hardware cache controlled by the RAID controller 3, and finally stores them in the disk array 4.

[0059] This method allows hardware devices of different speeds to communicate without relying on a large number of CPU interrupt loads, and has a continuous data recording function; the acquisition card 1 can continuously acquire and store data at a speed of up to 2.5GByte / s; the data storage capacity of the host computer 2 supports 1-32TByte; it supports 1-4 channels and 1-3 acquisition cards 1; the acquisition card 1 can be controlled by QTCapture Lab software to achieve continuous acquisition and storage; data can be divided into multiple files according to the size specified by the user; the user specifies the file name and storage directory; each data file has a matching configuration file.

[0060] In summary, in the first aspect, the embodiment of the present invention provides a high-speed streaming system based on PCIe bus, which includes an acquisition card 1, a PCIe bus, a host computer 2, a RAID controller 3 and a disk array 4; the acquisition card 1 is used to acquire data, and the acquired data is directly written into the memory of the host computer 2 in DMA mode via the PCIe bus, and then the CPU of the host computer 2 moves the block data to the hardware cache controlled by the RAID controller 3, and finally stores it in the disk array 4. The configuration of the host computer 2 is: memory capacity 16GB, DDR4; support 4 PCIe Gen2 X8 slots; support all slots to transmit data to the PC memory in parallel; system disk SSD 256GB, DVD R / W optical drive; using Supermicro server (Super Micro) motherboard, integrated graphics card; Xeon Quad Core 2.4GHz processor, pre-installed genuine Windows 7 64bit Professional Edition operating system (SP1); power supply 1280W.

[0061] Such a system allows hardware devices of different speeds to communicate without relying on a large number of interrupt loads on the CPU, and has a continuous data recording function; the continuous acquisition and storage speed of acquisition card 1 can reach 2.5GByte / s; the data storage capacity of host computer 2 supports 1-32TByte; supports 1-4 channels and 1-3 acquisition cards 1; the acquisition card 1 can be controlled by QTCapture Lab software to achieve continuous acquisition and storage; data can be divided into multiple files according to the size specified by the user; the user specifies the file name and storage directory; each data file has a matching configuration file.

[0062] Please refer to Figure 4 In the second aspect, an embodiment of the present invention provides a high-speed disk streaming method based on a PCIe bus, comprising the following steps: collecting data; directly writing the collected data to the memory of a host computer 2 in DMA mode via the PCIe bus; and then the CPU of the host computer 2 moves the block data to the hardware cache controlled by a RAID controller 3, and finally stores it in a disk array 4.

[0063] This method allows hardware devices of different speeds to communicate without relying on a large number of CPU interrupt loads, and has a continuous data recording function; the acquisition card 1 can continuously acquire and store data at a speed of up to 2.5GByte / s; the data storage capacity of the host computer 2 supports 1-32TByte; it supports 1-4 channels and 1-3 acquisition cards 1; the acquisition card 1 can be controlled by QTCapture Lab software to achieve continuous acquisition and storage; data can be divided into multiple files according to the size specified by the user; the user specifies the file name and storage directory; each data file has a matching configuration file.

[0064] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0065] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A high-speed streaming system based on PCIe bus, characterized in that: Including acquisition card, PCIe bus, host computer, RAID controller and disk array; The acquisition card is used to acquire data, and the acquired data is directly written into the memory of the host computer in DMA mode via the PCIe bus, and then the CPU of the host computer moves the data in blocks to the hardware cache controlled by the RAID controller, and finally stores it into the disk array; The acquisition card is provided with: The infinite point single trigger module is used to virtualize the onboard memory of the host computer into a large-capacity FIFO memory, allowing the data collected by the acquisition card to be buffered by the FIFO memory and then continuously transmitted to the host memory or hard disk through the PCIe bus; The acquisition card is also provided with: An infinite-point multiple trigger module, used to divide the storage space of the host computer into multiple memory sub-segments, capable of receiving continuous trigger operations; The infinite point single trigger module also includes: Used to pre-set the capacity of the FIFO memory and monitor the status of the FIFO memory in real time, and automatically start DMA operation to carry out data transfer; The infinite point multiple triggering module also includes: Used to automatically store the data collected before and after each trigger into the corresponding memory subsegment.

2. A high-speed disk streaming method based on PCIe bus, characterized in that: The following steps are involved: Collect data; The collected data is directly written to the host computer's memory via the PCIe bus in DMA mode; The CPU of the host computer then moves the data blocks to the hardware cache controlled by the RAID controller, and finally stores them in the disk array; The onboard memory of the host computer is virtualized as a large-capacity FIFO memory, allowing the data collected by the acquisition card to be buffered by the FIFO memory and then continuously transmitted to the host memory or hard disk through the PCIe bus; Dividing the storage space of the host computer into a plurality of memory sub-segments capable of receiving continuous trigger operations; Presetting the capacity of the FIFO memory and monitoring the status of the FIFO memory in real time, and automatically starting the DMA operation to carry out data transfer; Used to automatically store the data collected before and after each trigger into the corresponding memory subsegment.