A data management system for high-speed data acquisition

Abstract

A data management system for high-speed data acquisition includes an FPGA platform equipped with a data acquisition interface module, a data management module, a data storage module, and a data transmission module. The data management module is bidirectionally interconnected with the data acquisition interface module, the data storage module, and the data transmission module via an AXI bus, thereby enabling local storage of data and long-distance, high-speed online transmission of data. For scenarios requiring the data acquisition system to operate independently or remotely without other peripherals, the data management system is provided on an FPGA board, i.e., an FPGA platform. Only one FPGA board is required for deployment in any environment and scenario, providing only power without relying on other devices. Furthermore, the system can independently perform data processing, storage, and high-speed transmission as required. The system is equipped with a non-volatile storage device to support offline operation, enabling data pre-processing and storage in a local storage device, and data extraction / transmission after completion of the operation. The system has the advantages of efficient operation, low deployment cost, and high availability.

Description

Technical Field

[0001] The present invention relates to the technical field of data communication, and in particular to a data management system for high-speed data acquisition. Background Art

[0002] Currently, common data management systems are divided into three types: computer as the main control of the management system, collection terminal + server, and independent working mode.

[0003] The use of computers as the main control system of a management system is common in cloud storage, cloud computing or big data services, and is used to integrate and calculate data from all parties, but is not very good at applications such as data collection. For example, the invention with application number CN202110226321.3, publication number CN112860955B, and titled "Business Data Management System and Method Based on Cloud Computing and Big Data" discloses a business data management system and method for managing, integrating, and processing user data. This invention can only be deployed locally as a data management system. When remote data collection is required, the data management system often needs to be deployed remotely and needs to have a certain degree of independent working capability. This invention system is not suitable for deployment in scenarios that require remote independent work.

[0004] The acquisition terminal + server is often used in a method that requires multi-point intensive acquisition. The single-point acquisition rate is not high. After the terminal collects data, it is uploaded to the server for unified processing. For example, the invention with application number CN201610952313.6, publication number CN106548440A, and name "Big Data Management Method and Big Data Management System" discloses a smart city data management system that obtains the operating data of each subsystem and transmits it to the server, and then analyzes and integrates the data. This invention can manage more acquisition terminals at the same time through the server, but the data rate supported by the acquisition terminal in this way is often low, and the server architecture is more massive, which is not suitable for deployment in remote independent work scenarios.

[0005] Independent management systems are usually used in some specific working conditions, such as when there is no network environment. The data is stored locally in the system and the data is further processed when it is needed later. For example, the invention with application number CN201620995183.X, publication number CN206117704U, and name "An integrated data management processor based on FC network" discloses a data management system based on PCIe bus and FC interface, which collects data on the PCIe bus locally and then sends it to external devices through the FC interface. When adopting the independent collection system mode, the data collection interface is often relatively fixed, and can only provide some interfaces that cannot be adjusted. It cannot flexibly adapt to different data collection tasks. Its application and scenarios are not flexible enough, and it is also not suitable for deployment in remote independent work scenarios.

[0006] High-speed data collection often encounters unexpected situations, requiring data management systems to operate independently and have error correction capabilities to handle these situations. At the same time, data management systems must be versatile enough to function effectively across a wide range of collection targets. Furthermore, data management systems must offer both local storage and high-speed online transmission capabilities to adapt to diverse working environments and conditions. Therefore, existing solutions are not directly suitable for high-speed data collection tasks requiring independent operation. Summary of the Invention

[0007] In order to overcome the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a data management system for high-speed data acquisition. For scenarios where the data acquisition system needs to work independently or remotely without other peripherals, the data management system is set on an FPGA board, i.e., an FPGA platform. Only one FPGA board is required to be deployed in any environment and scenario. It only provides power supply without relying on other equipment. At the same time, it can independently perform data processing, storage, high-speed transmission and other tasks according to needs; the system in the present invention is equipped with a non-volatile storage device to support offline work, and can pre-process and save data to a local storage device, and extract / transmit data after the work is completed. It has the advantages of efficient operation, low deployment cost and high availability.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A data management system for high-speed data acquisition includes an FPGA platform equipped with a data acquisition interface module, a data management module, a data storage module, and a data transmission module. The data management module is bidirectionally interconnected with the data acquisition interface module, the data storage module, and the data transmission module via an AXI bus, thereby enabling local storage of data and long-distance, high-speed online transmission of data.

[0010] The data acquisition interface module includes a variety of different data acquisition interface boards and data receiving submodules. The data acquired by the data acquisition interface board is transmitted unidirectionally to the data receiving submodule.

[0011] The input end of the data acquisition interface board is connected to data sources of different interfaces and data types, and the output end uses the FMC interface to connect to the FPGA, which is used to interconnect the FPGA with various high-speed peripherals.

[0012] The data receiving submodule is used to convert different types of data sources into a bus format that can be processed by the FPGA.

[0013] The data management module includes a main control processing submodule, a data checking and processing submodule, a data caching submodule, and a queue management submodule; wherein the main control processing submodule is bidirectionally interconnected with the data checking and processing submodule, the data caching submodule, and the queue management submodule.

[0014] The main control processing submodule is used to control the working logic, data storage and data transmission of the entire system, including:

[0015] ① Determine whether to enable offline storage based on the data acquisition rate: When the data acquisition rate exceeds the negotiated rate of online transmission, offline storage is enabled; otherwise, only online transmission is enabled;

[0016] ② Determine whether to enable online transmission based on the device's online status: When the FPGA platform equipped with the data acquisition interface module, data management module, data storage module, and data transmission module is successfully connected to the host computer, online transmission is enabled; otherwise, only offline storage is enabled;

[0017] ③ Whether to enable data preprocessing and management of data preprocessing methods: There are two situations: one is that the user specifies to enable data preprocessing including compression through the configuration register; the other is that the data acquisition rate exceeds the upper limit of the sum of online transmission and offline storage rates, and data preprocessing including compression is automatically enabled;

[0018] ④ Scheduling of all modules and submodules except the data acquisition interface board and the main control processing submodule: First, control whether each module is enabled based on the current data acquisition rate; second, distribute the contents of the user-configurable registers to all modules of this system; finally, adjust the working mode of the data cache module, including ping-pong BUFFER or FIFO form, based on the current data acquisition rate and data type (small data packet / large data packet).

[0019] The data checking and processing submodule is used to receive and process the data transmitted by the data receiving submodule; the processing process is as follows: checking the data transmitted by the data receiving submodule to ensure the integrity and correctness of the received data; transmitting and storing the data according to the user's requirements and register configuration, and performing preprocessing work including compression, encoding, and feature value extraction before storage.

[0020] The data cache submodule includes a high-speed volatile memory for caching the collected data.

[0021] The interface of the queue management submodule is connected to the data storage module and the data transmission module through the data management module, so that the data storage module and the data transmission module transmit according to the queue, ensuring the integrity of the data during the transmission process and avoiding packet loss during the data transmission process; in addition, the generation of the data table in the volatile storage of the data cache submodule is maintained.

[0022] The data storage module includes a bidirectionally interconnected non-volatile storage medium submodule and a storage instruction control submodule;

[0023] Among them, the non-volatile storage medium submodule includes a solid-state drive (SSD), a mechanical hard disk (HDD), and a storage array storage medium. The non-volatile storage medium submodule is connected to the FPGA through an FMC interface expansion card;

[0024] The storage instruction control submodule is responsible for determining the data storage protocol: identifying different connection handshake signals for different protocols; and controlling the data storage protocol: enabling different protocol stacks and disabling other protocol stacks when device connections with different protocols are identified; and packaging the data into various storage user layer protocols, and then storing the data in non-volatile storage media.

[0025] The data transmission module includes a bidirectionally interconnected network protocol stack submodule and a network connection control submodule;

[0026] Among them, the network protocol stack submodule uses RDMA (remote direct memory access) to transmit the collected data to the host computer at high speed through the QSFP interface;

[0027] Among them, the network connection control submodule is used to control the connection between this data management system and the remote server: after the QSFP interface detects that the module is inserted and the physical layer connection has been established, the network connection control submodule autonomously starts the connection establishment process, actively sends a connection request (REQ) packet, and then, after receiving the host computer connection reply (REP) agreeing to the connection, it actively sends a ready to use (RTU) packet, and the connection is established at this time; after the connection is established, the network connection control submodule actively requests to obtain the data transmission parameters of the host computer, including memory address, memory size, access key, and prepare for data transmission; at the same time, after the connection is established, it continuously monitors the control packets sent by the host computer, including adding queues, reducing queues, disconnecting, and updating parameters; it also receives control from the main control processing submodule in the data management module, and performs operations including adding queues, reducing queues, disconnecting, and updating parameters.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. For scenarios requiring independent data acquisition system operation or remote operation without other peripherals, the data management system of this invention is deployed on an FPGA board. This single board can be deployed in any environment and scenario, requiring only power and without relying on other devices. It can independently perform data processing, storage, and high-speed transmission as needed. The use of replaceable data acquisition interface boards significantly reduces equipment purchase costs, improves system versatility, and reduces maintenance costs.

[0030] 2. In scenarios where it is inconvenient to deploy a network or to use manpower, the system of the present invention is equipped with a non-volatile storage device to support offline work. It can work completely independently and store data, and extract / transmit data after the work is completed, which is convenient for data collection work in harsh environments. At the same time, some user operation interfaces are reserved. By abstracting the user interface, the difficulty of using the system is reduced while the ease of use is improved, which facilitates the deployment of the system.

[0031] 3. To address the problems of storage device failure caused by harsh environments such as high-energy particle environments, the present invention can also process, integrate, and compress data in real time through the data management module in the system and transmit it to the server in real time to avoid loss of collected data.

[0032] 4. When performing extremely high-speed data acquisition, the present invention can simultaneously store data locally and transmit it online to meet high-speed acquisition requirements. After the acquisition is completed or the data source rate drops, the local data is uploaded back to the remote server, further improving the system's throughput, reaching a maximum throughput of over 200Gbps.

[0033] 5. The system of the present invention uses optical fiber to communicate with a remote host computer, which has less restriction on transmission distance and can flexibly select the rate, volume, capacity, interface type, etc. of the storage medium, further improving the usability of the system and reducing the cost of use, and efficiently realizing long-distance high-speed data transmission.

[0034] In summary, the data management system of the present invention has both local storage capabilities and high-speed online transmission capabilities, so as to adapt to different working environments and working conditions. It can be directly applied to high-speed data acquisition work that requires independent operation, and has the advantages of high efficiency, low deployment cost and high availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to illustrate the technical solution of the present invention more clearly and effectively, the present invention is further described below with reference to the accompanying drawings. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 It is a relationship diagram of the system function modules of the present invention.

[0037] Figure 2 It is the data management module of the present invention.

[0038] Figure 3 It is a typical deployment diagram of the present invention.

[0039] Figure 4 This is a typical working principle diagram of the present invention.

[0040] Figure 5 It is an online transmission workflow diagram of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0042] like Figure 1 As shown, a data management system for high-speed data acquisition includes an FPGA platform, on which a data acquisition interface module, a data management module, a data storage module, and a data transmission module are mounted; wherein the data management module is bidirectionally interconnected with the data acquisition interface module, the data storage module, and the data transmission module via an AXI bus.

[0043] The FPGA is a programmable chip.

[0044] The AXI bus is an on-chip data transmission bus with a handshake mechanism.

[0045] The data acquisition interface module includes a variety of different data acquisition interface boards and data receiving submodules. The data acquired by the data acquisition interface board is transmitted unidirectionally to the data receiving submodule.

[0046] The input end of the data acquisition interface board is connected to data sources of different interfaces and data types, and the output end uses the FMC interface to connect to the FPGA. The FMC interface is a standard FPGA mezzanine card used to interconnect the FPGA with various high-speed peripherals.

[0047] The data receiving submodule is used to convert different types of data sources into a bus format that can be processed by the FPGA.

[0048] like Figure 2As shown, the data management module includes a main control processing submodule, a data inspection and processing submodule, a data cache submodule, and a queue management submodule; wherein the main control processing submodule is bidirectionally interconnected with the data inspection and processing submodule, the data cache submodule, and the queue management submodule. The main control processing submodule is used to control the working logic, data storage, and data transmission of the entire system, and includes:

[0049] ① Determine whether to enable offline storage based on the data acquisition rate: When the data acquisition rate exceeds the negotiated rate of online transmission, offline storage is enabled; otherwise, only online transmission is enabled;

[0050] ② Determine whether to enable online transmission based on the device's online status: When the FPGA platform equipped with the data acquisition interface module, data management module, data storage module, and data transmission module is successfully connected to the host computer, online transmission is enabled; otherwise, only offline storage is enabled;

[0051] ③ Whether to enable data preprocessing and management of data preprocessing methods: There are two situations: one is that the user specifies to enable data preprocessing including compression through the configuration register; the other is that the data acquisition rate exceeds the upper limit of the sum of online transmission and offline storage rates, and data preprocessing including compression is automatically enabled;

[0052] ④ Scheduling of all modules and submodules except the data acquisition interface board and the main control processing submodule: First, control whether each module is enabled based on the current data acquisition rate; second, distribute the contents of the user-configurable registers to all modules of this system; finally, adjust the working mode of the data cache module, including ping-pong BUFFER or FIFO form, based on the current data acquisition rate and data type (small data packet / large data packet).

[0053] The data checking and processing submodule is used to receive and process the data transmitted by the data receiving submodule; the processing process is as follows: checking the data transmitted by the data receiving submodule to ensure the integrity and correctness of the received data; transmitting and storing the data according to the user's requirements and register configuration, and performing preprocessing work including compression, encoding, and feature value extraction before storage.

[0054] The data cache submodule includes a high-speed volatile memory for caching the collected data; it facilitates the subsequent data processing and the work of the queue control module, and also leaves space for subsequent data storage and transmission.

[0055] The interface of the queue management submodule is connected to the data storage module and the data transmission module through the data management module, so that the data storage module and the data transmission module transmit according to the queue, ensuring the integrity of the data during the transmission process and avoiding packet loss during the data transmission process; in addition, the generation of the data table in the volatile storage of the data cache submodule is maintained.

[0056] The data storage module includes a bidirectionally interconnected non-volatile storage medium submodule and a storage instruction control submodule;

[0057] Among them, the non-volatile storage medium submodule includes a solid-state drive (SSD), a mechanical hard disk (HDD), and a storage array storage medium. The non-volatile storage medium submodule is connected to the FPGA through an FMC interface expansion card;

[0058] The storage instruction control submodule is responsible for determining the data storage protocol: identifying different connection handshake signals for different protocols; and controlling the data storage protocol: enabling different protocol stacks and disabling other protocol stacks when device connections with different protocols are identified; and packaging the data into various storage user layer protocols, and then storing the data in non-volatile storage media.

[0059] The data transmission module includes a bidirectionally interconnected network protocol stack submodule and a network connection control submodule;

[0060] Among them, the network protocol stack submodule uses RDMA (remote direct memory access) to transmit the collected data to the host computer at high speed through the QSFP interface;

[0061] Among them, the network connection control submodule is used to control the connection between this data management system and the remote server: after the QSFP interface detects that the module is inserted and the physical layer connection has been established, the network connection control submodule autonomously starts the connection establishment process, actively sends a connection request (REQ) packet, and then, after receiving the host computer connection reply (REP) agreeing to the connection, it actively sends a ready to use (RTU) packet, and the connection is established at this time; after the connection is established, the network connection control submodule actively requests to obtain the data transmission parameters of the host computer, including memory address, memory size, access key, and prepare for data transmission; at the same time, after the connection is established, it continuously monitors the control packets sent by the host computer, including adding queues, reducing queues, disconnecting, and updating parameters; it also receives control from the main control processing submodule in the data management module, and performs operations including adding queues, reducing queues, disconnecting, and updating parameters.

[0062] The QSFP interface is a high-speed optical fiber interface.

[0063] like Figure 3As shown, a typical deployment of this system involves connecting the FPGA signal input side to a data acquisition interface card via an FMC interface. This card is available in various models and can be flexibly adjusted to suit different data source types. The storage side also connects to non-volatile storage media via the FMC interface, which can also be flexibly adjusted based on needs, but a solid-state storage array is recommended. The data transmission side uses a QSFP fiber optic interface for high-speed data transmission, transferring collected and processed data to a host computer. The system also requires a 12V / 2A power supply. With this deployment, the system can operate completely independently or offline.

[0064] like Figure 4 As shown, the typical workflow of this system is that after the data source enters this system, the data will first be pre-processed, and then will be cached in volatile storage. At this time, it will be judged based on the connection status between the system and the host computer and the rate of the data source: if the system is offline, all collected data will be stored offline; if the system is online, online transmission must be started, and RDMA will be used for transmission. At this time, if the data source rate exceeds the online transmission throughput, offline storage will be enabled at the same time, and the data that exceeds the throughput range will be stored in non-volatile storage. After the acquisition task is completed, all data in the offline storage will be retransmitted to the host computer after the system is online. It should be emphasized that in this system, all work processes are CPU-free, which means that each step in such a process is running at any time, rather than in a sequential order. Figure 4 The workflow shown in is for ease of understanding only.

[0065] like Figure 5 As shown in the figure, the typical online transmission workflow of this system is as follows: once the system is online and the physical connection is clear, it first initiates an RDMA handshake request to the host computer. After the host computer agrees to the connection, it publishes a connection available network packet, at which point network transmission can begin. Before officially transmitting the data packet, the system first notifies the host computer that the transmission is about to begin. After receiving the response, the data packet transmission begins. At this time, the data packet is sent directly to the host computer's memory to ensure the link rate. After the data packet transmission is completed, it waits for the host computer's response signal. If no response signal is received from the host computer, the packet is considered lost and the data packet transmission process is restarted.

[0066] Simulation experiment

[0067] To evaluate the performance of the present invention, a simulation was conducted. Based on the system, a simulated data source was constructed, which can generate data sources with different interface types and flexibly adjust the rate. Furthermore, a simulated host computer was constructed, which can simulate the host computer's online and offline states and different network interface rates. A simple model of a virtual non-volatile storage medium was also constructed to simulate data storage.

[0068] First, a functional verification is performed: when switching different data source interface types, the present invention can normally identify the interface type and use appropriate logic to communicate with the simulated data source and perform data pre-processing; when switching different types of non-volatile storage medium models, the present invention can also correctly identify and switch to the protocol stack of the corresponding protocol. Then a performance verification is performed: when the simulated data source is started and the system is working: 1. When the rate of the simulated data source is less than 100Gbps, the present invention only starts the online transmission corresponding module to work. At this time, if the host computer status is switched to offline, it can be found that the present invention automatically switches to the offline storage working mode, and continuously tries to re-establish the connection with the host computer. 2. When the rate of the simulated data source is between 100-150Gbps, the present invention enables online transmission and offline storage at the same time. After a period of time, the simulated data source is turned off, and the present invention automatically reads the stored data from the non-volatile medium, labels it in the order of data acquisition, and sends it to the host computer. 3. When the rate of the analog data source is between 150-210Gbps, the present invention forces the start of data compression processing, online transmission and offline storage. All collected data must be encoded and compressed before transmission or storage, and the encoding method adopted by the host computer is notified at the same time. In addition, the working logic is the same as above. When starting data compression, the compression capability of the invention varies depending on the type of data source, but the upper limit of the processing rate is guaranteed to be no less than 210Gbps. 4. When the rate of the analog data source is greater than 210Gbps, it has exceeded the processing rate limit of the present invention. At this time, the present invention will be based on the options of the user-configurable registers. 1) Default item: Enable downsampling, and discard the sampled data according to the real-time rate and processing capacity ratio to ensure normal operation 2) Enable a fixed data transmission rate, such as 150Gbps, which is determined by the user. Data packets exceeding this data rate will be discarded. 3) The system stops working. The rest of the working logic is the same as above.

[0069] Through simulation experiments, it can be seen that 1. The present invention can meet the conditions for normal operation under different data acquisition sources. 2. The present invention can work normally when equipped with different types of non-volatile storage media. 3. The present invention can automatically adjust the working mode according to different working conditions and can work independently without supervision. 4. The theoretical acceptable data throughput of the present invention is not less than 150Gbps, and when data compression encoding is enabled, it is not less than 210Gbps. 5. The present invention highly abstracts the user interface to further improve user usability.

[0070] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A data management system for high-speed data acquisition, comprising an FPGA platform, characterized in that: The FPGA platform is equipped with a data acquisition interface module, a data management module, a data storage module, and a data transmission module; wherein the data management module is bidirectionally interconnected with the data acquisition interface module, the data storage module, and the data transmission module via an AXI bus, thereby realizing local data storage and long-distance, high-speed online data transmission; the data management module includes a main control processing submodule, a data inspection and processing submodule, a data cache submodule, and a queue management submodule; wherein the main control processing submodule is bidirectionally interconnected with the data inspection and processing submodule, the data cache submodule, and the queue management submodule; The main control processing submodule is used to control the working logic, data storage and data transmission of the entire system, including: ① Determine whether to enable offline storage based on the data acquisition rate: When the data acquisition rate exceeds the negotiated rate of online transmission, offline storage is enabled; otherwise, only online transmission is enabled; ② Determine whether to enable online transmission based on the device's online status: When the FPGA platform equipped with the data acquisition interface module, data management module, data storage module, and data transmission module is successfully connected to the host computer, online transmission is enabled; otherwise, only offline storage is enabled; ③ Whether to enable data preprocessing and management of data preprocessing methods: There are two situations: one is that the user specifies to enable data preprocessing including compression through the configuration register; the other is that the data acquisition rate exceeds the upper limit of the sum of online transmission and offline storage rates, and data preprocessing including compression is automatically enabled; ④ Scheduling of all modules and submodules except the data acquisition interface board and the main control processing submodule: First, control whether each module is enabled based on the current data acquisition rate; second, distribute the contents of the user-configurable registers to all modules of this system; finally, adjust the working mode of the data cache module, including ping-pong BUFFER or FIFO form, based on the current data acquisition rate and data type, i.e. small data packet / large data packet.

2. A data management system for high-speed data acquisition according to claim 1, characterized in that: The data acquisition interface module includes a variety of different data acquisition interface boards and data receiving submodules. The data acquired by the data acquisition interface board is transmitted unidirectionally to the data receiving submodule. The input end of the data acquisition interface board is connected to data sources of different interfaces and data types, and the output end uses the FMC interface to connect to the FPGA, which is used to interconnect the FPGA with various high-speed peripherals. The data receiving submodule is used to convert different types of data sources into a bus format that can be processed by the FPGA.

3. A data management system for high-speed data acquisition according to claim 1, characterized in that: The data checking and processing submodule is used to receive and process the data transmitted by the data receiving submodule; the processing process is as follows: checking the data transmitted by the data receiving submodule to ensure the integrity and correctness of the received data; transmitting and storing the data according to the user's requirements and register configuration, and performing preprocessing work including compression, encoding, and feature value extraction before storage.

4. A data management system for high-speed data acquisition according to claim 1, characterized in that: The data cache submodule includes a high-speed volatile memory for caching the collected data.

5. A data management system for high-speed data acquisition according to claim 1, characterized in that: The interface of the queue management submodule is connected to the data storage module and the data transmission module through the data management module, so that the data storage module and the data transmission module transmit according to the queue, ensuring the integrity of the data during the transmission process and avoiding packet loss during the data transmission process; in addition, the generation of the data table in the volatile storage of the data cache submodule is maintained.

6. A data management system for high-speed data acquisition according to claim 1, characterized in that: The data storage module includes a bidirectionally interconnected non-volatile storage medium submodule and a storage instruction control submodule; Among them, the non-volatile storage medium submodule includes a solid-state drive (SSD), a mechanical hard disk (HDD), and a storage array storage medium. The non-volatile storage medium submodule is connected to the FPGA through an FMC interface expansion card. The storage instruction control submodule is responsible for determining the data storage protocol: identifying different connection handshake signals for different protocols; and controlling the data storage protocol: enabling different protocol stacks and disabling other protocol stacks when device connections with different protocols are identified; and packaging the data into various storage user layer protocols, and then storing the data in non-volatile storage media.

7. A data management system for high-speed data acquisition according to claim 1, characterized in that: The data transmission module includes a bidirectionally interconnected network protocol stack submodule and a network connection control submodule; Among them, the network protocol stack submodule uses RDMA (remote direct memory access) to transmit the collected data to the host computer at high speed through the QSFP interface; Among them, the network connection control submodule is used to control the connection between this data management system and the remote server: after the QSFP interface detects that the module is inserted and the physical layer connection has been established, the network connection control submodule autonomously starts the connection establishment process, actively sends a connection request REQ packet, and then, after receiving the host computer connection reply REP agreeing to the connection, it actively sends a ready to use RTU packet, and the connection is established at this time; after the connection is established, the network connection control submodule actively requests to obtain the data transmission parameters of the host computer, including memory address, memory size, and access key, to prepare for data transmission; at the same time, after the connection is established, it continuously monitors the control packets sent by the host computer, including adding queues, reducing queues, disconnecting, and updating parameters; it also receives control from the main control processing submodule in the data management module, and performs operations including adding queues, reducing queues, disconnecting, and updating parameters.

Images