Data migration method and device

Abstract

This application provides a data transfer method and apparatus. The method includes: acquiring sampled data to be stored; converting the sampled data into multiple data files in a preset format, and storing the multiple data files into multiple storage areas in a designated memory; receiving read requests through a first communication interface of a QDMA-based direct memory access subsystem, and reading the data files indicated by the read requests from the multiple storage areas; and sending the read data files indicated by the read requests to a storage server through a second communication interface of the QDMA subsystem. This application solves the technical problem of low transfer efficiency during the data transfer process.

Description

Technical Field

[0001] This application relates to the field of data transfer technology, and more specifically, to a data transfer method and apparatus. Background Technology

[0002] With the rapid development of digitalization and informatization, sensors, radars, and other monitoring devices are ubiquitous in our lives, constantly generating massive amounts of data. Simultaneously, the rapid advancement of electronic and digital technologies has led to increasingly faster data acquisition systems and larger storage capacities, placing higher demands on the performance of data transfer systems. Summary of the Invention

[0003] This application provides a data transfer method and apparatus to at least solve the technical problem of low transfer efficiency during the data transfer process.

[0004] According to one embodiment of this application, a data transfer method is provided, comprising: acquiring sampled data to be stored; converting the sampled data into multiple data files in a preset format, and storing the multiple data files respectively in multiple storage areas in a specified memory; receiving a read request through a first communication interface of QDMA (Queue-based Direct Memory Access), and reading the data file indicated by the read request from the multiple storage areas; and sending the read data file indicated by the read request to a storage server through a second communication interface of the QDMA subsystem.

[0005] In an exemplary embodiment, storing the plurality of data files into a plurality of storage areas in a designated memory includes: determining a plurality of unused storage areas from the plurality of storage areas; processing the plurality of data files and storing them sequentially into the plurality of unused storage areas; and outputting a status table indicating the storage status of the plurality of storage areas and a descriptor table indicating the data storage information in the plurality of storage areas, wherein the data files correspond one-to-one with the unused storage areas.

[0006] In an exemplary embodiment, after processing the plurality of data files, they are sequentially stored in the plurality of unused storage areas, including: removing invalid information from the frame header corresponding to the first frame of data in the data file, and removing the frame headers of subsequent data in the data file to complete the processing of the data file; and storing the processed data file in the corresponding unused storage area.

[0007] In an exemplary embodiment, outputting a status table indicating the storage status of the plurality of storage areas and a descriptor table indicating data storage information in the plurality of storage areas includes: determining a plurality of entries in the descriptor table, wherein each entry in the descriptor table includes at least: the starting address of the data storage, the length of the data storage, and the position of the next data segment; determining a plurality of entries in the status table, wherein each entry in the status table includes at least: a descriptor index, a read / write status, and a data identifier, wherein the descriptor index is used to represent the label of the descriptor, the read / write status is used to indicate whether the descriptor is used, and the data identifier is used to indicate the data belonging to the descriptor; and determining the information corresponding to each entry in the descriptor table and the status table based on the storage status of the plurality of storage areas and the data storage information in the plurality of storage areas.

[0008] It should be noted that data attribution is used to indicate which position of data within a data file it belongs to, such as the header, middle, or tail of the data file.

[0009] In an exemplary embodiment, reading the data file indicated by the read request from the plurality of storage areas includes: traversing the information corresponding to the read / write status table entries and data identifier table entries in the status table to determine the first descriptor corresponding to the data file indicated by the read request; and sequentially retrieving multiple segments of data from the data file based on the data storage information indicated by the descriptor, starting from the first descriptor.

[0010] In an exemplary embodiment, starting from the first descriptor, multiple segments of data in the data file are retrieved sequentially based on the data storage information indicated by the descriptor. This includes: starting from the first descriptor, sequentially determining the starting address of the data storage corresponding to each descriptor, the length of the data storage, and the position of the next segment of data; reading data of the corresponding length of data from the starting address of the data storage corresponding to each descriptor, and then selecting the starting address of the data storage of the next descriptor according to the position of the next segment of data corresponding to each descriptor to read data until all data of the data identifier mark has been read.

[0011] In an exemplary embodiment, obtaining the sampled data to be stored includes: after detecting a synchronization header in the sensor data stream, and if the synchronization header is detected consecutively a preset number of times at a preset position in the data stream, caching the sensor data stream to obtain the sampled data.

[0012] According to another embodiment of this application, a data transfer device is provided, comprising: an acquisition module for acquiring sampled data to be stored; a storage module for converting the sampled data into multiple data files of a preset format and storing the multiple data files into multiple storage areas in a specified memory; a reading module for receiving a read request through a first communication interface of a QDMA queue-based direct memory access subsystem and reading the data file indicated by the read request from the multiple storage areas; and a sending module for sending the read data file indicated by the read request to a storage server through a second communication interface of the QDMA subsystem.

[0013] According to yet another embodiment of this application, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0014] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein a computer program is stored in the memory and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0015] By converting the sampled data into a data file and storing it in a designated memory, and then directly transferring the data file to the storage server, the data processing time is saved by avoiding the need to process the data after reading it. At the same time, QDMA is used to complete the data reading and transfer the data to the storage server, which improves the data transfer bandwidth and achieves the effect of improving the transfer efficiency during the data transfer process. Attached Figure Description

[0016] Figure 1 This is a hardware structure block diagram of a computer terminal for a data transfer method according to an embodiment of this application;

[0017] Figure 2 This is a flowchart of a data transfer method according to an embodiment of this application;

[0018] Figure 3 This is a schematic diagram of a data transfer system according to an embodiment of this application;

[0019] Figure 4 This is a schematic diagram of another data transfer system according to an embodiment of this application;

[0020] Figure 5 This is a schematic diagram of a data file storage process according to an embodiment of this application;

[0021] Figure 6 This is a schematic diagram of a frame synchronization state transition according to an embodiment of this application;

[0022] Figure 7 This is a schematic diagram of a data transfer device according to an embodiment of this application. Detailed Implementation

[0023] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0025] The methods and embodiments provided in this application can be executed on a computer terminal or similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for a data transfer method according to an embodiment of this application. For example... Figure 1 As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0026] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the network transmission method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0027] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0028] This embodiment provides a data transfer method running on the aforementioned computer terminal. Figure 2 This is a flowchart based on an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0029] Step S202: Obtain the sampled data to be stored;

[0030] Step S204: Convert the sampled data into multiple data files in a preset format, and store the multiple data files into multiple storage areas in a specified memory.

[0031] Step S206: Receive a read request through the first communication interface of the QDMA queue-based direct memory access subsystem, and read the data file indicated by the request from multiple storage areas;

[0032] Step S208: The data file indicated by the read request is sent to the storage server through the second communication interface of the QDMA subsystem.

[0033] It should be noted that the specified memory can be DDR4 (Double Data Rate 4, fourth-generation double data rate synchronous dynamic random access memory). In related technologies, the data transfer system uses FIFO / DDR3 to buffer sensor data in a ping-pong manner. Then, the software reads the data into the HOST device via DMA (Direct Memory Access) operations. The data is processed on the HOST device to form a file, and finally, the file is transferred to the storage device. Using FIFO (First In First Out) for data buffering is limited by FPGA resources, which restricts the amount of data that can be transferred in bursts via DMA. The upper limit of DDR3 clock frequency also limits the data transfer rate. DMA can only read data from one FIFO at a time, and cannot transfer data from multiple files simultaneously. The software needs to process the read data again to form a file, resulting in low data transfer and storage efficiency.

[0034] This application converts the sampled data into a data file and stores it in a designated memory. Then, it directly transmits the data file to the storage server, avoiding data processing after reading the sampled data and saving data processing time. At the same time, it uses QDMA to complete the data reading and data transmission to the storage server, which improves the data transmission bandwidth and achieves the effect of improving the transfer efficiency during the data transfer process.

[0035] In practical applications, after storing the data file in the designated storage and processing the data file, the host device reads the data file to be transferred via QDMA and then transfers it to the storage server via QDMA.

[0036] It should be further explained that the first communication interface includes: the AXI4 interface (a communication interface), and the second communication interface includes: the AXIS interface (a communication interface).

[0037] Figure 3 A schematic diagram of a data transfer system is shown, such as... Figure 3 As shown, the system includes a data acquisition unit, a data transfer device, and a storage server. The data acquisition unit uses various sensors to collect data and sends the data to the data transfer device via a high-speed serial transmission interface. The sensor data frame structure consists of a frame header, valid data, and a checksum. The frame header contains a synchronization header, frame information, device information, and time information. The data transfer device processes the received sampled data, stores it in a file format, and then transfers the data file to the storage server via a network transmission protocol.

[0038] Figure 4 Another data transfer system diagram is shown, such as Figure 4 As shown, it includes: a HOST device and a data transfer device, wherein the data transfer device includes: a QDMA subsystem, an interface processing module, a network frame data buffer module, an Ethernet MAC module, a file management module, a sensor data buffer module, a sensor data receiving module, a DDR controller, a DDR chip, and an AXIL interconnect module.

[0039] The following is based on Figure 4 The data transfer system shown illustrates the data transfer method provided in this application.

[0040] QDMA operates by receiving instructions from the host operating system. QDMA provides AXIL, AXI4, and AXIS interfaces, allocating queues as resources to multiple PCIe Physical Functions (PFs) and Virtual Functions (VFs) for use by the host system. The host accesses the FPGA's internal registers through the QDMA's AXIL interface to configure and read the status of relevant modules. The host implements network interface functionality through the QDMA's AXIS interface, transferring files to a storage server via the NFS (Network File System) protocol. Figure 4 For example, data is transmitted to the storage server through the interface management module, the network frame data caching module, and the Ethernet MAC module.

[0041] Figure 4 The file management module also includes a BRAM (Block Resource RAM) management table for generating data files. The 100G optical module and the 40G optical module are interfaces for connecting to external devices, and the DDR chip is a designated storage device.

[0042] The data acquisition unit and the data transfer device communicate via a sensor data receiving module that uses the Aurora protocol (a serial transmission protocol) for data exchange. The Aurora protocol is relatively simple and customizable, controlling only the link and physical layers. Based on the defined frame format, the data transfer device can accurately and quickly receive sensor data. The sensor data receiving module decodes the sensor data to form an AXIS data stream, which is stored in the sensor data buffer module and then passed to the file management module. The file management module processes the data stream, removing redundant information, converting the data interface from AXIS to AXI4, and storing the collected data in DDR according to file organization. Simultaneously, it provides the storage status of the files in DDR to the host device (accessed via the AXIL interface) in the form of a status table. When the host detects data that needs to be read, it initiates a DMA read request to the DDR via the AXI4 interface, reading the data from the DDR to form a data file. Once a file to be sent is read, it can be sent to the storage server via the QDMA AXIS interface.

[0043] Steps S202 to S208 are described in detail below through examples.

[0044] In step S204, the specific method of storing the multiple data files into multiple storage areas in the specified memory includes: determining multiple unused storage areas from the multiple storage areas; processing the multiple data files and storing them sequentially into the multiple unused storage areas; and outputting a status table indicating the storage status of the multiple storage areas and a descriptor table indicating the data storage information in the multiple storage areas, wherein the data files correspond one-to-one with the unused storage areas.

[0045] In an exemplary embodiment, after processing the plurality of data files, they are sequentially stored in the plurality of unused storage areas. Specifically, this includes: removing invalid information from the frame header corresponding to the first frame of data in the data file, and removing the frame headers of subsequent data in the data file to complete the processing of the data file; and storing the processed data file in the corresponding unused storage area.

[0046] In practical application scenarios, the data storage process in data files is as follows: Figure 5As shown, the file management module manages the sensor-received data in the DDR storage space according to a file organization format. After reading data from the sensor data cache module, it first enters the "address selection" state, selects an unused storage area from the DDR storage space, and begins the data processing and storage process. For the first frame of data stored in the file, some useful information in the frame header is retained, and then the data portion is received and verified. For subsequent data frames, the frame header is directly discarded in the "process frame header" state, and then the "store data" state and "perform verification" state are entered to complete the data storage and verification. Upon receiving the last frame of data in the file or after the sensor data transmission is interrupted, the storage of this file ends. The state machine enters the "address selection" state, selects a new unused storage area, and begins the storage process for a new file.

[0047] In an exemplary embodiment, outputting a status table indicating the storage status of the plurality of storage areas and a descriptor table indicating data storage information in the plurality of storage areas includes: determining a plurality of entries in the descriptor table, wherein each entry in the descriptor table includes at least: the starting address of the data storage, the length of the data storage, and the position of the next data segment; determining a plurality of entries in the status table, wherein each entry in the status table includes at least: a descriptor index, a read / write status, and a data identifier, wherein the descriptor index is used to represent the label of the descriptor, the read / write status is used to indicate whether the descriptor is used, and the data identifier is used to indicate the data belonging to the descriptor; and determining the information corresponding to each entry in the descriptor table and the status table based on the storage status of the plurality of storage areas and the data storage information in the plurality of storage areas.

[0048] In one optional approach, the file management module divides the DDR storage space into multiple storage regions for management, with each region managed using multiple descriptors. Taking a 4GB DDR storage space as an example, dividing it into 512MB regions results in 8 storage regions, each corresponding to a data file. Each storage region is managed using 16 descriptors, with each descriptor managing a maximum of 32MB of sub-storage regions. The file management module maintains two tables in its BRAM (Block Resource Area): a "Descriptor Table" and a "Status Table." The "Descriptor Table" manages the DDR storage space, indicating basic information for each segment. Each entry in the "Descriptor Table" consists of the "starting address of the data storage (addr)," the "length of the data storage (len)," and the "position (index) of the next data segment (next)." The "Status Table" indicates the usage of descriptors and consists of the "descriptor index (idx)," the "read / write status (state)," and the "data identifier (flag)." The meanings of each item are shown in Table 1.

[0049]

[0050] Table 1

[0051] In another alternative approach, unused storage areas can be identified by determining that the storage area is unused if all descriptors in the storage area are in an unused state (e.g., all descriptors in the entire storage area have a state of 0).

[0052] In practical applications, the data file storage process is as follows: When storing data for a new file, the state information in the "state table" is polled first to find an unused storage area (where the state of the entire storage area is 0). Data is written to this storage area, and the state of the corresponding descriptor for that storage area is set to 1. Simultaneously, flag information is set according to the position of this data in the file. The addr (starting address of data storage), len (length of data storage), and next (position of the next data segment) information are updated based on the data storage status. Then, the next descriptor is selected to continue storing data until the entire file is stored. If a file is stored in the same storage area and its size exceeds 512MB, data will continue to be written to this storage area after the host reads the data and releases the storage space. If all DDR storage areas are found to be occupied, an alarm message is generated. Alarms can be generated in various ways, such as pop-ups, sending alarm messages, and audible and visual alarms, to notify the host to handle the situation.

[0053] In an exemplary embodiment, the process of reading the data file indicated by the read request from the plurality of storage areas includes: traversing the information corresponding to the read / write status table entries and data identifier entries in the status table to determine the first descriptor corresponding to the data file indicated by the read request; and sequentially retrieving multiple segments of data from the data file based on the data storage information indicated by the descriptor, starting from the first descriptor.

[0054] In one alternative approach, the starting address of the data storage, the length of the data storage, and the position of the next segment of data corresponding to each descriptor can be determined sequentially, starting from the first descriptor; data of the length corresponding to the data storage of each descriptor can be read from the starting address of the data storage of each descriptor, and then the starting address of the data storage of the next descriptor can be selected according to the position of the next segment of data corresponding to each descriptor to read data until all the data of the data identifier mark has been read.

[0055] Specifically, when reading data, the HOST polls the state and flag information of each entry in the "status table" to find the first descriptor of each file. The HOST creates a new file, generates a QDMA DMA descriptor based on the information, retrieves the data of length len from the corresponding addr, and then reads data from the address of the next descriptor based on the next information, until the last segment of data indicated by the flag is read, and then ends the reading of this file. For example, after reading all the header data of a data file, such as the data with a flag value of 01 in Table 2, after reading the data of a descriptor, the state of this descriptor is set to 0, and the storage space is released. If the HOST does not find the data to be read for a long time, an alarm message is also generated.

[0056] Tables 2 and 3 show a state table and a descriptor table, respectively.

[0057]

[0058]

[0059] Table 2

[0060]

[0061] Table 3

[0062] In an exemplary embodiment, obtaining the sampled data to be stored includes: after detecting a synchronization header in the sensor data stream, and if the synchronization header is detected consecutively a preset number of times at a preset position in the data stream, caching the sensor data stream to obtain the sampled data.

[0063] The sensor receiving module first instantiates the Aurora IP to receive sensor data, and then performs frame synchronization state detection. Its state transition diagram is as follows: Figure 6 As shown. After system reset, it enters the "idle" state. Upon detecting a synchronization header in the data stream, it enters the "pre-synchronization" state. If no synchronization header is detected at a predetermined data stream position in the "pre-synchronization" state, it returns to the "idle" state; however, if a synchronization header is detected three times consecutively at the predetermined position, it enters the "synchronization" state. In the "synchronization" state, if no synchronization header is detected at the predetermined position four times consecutively, it exits the "synchronization" state and sends an alarm to the HOST. After entering the "synchronization" state, the sensor receiving module buffers the received data into the sensor data buffer module for caching.

[0064] This application offloads file management functionality to the FPGA, where data processing and storage according to file formats are performed, improving file transfer efficiency. QDMA is used to handle data transfer between the host and the FPGA, and between the host and the storage server, enabling data read / write operations across large storage areas and increasing data read / write bandwidth. By introducing a "descriptor table" and a "status table" to manage DDR storage space, the host can simultaneously read multiple files, improving parallelism. QDMA is used to implement virtualization management of the device, allowing multiple virtual machines to use the same device, improving device utilization efficiency.

[0065] The software implementation flow of the data transfer method provided in this application is shown below.

[0066] 1) The host implements the QDMA application in kernel mode or user mode, and is responsible for the initialization of the sensor port; at the same time, it designs 8 threads corresponding to 8 data blocks inside the FPGA for the sensor interface.

[0067] 2) Each thread independently listens for read requests from the FPGA, either in a loop or by interruption. When the FPGA receives a 32MB data entry, the host retrieves the data from the FPGA.

[0068] 3) After receiving the read request, the HOST reads the corresponding data from the FPGA (QDMA) and analyzes the address information of the data in the FPGA.

[0069] 4) Based on the address information, the host uses libnfs + multithreading + socket to store the data as a file on the distributed storage server via the storage port.

[0070] Figure 7 This is a structural block diagram of a data transfer device according to an embodiment of this application, such as... Figure 7 As shown, the device includes: an acquisition module 70 for acquiring sampled data to be stored; a storage module 72 for converting the sampled data into multiple data files in a preset format and storing the multiple data files into multiple storage areas in a designated memory; a reading module 74 for receiving a read request through a first communication interface of the QDMA queue-based direct memory access subsystem and reading the data file indicated by the read request from the multiple storage areas; and a sending module 76 for sending the read data file indicated by the read request to a storage server through a second communication interface of the QDMA subsystem.

[0071] Storage module 72 includes: a storage submodule, which is used to determine multiple unused storage areas from the multiple storage areas; process the multiple data files and store them sequentially in the multiple unused storage areas; and output a status table indicating the storage status of the multiple storage areas and a descriptor table indicating the data storage information in the multiple storage areas, wherein the data files correspond one-to-one with the unused storage areas.

[0072] The storage submodule includes a processing unit and an output unit. The processing unit is used to remove invalid information from the frame header corresponding to the first frame of data in the data file and remove the frame headers of subsequent data in the data file to complete the processing of the data file; and to store the processed data file in the corresponding unused storage area.

[0073] The output unit is configured to determine multiple entries of the descriptor table, wherein each entry of the descriptor table includes at least: the starting address of the data storage, the length of the data storage, and the position of the next data segment; determine multiple entries of the status table, wherein each entry of the status table includes at least: a descriptor index, a read / write status, and a data identifier, wherein the descriptor index is used to represent the label of the descriptor, the read / write status is used to indicate whether the descriptor is used, and the data identifier is used to indicate the data belonging to the descriptor; and determine the information corresponding to each entry of the descriptor table and the status table based on the storage status of multiple storage areas and the data storage information in the multiple storage areas.

[0074] The reading module 74 includes: a reading submodule, which is used to traverse the information corresponding to the read / write status table entries and data identifier entries in the status table to determine the first descriptor corresponding to the data file indicated by the read request; and sequentially retrieve multiple segments of data in the data file based on the data storage information indicated by the descriptor, starting from the first descriptor.

[0075] The reading submodule includes a reading unit, which is used to sequentially determine the starting address, length, and position of the next data segment for each descriptor, starting from the first descriptor; read the data corresponding to the length of the data segment for each descriptor from the starting address of the data segment for each descriptor; and then select the starting address of the data segment for the next descriptor according to the position of the next data segment for each descriptor to read the data until all the data of the data identifier mark has been read.

[0076] The aforementioned data transfer device acquires sampled data to be stored; converts the sampled data into multiple data files in a preset format, and stores the multiple data files into multiple storage areas in a designated memory; receives read requests through the first communication interface of QDMA, reads the data file indicated by the read request from the multiple storage areas; and sends the read data file indicated by the read request to the storage server through the second communication interface of the QDMA subsystem. This process converts the sampled data into data files, stores them in the designated memory, and then directly transmits the data files to the storage server. This avoids processing the data after reading the sampled data, saving data processing time. Furthermore, using QDMA to complete the data reading and transmission to the storage server increases the data transmission bandwidth, achieving the effect of improving transfer efficiency during the data transfer process.

[0077] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0078] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the above-described method of acquiring sampled data to be stored; converting the sampled data into multiple data files in a preset format and storing the multiple data files into multiple storage areas in a designated memory; receiving a read request through a first communication interface of QDMA and reading the data file indicated by the read request from the multiple storage areas; and sending the read data file indicated by the read request to a storage server through a second communication interface of the QDMA subsystem.

[0079] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0080] Embodiments of this application also provide a computer device, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps of the method described above: acquiring sampled data to be stored; converting the sampled data into multiple data files in a preset format and storing the multiple data files into multiple storage areas in a designated memory; receiving read requests through a first communication interface of a QDMA queue-based direct memory access subsystem and reading the data files indicated by the read requests from the multiple storage areas; and sending the read data files indicated by the read requests to a storage server through a second communication interface of the QDMA subsystem.

[0081] In an exemplary embodiment, storing the plurality of data files into a plurality of storage areas in a designated memory includes: determining a plurality of unused storage areas from the plurality of storage areas; processing the plurality of data files and storing them sequentially into the plurality of unused storage areas; and outputting a status table indicating the storage status of the plurality of storage areas and a descriptor table indicating the data storage information in the plurality of storage areas, wherein the data files correspond one-to-one with the unused storage areas.

[0082] In an exemplary embodiment, after processing the plurality of data files, they are sequentially stored in the plurality of unused storage areas, including: removing invalid information from the frame header corresponding to the first frame of data in the data file, and removing the frame headers of subsequent data in the data file to complete the processing of the data file; and storing the processed data file in the corresponding unused storage area.

[0083] In an exemplary embodiment, outputting a status table indicating the storage status of the plurality of storage areas and a descriptor table indicating data storage information in the plurality of storage areas includes: determining a plurality of entries in the descriptor table, wherein each entry in the descriptor table includes at least: the starting address of the data storage, the length of the data storage, and the position of the next data segment; determining a plurality of entries in the status table, wherein each entry in the status table includes at least: a descriptor index, a read / write status, and a data identifier, wherein the descriptor index is used to represent the label of the descriptor, the read / write status is used to indicate whether the descriptor is used, and the data identifier is used to indicate the data belonging to the descriptor; and determining the information corresponding to each entry in the descriptor table and the status table based on the storage status of the plurality of storage areas and the data storage information in the plurality of storage areas.

[0084] It should be noted that data attribution is used to indicate which position of data within a data file it belongs to, such as the header, middle, or tail of the data file.

[0085] In an exemplary embodiment, reading the data file indicated by the read request from the plurality of storage areas includes: traversing the information corresponding to the read / write status table entries and data identifier table entries in the status table to determine the first descriptor corresponding to the data file indicated by the read request; and sequentially retrieving multiple segments of data from the data file based on the data storage information indicated by the descriptor, starting from the first descriptor.

[0086] In an exemplary embodiment, starting from the first descriptor, multiple segments of data in the data file are retrieved sequentially based on the data storage information indicated by the descriptor. This includes: starting from the first descriptor, sequentially determining the starting address of the data storage corresponding to each descriptor, the length of the data storage, and the position of the next segment of data; reading data of the corresponding length of data from the starting address of the data storage corresponding to each descriptor, and then selecting the starting address of the data storage of the next descriptor according to the position of the next segment of data corresponding to each descriptor to read data until all data of the data identifier mark has been read.

[0087] In an exemplary embodiment, obtaining the sampled data to be stored includes: after detecting a synchronization header in the sensor data stream, and if the synchronization header is detected consecutively a preset number of times at a preset position in the data stream, caching the sensor data stream to obtain the sampled data.

[0088] In one exemplary embodiment, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0089] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0090] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0091] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A data migration method, characterized by, The method comprises: obtaining sampling data to be stored; converting the sampling data into a plurality of data files in a preset format, and storing the plurality of data files in a plurality of storage areas in a designated memory respectively; receiving a read request through a first communication interface of a QDMA queue-based direct memory access subsystem, and reading a data file indicated by the read request from the plurality of storage areas; sending the read data file indicated by the read request to a storage server through a second communication interface of the QDMA subsystem; wherein storing the plurality of data files in the plurality of storage areas in the designated memory respectively comprises: determining a plurality of unused storage areas from the plurality of storage areas; wherein the unused storage areas are storage areas in which the read-write states of all descriptors are in an unused state; after processing the plurality of data files, storing the plurality of data files in the plurality of unused storage areas in sequence, and outputting a state table indicating the storage states of the plurality of storage areas and a descriptor table indicating the data storage information in the plurality of storage areas, wherein the data files correspond one-to-one to the unused storage areas; after processing the plurality of data files, storing the plurality of data files in the plurality of unused storage areas in sequence comprises: eliminating invalid information in a frame header corresponding to first frame data in the data file, and eliminating frame headers of subsequent data in the data file to complete processing of the data file; storing the processed data file in the corresponding unused storage area; wherein obtaining the sampling data to be stored comprises: after detecting a synchronization header in a sensor data stream, if the synchronization header is detected at a preset position of the data stream for a preset number of times, caching the sensor data stream to obtain the sampling data; wherein the first communication interface is an AXI4 interface, the second communication interface is an AXIS interface, and a HOST device accesses FPGA registers through an AXIL interface of a QDMA subsystem; the FPGA is configured to implement file management functions, perform data processing, and store data in a file format; the QDMA subsystem is configured to complete data transmission between the HOST and the FPGA, and between the HOST and a storage server; each thread in a plurality of threads of the HOST device is configured to listen to read requests sent by the FPGA in a loop or interrupt manner; the HOST device is configured to read data corresponding to the read request from the FPGA upon receiving the read request, and analyze the data corresponding to the read request to obtain address information in the FPGA.

2. The method of claim 1, wherein, outputting the state table indicating the storage states of the plurality of storage areas and the descriptor table indicating the data storage information in the plurality of storage areas comprises: determining a plurality of table entries of the descriptor table, wherein the table entries of the descriptor table at least include: a start address of data storage, a length of data storage, and a position of next data; Determine a plurality of table entries of the state table, wherein the table entries of the state table at least include a descriptor index, a read-write state and a data identifier, the descriptor index is used to represent a label of a descriptor, the read-write state is used to represent whether the descriptor is used, and the data identifier is used to represent data corresponding to the descriptor; According to the storage state of the plurality of storage areas and the data storage information in the plurality of storage areas, respectively determine the information corresponding to each table entry of the descriptor table and the state table.

3. The method of claim 2, wherein, Read the data file indicated by the read request from the plurality of storage areas, including: Traverse the information corresponding to the read-write state table entry and the data identifier table entry in the state table to determine the first descriptor corresponding to the data file indicated by the read request; From the first descriptor, sequentially call the plurality of pieces of data in the data file based on the data storage information indicated by the descriptor.

4. The method of claim 3, wherein, From the first descriptor, sequentially call the plurality of pieces of data in the data file based on the data storage information indicated by the descriptor, including: From the first descriptor, sequentially determine the starting address of the data storage, the length of the data storage and the position of the next piece of data corresponding to each descriptor; From the starting address of the data storage corresponding to each descriptor, read the data with the length of the data corresponding to the descriptor, and then select the starting address of the data storage of the next descriptor according to the position of the next piece of data corresponding to each descriptor to read data until all the data marked by the data identifier is read.

5. A data migration apparatus, characterized by comprising: Including: An acquisition module is configured to acquire sampling data to be stored; A storage module is configured to convert the sampling data into a plurality of data files in a preset format, and store the plurality of data files in a plurality of storage areas in a designated memory, respectively; A read module is configured to receive a read request through a first communication interface of a QDMA (Queue Direct Memory Access) subsystem, and read a data file indicated by the read request from the plurality of storage areas; A sending module is configured to send the read data file indicated by the read request to a storage server through a second communication interface of the QDMA subsystem; The storage module includes a storage submodule configured to determine a plurality of unused storage areas from the plurality of storage areas, wherein the unused storage areas are storage areas in which the read-write states of all descriptors are unused states; after processing the plurality of data files, sequentially store the plurality of data files in the plurality of unused storage areas, and output a state table indicating the storage state of the plurality of storage areas and a descriptor table indicating the data storage information in the plurality of storage areas, wherein the data files correspond to the unused storage areas one by one; The storage submodule includes a processing unit configured to remove invalid information in a frame header corresponding to first frame data in the data file, and remove the frame header of subsequent data in the data file, to complete the processing of the data file; and store the processed data file in the corresponding unused storage area. The acquisition module is further configured to, after detecting the sync header in the sensor data stream, cache the sensor data stream to obtain the sampling data in a case where the sync header is detected continuously for a preset number of times at a preset position of the data stream.

6. An electronic device, comprising: The first communication interface is an AXI4 interface, the second communication interface is an AXIS interface, a HOST device accesses FPGA registers through an AXIL interface of a QDMA subsystem; the FPGA is configured to implement a file management function, perform data processing, and store data in a file format; the QDMA subsystem is configured to complete data transmission between the HOST and the FPGA and between the HOST and a storage server; each thread of a plurality of threads of the HOST device is configured to listen to a read request sent by the FPGA in a loop or interrupt manner; the HOST device is configured to read data corresponding to the read request from the FPGA and analyze the data corresponding to the read request in a case where the read request is received, and address information in the FPGA.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored computer program, and a device where the computer readable storage medium is located executes the data migration method in any one of claims 1 to 4 by running the computer program. The computer readable storage medium comprises a stored computer program, and a device where the computer readable storage medium is located executes the data migration method in any one of claims 1 to 4 by running the computer program.

Images