Method and system for realizing high-speed data transmission through storage depth adjustment

Abstract

The invention discloses a method and system for realizing high-speed data transmission through storage depth adjustment, and the method comprises the steps: mapping a storage space of PCIE (Peripheral Component Interface Express) equipment into a storage region with a set number of blocks, and carrying out the coding; the PCIE device stores data needing to be transmitted in storage areas in sequence, and each storage area serves as a data block; dynamically adjusting the length of the data block to ensure that the computer can respond to the DMA data transmission request; and the computer responds to the DMA data transmission request, reads the codes of the data blocks and processes the codes. According to the method, the problem of data loss caused by the fact that interrupt signals are not collected is solved through storage depth adjustment and data block coding methods. And the size of the data block in the storage area can be adjusted to adapt to data traffic under different sampling rates and match the processing speed of a computer.

Description

Technical Field

[0001] The present invention belongs to the field of electric power technology, and more specifically, relates to a method and system for realizing high-speed data transmission through storage depth adjustment. Background Art

[0002] High-speed data transmission is achieved based on a high-speed serial computer bus (Peripheral Component Interconnect Express, PCIE). In the related art, when data is transmitted between a computer and a PCIE device, there is a speed matching problem, which sometimes makes it impossible to adapt to the PCIE high-speed data transmission performance, and data processing is not timely, which may cause data loss. There are many reasons for data loss in PCIe data transmission, including virus infection, physical damage, hardware failure, interrupt signal failure, communication abnormality, etc. Among them, the most important and common is the data loss problem caused by the failure to collect the interrupt signal.

[0003] Prior art document 1 (CN112988646A) discloses a data acquisition method, device and data acquisition card. The document does not provide a specific and detailed solution to avoid data loss due to failure to acquire a DMA interrupt signal, but only points out that the data is stored in a memory. Summary of the invention

[0004] In order to solve the deficiencies in the prior art, the present invention designs a unique method based on a storage unit to actively adjust the storage depth according to the sampling rate, adapt to the transmission rate of PCIE, complete data processing in a timely manner, and avoid the occurrence of data loss problems as much as possible.

[0005] The present invention adopts the following technical solution.

[0006] A first aspect of the present invention provides a method for implementing high-speed data transmission by adjusting storage depth, comprising: Map the storage space of the PCIE device to a storage area with a set number of blocks and encode it; The PCIE device stores the data to be transmitted in the storage area in sequence, and each storage area is regarded as a data block; Dynamically adjust the length of the data block to ensure that the computer can respond to DMA high-speed data transfer requests; The computer responds to the DMA high-speed data transfer request, reads the data block code, and processes it.

[0007] Preferably, each storage area has the same maximum capacity and is encoded continuously.

[0008] Preferably, the data block length is adjusted according to the storage hardware capacity and sampling rate of the PCIE device to ensure that the computer can respond to the DMA data transmission request and read and process the data.

[0009] Preferably, the sampling rate is set according to the actual application scenario.

[0010] Preferably, historical sampling rate data and user behavior data are collected as training data for the neural network, the neural network is trained, and then the sampling rate change is predicted.

[0011] Preferably, a confidence assessment is performed on the prediction result. If it is lower than a threshold, the prediction is stopped and the data block length is adjusted according to the sampling rate set by the user. If it is higher than the threshold, the data block length is adjusted in advance according to the predicted sampling rate.

[0012] Preferably, the data processing is specifically as follows: The computer compares the data block codes read twice. If the data block codes are continuous, it means that there is no data loss and enters the next data reading.

[0013] Preferably, if the codes read twice by the computer are not continuous, it indicates that data is lost, and the missed data block is determined based on the codes read twice.

[0014] Preferably, the computer reads the missed data blocks in the coding order, and after the missed data blocks are read, the next data reading is started.

[0015] A second aspect of the present invention provides a system for implementing high-speed data transmission by adjusting storage depth, comprising: A partitioning module is used to encode the storage area into blocks; A storage module, used for storing data to be transmitted as data blocks; An adjustment module, used for dynamically adjusting the length of a data block; The transmission module is used to respond to DMA high-speed data transmission requests.

[0016] Compared with the prior art, the beneficial effects of the present invention include at least: the present invention solves the problem of data loss caused by failure to collect interrupt signals through storage depth adjustment and data block encoding methods. Moreover, the storage area data block size adjustment can adapt to data flow at different sampling rates and match computer processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of storage depth adjustment and encoding. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the spirit of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0019] The present invention designs a unique method based on a storage unit to actively adjust the storage depth according to the sampling rate, adapts to the transmission rate of PCIE, completes data processing in time, and avoids the occurrence of data loss problems as much as possible.

[0020] like Figure 1 As shown, embodiment 1 of the present invention provides a method for realizing high-speed data transmission by adjusting storage depth, comprising the following steps: The storage space of the PCIE device is mapped into 8 storage areas, each of which has the same maximum capacity and is encoded continuously. The data that the PCIE device needs to transmit is stored in the storage area in sequence, and each storage area is a data block.

[0021] The length of the data block is set according to the storage hardware capacity of the PCIE device and the user sampling rate requirements to ensure that the computer has enough time to respond to the PCIE request, read and process the data block data in time, and store it in the computer hard disk.

[0022] The sampling rate is set by the user, determined according to actual needs, and transmitted to the PCIE device through the computer. When the sampling rate is high, the amount of data will inevitably increase, and the corresponding data block length will also increase. When the sampling rate is low, the amount of data will inevitably decrease, and the corresponding data block length will also decrease. The change in data block length can change the DMA interrupt response cycle accordingly (the data packet transmitted by DMA has the same data block capacity as the PCIE device). That is, when the sampling rate changes, the data block changes, and the DMA interrupt response cycle also changes to ensure that the computer has the necessary time to respond to the DMA request and complete the data reading.

[0023] Preferably, historical sampling rate data, user behavior and other related data are collected as training data for the neural network, and the neural network is trained to enable it to predict future sampling rate changes. The prediction result is evaluated for confidence. If it is lower than a threshold, the prediction is stopped and the data block length is adjusted according to the sampling rate set by the user. If it is higher than the threshold, the data block length is adjusted in advance according to the predicted sampling rate to further improve the transmission speed. In the present invention, in order to ensure the accuracy of the data block length setting, the threshold is set to 0.8.

[0024] The computer responds to the DMA data transfer request, reads the encoding register in the data block, and compares it with the last data block encoding. If the two data block encodings are continuous, it means that the current response is timely and the data is received normally. If the encoding is not continuous, by comparing the data block encoding read last time with the data block encoding read this time, it is determined that there are missed data blocks. The computer needs to actively read the data blocks that did not have time to respond in the encoding order. For example: the data block encoding read in the last response is 1, and the data encoding read in this response is 2, which means that the data is continuous and no data is missed. If the data block encoding read in this response is 4, it means that the data of data blocks 2 and 3 have been missed. It is necessary to read data blocks 2 and 3 in sequence before reading the data of data block 4. Similarly. Figure 1 The abnormal DMA request sequence at a sampling rate of 200K is that the data block code read in the last response is 4, and the data block code read in this response is 7, which means that the data of data blocks 5 and 6 are missed. It is necessary to read data blocks 5 and 6 in sequence before reading data block 7. This method can effectively solve the problem of data loss.

[0025] As one of the outstanding essential features of the present invention and a significant improvement over the prior art, in the present invention, when the computer receives a DMA interrupt, it reads the data block identification register and compares it with the data read last time to see if it is continuous. The purpose of reading the code is to tell the computer software which storage area to read the data in this time. If it is continuous, it means that the interrupt signal has not been lost, and if it is discontinuous, it means that the interrupt signal has been lost. By numbering, it can be known which data areas have not been read, and the data is read together this time.

[0026] Embodiment 2 of the present invention provides a system for realizing high-speed data transmission by adjusting storage depth, and the method for realizing high-speed data transmission by adjusting storage depth described in embodiment 1 includes: A partitioning module is used to encode the storage area into blocks; A storage module, used for storing data to be transmitted as data blocks; An adjustment module, used for dynamically adjusting the length of a data block; The transmission module is used to respond to DMA high-speed data transmission requests.

[0027] Embodiment 3 of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is loaded into the processor, the method for realizing high-speed data transmission by adjusting storage depth according to embodiment 1 is implemented.

[0028] Embodiment 4 of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for achieving high-speed data transmission by adjusting storage depth according to embodiment 1 is implemented.

[0029] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0030] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for achieving high-speed data transmission by adjusting storage depth, characterized in that: include: Map the storage space of the PCIE device to a storage area with a set number of blocks and encode it; The PCIE device stores the data to be transmitted in the storage area in sequence, and each storage area is regarded as a data block; Dynamically adjust the length of the data block to ensure that the computer can respond to DMA high-speed data transfer requests; The computer responds to the DMA high-speed data transfer request, reads the data block code, and performs data loss prevention processing.

2. The method for realizing high-speed data transmission by adjusting storage depth as claimed in claim 1, characterized in that: The maximum capacity of each storage area is equal and the encoding is continuous.

3. The method for realizing high-speed data transmission by adjusting storage depth as claimed in claim 1, characterized in that: The data block length is adjusted according to the storage hardware capacity and sampling rate of the PCIE device to ensure that the computer can respond to the DMA data transfer request and read and process the data.

4. The method for realizing high-speed data transmission by adjusting storage depth as claimed in claim 3, characterized in that: The sampling rate is set by the user according to the actual application scenario.

5. The method for realizing high-speed data transmission by adjusting storage depth as claimed in claim 3, characterized in that: Collect historical sampling rate data and user behavior data as training data for the neural network, train the neural network, and then predict the sampling rate changes.

6. A method for realizing high-speed data transmission by adjusting memory depth as claimed in claim 5, characterized in that: Perform confidence assessment on the prediction result. If it is lower than the threshold, stop the prediction and adjust the data block length according to the sampling rate set by the user. If it is higher than the threshold, adjust the data block length in advance according to the predicted sampling rate.

7. The method for realizing high-speed data transmission by adjusting memory depth as claimed in claim 1, characterized in that: Data loss prevention processing includes: The computer compares the data block codes read twice. If the data block codes are continuous, it means that there is no data loss and enters the next data reading.

8. A method for realizing high-speed data transmission by adjusting storage depth as claimed in claim 7, characterized in that: If the codes read by the computer twice are not continuous, it means that data is lost. The missed data block is determined based on the codes read twice.

9. The method for realizing high-speed data transmission by adjusting storage depth as claimed in claim 8, characterized in that: The computer reads the missed data blocks in the coding order, and after the missed data are read, it enters the next data reading.

10. A system for realizing high-speed data transmission by adjusting storage depth, based on a method for realizing high-speed data transmission by adjusting storage depth as claimed in any one of claims 1 to 7, characterized in that: include: A partitioning module is used to encode the storage area into blocks; A storage module, used for storing data to be transmitted as data blocks; An adjustment module, used for dynamically adjusting the length of a data block; The transmission module is used to respond to DMA high-speed data transmission requests.

Images