A data transmission method, electronic device, medium, and product
By transferring data packets to a network storage host and managing them using a cache queue, the problem of the unidirectional optical gate buffer easily becoming full is solved, achieving highly reliable and efficient data transmission.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING TOPSEC NETWORK SECURITY TECH
- Filing Date
- 2026-06-16
- Publication Date
- 2026-07-31
AI Technical Summary
When faced with continuous high-speed bursts of data, the fixed-size buffer of the unidirectional optical shutter is easily filled, leading to data packet loss and reducing the reliability of data transmission.
The burst data packets are transferred from the local memory of the one-way optical gate to the external network storage host. The storage and retrieval of data are precisely managed through a cache queue. Data is dynamically unloaded to the network storage host to overcome memory capacity limitations. The SPDK and SCSI protocols are used for initialization, connection and data transfer.
It effectively absorbs continuous high-speed data bursts, avoids data packet loss, and improves the reliability and overall forwarding performance of cross-domain one-way data transmission.
Smart Images
Figure CN122496484A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of network communication technology, specifically to a data transmission method, electronic device, readable storage medium, and computer program product. Background Technology
[0002] The core of cross-domain one-way data transmission methods lies in forwarding one-way data sent by the client to the server. However, in practice, it has been found that when the instantaneous peak of data forwarding exceeds the server's processing capacity, data loss usually occurs.
[0003] To address this issue, existing technologies typically use a fixed-size buffer in the local memory of the unidirectional optical gate to cache data and mitigate the aforementioned data loss problem. However, when faced with continuous high-speed data bursts, the fixed-size local buffer is easily filled, causing subsequent data packets to be discarded, resulting in poor data transmission reliability. Summary of the Invention
[0004] In view of the above problems, this application provides a data transmission method, electronic device, readable storage medium, and computer program product, which can solve the problems of packet loss and insufficient caching capacity of existing methods under burst data traffic.
[0005] In a first aspect, this application provides a data transmission method, including: Receive one-way data packets sent by the client; Detect the receiving rate of the one-way data packet and the sending rate of data to the server; Determine whether the receiving rate is greater than the sending rate; If so, the one-way data packet is stored in the network storage host, and the storage location of the one-way data packet is recorded through a cache queue; Historical cache data is read from the network storage host according to the cache queue, and the historical cache data is sent to the server.
[0006] In the above technical solution, the method can dynamically offload burst data packets to an external network storage host when the data transmission and reception rates are unbalanced (the receiving rate is greater than the sending rate), and accurately manage the storage and retrieval of data through a cache queue. This breaks through the physical limitation of single-machine memory capacity, effectively absorbs continuous high-speed burst data traffic, avoids data packet loss due to the buffer being full, and thus significantly improves the reliability of cross-domain unidirectional data transmission and overall forwarding performance.
[0007] In some implementations, the method further includes, prior to receiving the one-way data packet sent by the receiving client: The initialization interface is invoked to send SPDK parameters to the network storage host; wherein, the SPDK parameters are used for the initialization of the network storage host; Call the discovery server interface to search for the SCSI protocol server of the network storage host in the network; If the SCSI protocol server is found, call the connection server interface to establish a communication connection with the network storage host; The communication connection is used to transmit one-way data packets to be cached and / or historical cached data.
[0008] In the above technical solution, the method can automatically complete the SPDK initialization of the network storage host and the discovery and connection of the network storage host through standardized interface calls before the data transmission begins. This establishes an efficient and stable underlying channel for subsequent data caching and forwarding, avoids data transmission interruption caused by missing configuration or connection failure during the preparation stage, and thus improves the automation level of system deployment and overall operational reliability.
[0009] In some implementations, after establishing a communication connection between the call connection server interface and the network storage host, the method further includes: Call the server information retrieval interface to obtain the hard disk information of the network storage host; wherein, the hard disk information includes hard disk identifier, hard disk block size, total number of hard disk blocks, and total hard disk capacity; The hard drive type of the network storage host is obtained based on the hard drive information; wherein, the hard drive type includes mechanical hard drive or solid-state drive; The network storage host's hard drives are grouped according to the total number of hard drive blocks to obtain multiple storage block groups; Based on the hard disk information, the hard disk type, and the multiple storage block groups, construct an available storage block queue; The available storage block queue includes multiple available storage block nodes, and each available storage block node stores storage block information; the storage block information includes hard disk identifier, storage block start number, and actual stored data size; The available storage block queue is used to provide storage block nodes to the cache queue, where the nodes in the cache queue are storage block nodes that have been written with data.
[0010] In the above technical solution, the method can actively obtain detailed information about the hard disk of the network storage host after establishing a connection with the server, and perform fine-grained management of the storage space by grouping and queuing according to the hard disk type and capacity, thereby abstracting the physical hard disk resources into a queue of available storage blocks that can be dynamically allocated, laying the foundation for the subsequent cache queue to apply for and release storage nodes on demand.
[0011] In some implementations, constructing an available storage block queue based on the hard disk information, the hard disk type, and the plurality of storage block groups includes: When the hard drive type is a mechanical hard drive, create an initial empty queue; The starting sequence number of the storage block is selected sequentially from the multiple storage block groups in an alternating manner, and the corresponding available storage block node is generated sequentially according to the selected starting sequence number. The available storage block nodes are sequentially stored into the initial empty queue to obtain the available storage queue; The alternation method includes first selecting the starting sequence number of a storage block from the group with the smallest block number, then selecting the starting sequence number of a storage block from the group with the largest block number, and so on, until all storage block groups have been processed.
[0012] In the above technical solution, the method enables the storage blocks of the actual data to be distributed as dispersedly as possible in terms of physical address, thereby effectively reducing read and write conflicts and seek latency caused by the frequent concentration of the same track or adjacent area during continuous writing of mechanical hard disks, and thus improving the parallel writing efficiency and overall cache throughput of network storage hosts under high load data burst scenarios.
[0013] In some implementations, constructing an available storage block queue based on the hard disk information, the hard disk type, and the plurality of storage block groups includes: When the hard drive type is a mechanical hard drive, a first empty queue and a second empty queue are constructed according to the business data transmission requirements; The starting sequence number of the storage block is selected sequentially from the group with the smallest block sequence number, and the corresponding first node is generated according to the selected starting sequence number. The first node is then stored in the first empty queue to obtain the first queue. The first queue is used to cache the storage location of data with high transmission rate requirements. The starting sequence number of the storage block is selected sequentially from the group with the largest block sequence number, and the corresponding second node is generated according to the selected starting sequence number. The second node is then stored in the second empty queue to obtain the second queue. The second queue is used to cache the storage location of data with low transmission rate requirements. The available storage block queue includes the first queue and the second queue.
[0014] In the above technical solution, the method can construct independent first and second queues for small sequence number groups and large sequence number groups respectively according to the physical characteristics of mechanical hard disks. This allows data with high transmission rate requirements to obtain storage resources from both the inner and outer physical areas of the hard disk at the same time, thereby achieving parallel processing of the two areas during writing or reading, effectively reducing seek latency and congestion caused by concentrated reading and writing in a single area of the mechanical hard disk.
[0015] In some implementations, constructing an available storage block queue based on the hard disk information, the hard disk type, and the plurality of storage block groups includes: When the hard drive type is a solid-state drive, create an initial empty queue; Multiple storage block start numbers are selected sequentially from the multiple storage block groups in ascending order of block number; Multiple available storage block nodes are generated sequentially based on the starting sequence numbers of the multiple storage blocks; The multiple available storage block nodes are sequentially stored into the multiple initial empty queues to obtain the available storage block queue.
[0016] In the above technical solution, this method can take advantage of the characteristics of solid-state drives (SSDs) having no physical seek latency and excellent random read / write performance. It adopts a strategy of selecting the starting sequence number of storage blocks in ascending order of block number and enqueuing them sequentially, avoiding complex alternation or group isolation logic. This allows the construction of the available storage block queue to be completed with minimal management overhead. At the same time, sequential address mapping also helps the SSD controller to perform internal wear leveling and garbage collection, thereby reducing system complexity while ensuring efficient cache performance. This improves the initialization and data writing efficiency of network storage hosts in pure SSD scenarios.
[0017] In some implementations, storing the one-way data packet in a network storage host includes: Establish a first network connection with the client; Select a target node from the available storage block queue and read the target storage block information from the target node; Using the target storage block information as a parameter, the write data interface is called to write the one-way data packet to the hard disk of the network storage host; Update the actual stored data size in the target node according to the actual data size of the one-way data packet; The target node is written into the cache queue corresponding to the first network connection.
[0018] In the above technical solution, the method can dynamically bind an independent cache queue to each client connection and update the actual size of the storage block in real time when writing data, thereby realizing on-demand allocation and accurate tracking of the network storage host hard disk space, thus avoiding space waste caused by fixed pre-allocation, and ensuring cache isolation of different data streams in multi-connection scenarios.
[0019] In some implementations, reading historical cached data from the network storage host according to the cache queue includes: Read the target node from the cache queue corresponding to the first network connection; Obtain the storage block information in the target node; Using the storage block information as a parameter, the data read interface is invoked to read historical cached data from the hard disk of the network storage host.
[0020] In the above technical solution, the method can accurately locate the storage location of historical data on the hard disk based on the same connection-level cache queue as the writing stage, and realize efficient data back reading through a unified read interface, thereby ensuring the logical consistency of the writing and reading paths, avoiding the addressing overhead and error risk caused by cross-connection or cross-queue retrieval, and thus improving the reliability and throughput efficiency when cached data is dequeued and forwarded.
[0021] In some implementations, after sending the historical cache data to the server, the method further includes: The actual data size in the target node is reset to 0, and the target node is written back to the available storage block queue.
[0022] In the above technical solution, the method can realize closed-loop reuse of network storage host storage space, thereby avoiding the depletion of storage resources caused by long-term node occupation, and thus maximizing the total amount of cacheable data without increasing hardware capacity, effectively supporting continuous or intermittent data burst traffic.
[0023] In some embodiments, the method further includes: Detect the current cache status information; wherein, the cache status information includes the number of available storage block nodes in the available storage block queue or the queue length of the cache queue; When the number of nodes is lower than a first threshold or the queue length exceeds a second threshold, a rate reduction request is sent to the client so that the client can adjust the data transmission rate of single-line data packets according to the rate reduction request.
[0024] In the above technical solution, the method can realize reverse flow control from the data receiving end to the sending end, thereby avoiding the situation where the cache is completely full due to continuous high-speed writing and forced packet loss, thereby improving the self-healing ability and stability of the entire cross-domain data transmission system under extreme load.
[0025] In a second aspect, this application provides an electronic device, the electronic device including a memory and a processor, the memory being used to store a computer program, the processor running the computer program to cause the electronic device to perform the data transmission method described in any one of the first aspects.
[0026] Thirdly, this application provides a readable storage medium storing a computer program, which, when executed by a processor, performs the data transmission method described in any one of the first aspects.
[0027] Fourthly, this application provides a computer program product comprising a computer program that, when executed by a processor, performs the data transmission method described in any one of the first aspects.
[0028] The beneficial effects of this application are as follows: This method can effectively absorb continuous high-speed data burst traffic by caching burst data packets to a network storage host, and by leveraging the storage capacity of the network storage host (including using SSDs or NVMe hard drives for high-speed caching, or using multiple mechanical hard drives for low-cost expansion), thereby avoiding data packet loss and ensuring the reliability of data forwarding to the server. Attached Figure Description
[0029] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0030] Figure 1 This is a flowchart illustrating the data transmission method in some embodiments of this application; Figure 2 This is a schematic diagram of the architecture of the data transmission system in some embodiments of this application; Figure 3 This is a schematic diagram of the structure of an electronic device in some embodiments of this application. Detailed Implementation
[0031] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.
[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.
[0033] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more (including two), similarly, "multiple sets" refers to two or more sets (including two sets), and "multiple pieces" refers to two or more pieces (including two pieces) unless otherwise explicitly defined.
[0034] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0035] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.
[0036] In existing methods, the memory resources in a unidirectional optical shutter are limited. Therefore, when faced with continuous high-speed data bursts, the memory resources are easily filled, leading to data packet loss and reducing the reliability of data transmission.
[0037] To address the aforementioned technical problems, this application provides a data transmission method. The core concept of this method is to transfer bursty data from the local memory of a one-way optical gate to an external network storage host, thereby enabling existing methods to overcome the physical limitations of memory capacity, effectively avoiding data packet loss due to a full buffer, and thus improving the reliability of cross-domain one-way data transmission.
[0038] Based on this concept, the following content will describe the technical solution of this application in detail through specific implementation methods.
[0039] like Figure 1 As shown, some embodiments of this application provide a data transmission method, which includes: S100: Receive one-way data packets sent by the client.
[0040] In this embodiment, the data transmission method is executed by a data transmission system, which refers to a three-host architecture unidirectional optical gate. This three-host architecture unidirectional optical gate includes a transmitting host, a receiving host, and a network storage host.
[0041] In this embodiment, the client refers to the terminal device that sends one-way data packets to the data transmission system, and it is not a component of the data transmission system.
[0042] In this embodiment, step S100 is executed by the sending host in the data transmission system. Specifically, the sending host receives the one-way data packet sent by the client through the network card.
[0043] Specifically, when the proxy engine of the sending host receives network data (i.e., one-way data packets) sent from the client to the server through the network card, it first processes the data based on DPDK (Data Plane Development Kit) and the security engine, and then adjusts and encapsulates it according to the one-way private protocol so that the adjusted and encapsulated one-way data packets (i.e., one-way data packets; since this adjustment process is not emphasized, it is simply and uniformly referred to as one-way data packets) are sent to the receiving host through the one-way sending card.
[0044] Accordingly, the proxy engine of the receiving host receives the adjusted and encapsulated one-way data packets through the one-way receiving card.
[0045] In this embodiment, DPDK is used to achieve high-performance data packet transmission and reception from the network card to the user-space application, bypassing the operating system kernel protocol stack and reducing data copying and CPU interrupt overhead. The security engine is used to encrypt or decrypt data packets, verify their integrity, or perform signature verification to ensure the security of data during transmission.
[0046] S200: Detects the receiving rate of one-way data packets and the sending rate of data to the server.
[0047] In this embodiment, step S200 is executed by the receiving host in the data transmission system.
[0048] Specifically, the proxy engine of the receiving host reads one-way data packets from the one-way receiving card through DPDK and parses the data packets.
[0049] In this embodiment, the proxy engine processes data packets according to their type (including one-way protocol data packets, server network protocol packets, and network storage host protocol data packets), and simultaneously detects the receiving rate of one-way data packets received from the one-way receiving card and the sending rate of data sent to the server.
[0050] The receiving rate of one-way data packets refers to the receiving speed at which the receiving host receives one-way data packets from the one-way receiving card. The data transmission rate to the server refers to the speed at which the receiving host sends one-way data packets to the server through the network card.
[0051] S300. Determine whether the receiving rate is greater than the sending rate. If yes, proceed to step S300; otherwise, end the process.
[0052] In this embodiment, step S300 is executed by the receiving host in the data transmission system.
[0053] In this embodiment, when the data received by the proxy engine of the receiving host from the one-way receiving card exceeds the data sent to the server through the network card, it is determined that the caching process needs to be enabled. Otherwise, the caching process does not need to be enabled, and the process ends.
[0054] S400: Store one-way data packets in the network storage host and record the storage location of one-way data packets through a cache queue.
[0055] In this embodiment, step S400 is executed by the receiving host in the data transmission system.
[0056] In this embodiment, the proxy engine of the receiving host sends one-way data packets to the network storage host via the storage engine and SPDK (Storage Performance Development Kit) so that the network storage host can store the latest data.
[0057] The network storage host is equipped with DPDK, SPDK, and SCSI protocol servers (or NVMe protocol servers), which can obtain network data from the receiving host and write or read data to the hard drive according to the information in the SCSI protocol.
[0058] In this embodiment, the storage engine provides an interface library for writing and reading data to the network storage host, managing storage space, and obtaining storage information from the network storage.
[0059] The interfaces in the interface library include: initialization interface, server discovery interface, server connection interface, server information retrieval interface, data writing interface, data reading interface, and cleanup interface. These interfaces internally call SPDK interfaces to complete their corresponding functions.
[0060] Implementing this approach allows the storage engine to provide unified interfaces for initialization, connection, read / write, and cleanup, while encapsulating calls to the SPDK API. This shields the proxy engine from the complexity of the underlying storage protocol and enables efficient access to network storage hosts.
[0061] S500 reads historical cache data from the network storage host according to the cache queue and sends the historical cache data to the server.
[0062] In this embodiment, step S500 is executed by the receiving host in the data transmission system.
[0063] In this embodiment, the proxy engine of the receiving host sends a request to the network storage host to read historical cached data and obtains the historical cached data. Then, the cached data is sent to the server via the security engine, DPDK and network card, thereby creating the effect of temporary caching of data on the network storage host.
[0064] In this embodiment, the network connection management information of the proxy engine includes a cache queue, and the nodes in the cache queue are used to store cached data information.
[0065] Correspondingly, the available storage block nodes in the available storage block queue, as described later, are also used to store information about cached data. It is emphasized here that the two types of information are of the same kind, but this does not mean that the content of the two types of information is exactly the same.
[0066] As an optional implementation, after step S500, the method may further include: The receiving host resets the actual data size in the target node to 0 and writes the target node back to the available storage block queue.
[0067] For example, after the historical cached data is read, the proxy engine of the receiving host can set the actual data size in the corresponding node to 0, and write the node back to the corresponding available storage block queue according to the node's hard disk ID number.
[0068] In actual operation, when the receiving speed of the proxy engine's one-way receiving card is lower than the sending speed of the network card to the server, the amount of cached data will continue to decrease. When the amount of cached data is 0, the caching process will be shut down.
[0069] In this embodiment, steps S400 and S500 can be executed asynchronously, that is, only the execution order of step S500 can be flexibly adjusted.
[0070] In this embodiment, the network storage host does not participate in network protocol parsing and security checks, but only provides high-performance data storage services through standard storage protocols. The receiving host communicates directly with the host via a user-space storage protocol stack (such as a SCSI protocol stack or NVMe protocol stack based on SPDK), thereby achieving the decoupling of caching functionality and reducing hardware costs.
[0071] In the above embodiments, the method can dynamically offload burst data packets to an external network storage host when the data transmission and reception rates are unbalanced (the receiving rate is greater than the sending rate), and accurately manage the storage and retrieval of data through a cache queue. This breaks through the physical limitations of single-machine memory capacity, effectively absorbs continuous high-speed burst data traffic, avoids data packet loss due to a full buffer, and thus significantly improves the reliability and overall forwarding performance of cross-domain unidirectional data transmission.
[0072] In some embodiments, when faced with a situation where the receiving rate and transmitting rate are close, the system may frequently enter and exit buffer mode, resulting in unnecessary overhead and performance jitter, step S300 may include: S310. Calculate the first correction rate based on the preset first threshold and the transmission rate, and calculate the second correction rate based on the preset second threshold and the transmission rate; wherein the preset first threshold is greater than 1, and the preset second threshold is less than 1.
[0073] In this embodiment, the preset first threshold refers to the rate amplification coefficient used to determine whether to enter the cache mode, and its value is greater than 1. The preset second threshold refers to the rate reduction coefficient used to determine whether to exit the cache mode, and its value is less than 1.
[0074] In this embodiment, the preset first threshold and the preset second threshold are based on the following: In scenarios where the receiving rate and the sending rate are close, in order to avoid the system frequently entering and exiting the buffer mode due to instantaneous rate fluctuations, a hysteresis interval is formed by setting a first threshold greater than 1 and a second threshold less than 1, so that the system only enters the buffer mode when the receiving rate exceeds the sending rate by a certain margin, and only exits the buffer mode when the receiving rate is lower than the sending rate by a certain margin.
[0075] In this embodiment, the first correction rate refers to the rate value obtained by multiplying the sending rate by a preset first threshold, which serves as the trigger threshold for determining whether to enter the buffer mode. The second correction rate refers to the rate value obtained by multiplying the sending rate by a preset second threshold, which serves as the trigger threshold for determining whether to exit the buffer mode.
[0076] S320. When the receiving rate is greater than the first correction rate, it is determined that buffering is required.
[0077] S330. When the receiving rate is less than the second correction rate and the buffer queue is empty, exit the buffer mode.
[0078] In this embodiment, when the receiving speed of the one-way receiving card in the receiving host is consistently higher than the sending speed of the network card to the server, the amount of cached data will continue to increase; conversely, it will decrease until the amount of cached data is 0, at which point the cache mode will be exited.
[0079] In the above embodiments, the method can improve system stability by introducing a hysteresis interval to avoid frequent switching of buffer modes when the receiving rate fluctuates near the critical point.
[0080] In some embodiments, step S400 may include: S410: Establish the first network connection with the client.
[0081] In this embodiment, the first network connection refers to the logical communication channel established between the receiving host and the client for transmitting one-way data packets. This connection is used to identify and distinguish different clients or multiple data streams from the same client, so that the receiving host can independently maintain the cache state for each connection.
[0082] S420. Select a target node from the available storage block queue and read the target storage block information from the target node.
[0083] In this embodiment, the available storage block queue refers to a queue consisting of multiple available storage block nodes. Each available storage block node records the location information (including hard disk identifier, storage block start number, and actual stored data size) of a free storage area on the network storage host hard disk. This queue is used to allocate storage space on demand.
[0084] In this embodiment, the target node refers to an available storage block node selected from the available storage block queue, which is used to record the storage location information of the data to be written in the network storage host hard disk.
[0085] In this embodiment, the target storage block information refers to the specific content stored in the target node, including the hard disk identifier, the starting sequence number of the storage block, and the actual size of the currently stored data, which is used to guide the data writing operation.
[0086] S430: Using the target storage block information as a parameter, call the write data interface to write the one-way data packet to the hard drive of the network storage host.
[0087] In this embodiment, the write data interface refers to the function interface provided by the storage engine for writing data to the network storage host. This interface takes the hard disk identifier, storage block start number and other information in the target storage block information as input parameters, and internally calls the SPDK interface to complete the actual data writing operation.
[0088] In this embodiment, the hard disk refers to the physical storage device in the network storage host, including mechanical hard disks or solid-state hard disks, used to persistently store burst data that is unloaded by the receiving host.
[0089] S440. Update the actual stored data size in the target node based on the actual data size of the one-way data packet.
[0090] In this embodiment, the actual data size refers to the real data length of the one-way data packet, in bytes or blocks, and is used to record the space occupied by data in the storage area corresponding to the target node.
[0091] S450: Write the target node into the cache queue corresponding to the first network connection.
[0092] In this embodiment, the first network connection independently maintains a queue. The nodes in the queue are storage block nodes for which data has been written (i.e., target nodes for which data has been filled) to record the location information of cached data on the hard disk under this connection, so that historical cached data can be accurately located during subsequent reading.
[0093] For example, a network storage host can be configured with two CPU cores (one core is responsible for handling hard disk read / write requests, and the other core is responsible for system management and SCSI protocol server responses), two hard disks, and run a SCSI protocol server program on this basis to respond to read / write commands from the receiving host.
[0094] When client 1 sends data to the server, the proxy engine of the receiving host establishes network connection 1 for client 1. Then, it reads a node from queue 1, uses the block start number, actual data size, and number of blocks in that node as parameters to call the "write data interface," and sends the data to the SCSI protocol server program of the network storage host via SPDK, DPDK, and network card. After the SCSI protocol server program parses the command, it writes the data to hard disk 1; at the same time, it updates the actual data size field of the node, and then writes the node to the cache queue corresponding to network connection 1.
[0095] At this point, if client 1 sends data to the server again, the proxy engine of the receiving host reads a node from queue 2 and calls the "write data interface" with the block start number, actual data size, and number of blocks in that node as parameters. The data is then sent to the SCSI protocol server program of the network storage host via SPDK, DPDK, and the network card. After the SCSI protocol server program parses the command, it writes the data to hard disk 2. At the same time, it updates the actual data size field of the node and writes the node to the cache queue corresponding to network connection 1.
[0096] As can be seen, this method can write data from the same network connection to hard disk 1 and hard disk 2 in turn, thereby achieving the effect of balancing the load on the two hard disks.
[0097] It should be noted that the same rules apply to cache writing, whether it is another connection from the same client or a connection from another client.
[0098] In the above embodiments, the method can dynamically bind an independent cache queue to each client connection and update the actual usage size of the storage block in real time when writing data, thereby realizing on-demand allocation and accurate tracking of the network storage host hard disk space, thus avoiding space waste caused by fixed pre-allocation, and ensuring cache isolation of different data streams in multi-connection scenarios.
[0099] In some embodiments, step S500 may include: S510: Read the target node from the cache queue corresponding to the first network connection.
[0100] In this embodiment, one network connection corresponds to one cache queue.
[0101] In this embodiment, both the data caching process and the data reading process use this description of the target node. Only the identical descriptions here have a special referential relationship; that is, they can refer to the same node or not.
[0102] In this embodiment, the content here can also be referred to as the first target node and the second target node, wherein the first target node and the second target node may be the same node.
[0103] S520: Obtain storage block information in the target node.
[0104] In this embodiment, the storage block information includes the hard disk identifier (used to locate which hard disk the data is stored on), the starting sequence number of the storage block (used to locate the starting position of the data on the hard disk), and the actual stored data size (used to identify the length of valid data already stored in the storage block).
[0105] S530 uses storage block information as a parameter to call the data read interface and read historical cache data from the hard disk of the network storage host.
[0106] For example, after the proxy engine of the receiving host finishes writing a batch of data, it reads a node from the cache queue in the network connection and calls the "read data interface" according to the block sequence number and other information in the node. The data is then read from hard disk 1 or hard disk 2 through SPDK, DPDK, network card, network storage host's DPDK and SPDK, and sent to the server through security engine, DPDK and network card.
[0107] In this embodiment, in a high-speed cache data scenario, the method can configure multiple large-capacity hard drives in the network storage host, thereby improving the speed through parallel reading and writing of data from multiple hard drives.
[0108] In this embodiment, the method can also employ a server supporting the NVMe protocol and NVMe hard drives to improve caching speed. The processing flow of this protocol server is the same as that of the SCSI protocol server, except that it calls the corresponding NVMe protocol and interface in the interface.
[0109] In the above embodiments, the method can accurately locate the storage location of historical data on the hard disk based on the same connection-level cache queue as the writing phase, and realize efficient data back reading through a unified read interface, thereby ensuring the logical consistency of the writing and reading paths, avoiding the addressing overhead and error risk caused by cross-connection or cross-queue retrieval, and thus improving the reliability and throughput efficiency when cached data is dequeued and forwarded.
[0110] In some embodiments, prior to step S100, the method may further include: S600 calls the initialization interface to send SPDK parameters to the network storage host; the SPDK parameters are used for the initialization of the network storage host.
[0111] In this embodiment, step S600 is executed by the receiving host in the data transmission system.
[0112] In this embodiment, SPDK parameters refer to the configuration information used to initialize the network storage host SPDK runtime environment. Specifically, these may include the hard disk device identifier to be bound, NVMe or SCSI protocol stack configuration options, memory pool size, queue depth, CPU core binding strategy, etc.
[0113] These parameters determine how SPDK subsequently takes over the hard drive and performs high-performance data read and write operations.
[0114] S700: Call the discovery server interface to search for the SCSI protocol server of the network storage host in the network.
[0115] In this embodiment, step S700 is executed by the receiving host in the data transmission system.
[0116] In this embodiment, the SCSI protocol server refers to a service program (or software module) that runs on the network storage host, follows the SCSI protocol standard, and is used to receive and process data read and write requests.
[0117] S800. If the SCSI protocol server is found, call the connection server interface to establish a communication connection with the network storage host; wherein, the communication connection is used to transmit one-way data packets to be cached and / or historical cached data.
[0118] In this embodiment, step S800 is executed by the receiving host in the data transmission system.
[0119] In this embodiment, the storage engine's interfaces include an initialization interface, a server discovery interface, a server connection interface, a server information acquisition interface, a data writing interface, a data reading interface, and a cleanup interface. These interfaces internally call the SPDK interface to complete their corresponding functions.
[0120] Specifically, the proxy engine of the receiving host can call the "initialization interface" to initialize SPDK parameters, and call the "discover server interface" to search for SCSI protocol servers in the network. If no server is found, an alarm is written to the log and the discovery process is terminated; if a SCSI protocol server is found, the connection server interface is called to connect to the network storage host.
[0121] In this embodiment, the method may specifically involve calling the connection server interface to connect the receiving host to the SCSI protocol server running on the network storage host.
[0122] In the above embodiments, the method can automatically complete the SPDK initialization of the network storage host and the discovery and connection of the network storage host through standardized interface calls before the data transmission begins. This establishes an efficient and stable underlying channel for subsequent data caching and forwarding, avoiding data transmission interruptions caused by missing configurations or connection failures during the preparation phase, thereby improving the automation level of system deployment and overall operational reliability.
[0123] In some embodiments, after step S800 and before step S100, the method may further include: S900: Call the server information retrieval interface to obtain the hard disk information of the network storage host; the hard disk information includes the hard disk identifier, the block size of the hard disk, the total number of blocks on the hard disk, and the total capacity of the hard disk.
[0124] In this embodiment, steps S900 to S1200 are all executed by the receiving host in the data transmission system.
[0125] In this embodiment, the "Get Server Information Interface" is called to query the hard disk information of the network storage host. The obtained hard disk information includes the hard disk ID number, block size, total number of blocks, total capacity, and other information.
[0126] For example, the hard drives with ID numbers 1 and 2 were obtained successively, along with block sizes of 4096 and block information of 10,485,700. The two hard drives can store 800GB of data.
[0127] S1000: Obtain the hard drive type of the network storage host based on the hard drive information; where the hard drive type includes mechanical hard drive or solid-state hard drive.
[0128] S1100: Group the hard disks of the network storage host according to the total number of hard disk blocks to obtain multiple storage block groups.
[0129] In this embodiment, the method can group 1024 blocks together and store information such as the starting sequence number, block size, and hard disk ID of the blocks in the available storage block queue.
[0130] S1200: Based on hard disk information, hard disk type, and multiple storage blocks grouped together, construct an available storage block queue; the available storage block queue is used to provide storage block nodes to the cache queue, and the nodes in the cache queue are storage block nodes for which data has been written.
[0131] In this embodiment, the available storage block queue includes multiple available storage block nodes, and each available storage block node stores storage block information; the storage block information includes hard disk identifier, storage block starting sequence number, and actual stored data size.
[0132] For example, this method can establish a queue for each hard drive after obtaining the hard drive information. Each node in the queue stores the hard drive ID number, the starting sequence number of a group (e.g., 1024 blocks), and the actual data size. These nodes form a queue of available storage blocks, and each node is an available storage block node.
[0133] In this embodiment, when the network storage host is configured with multiple hard drives, the method can establish a corresponding available storage block queue for each hard drive.
[0134] For example, hard disk 1 corresponds to available storage block queue 1, and hard disk 2 corresponds to available storage block queue 2. Based on this, the proxy engine of the receiving host can alternately select storage block nodes from queue 1 and queue 2 when writing data to achieve load balancing across multiple hard disks.
[0135] In the above embodiments, the method can actively obtain detailed information about the hard disk of the network storage host after establishing a connection with the server, and perform fine-grained management of the storage space by grouping and queuing according to the hard disk type and capacity, thereby abstracting the physical hard disk resources into a queue of available storage blocks that can be dynamically allocated, laying the foundation for the subsequent cache queue to apply for and release storage nodes on demand.
[0136] In some embodiments, step S1200 may include: S1210. When the hard disk type is mechanical hard disk, create an initial empty queue.
[0137] S1220. Select the starting sequence number of the storage block from multiple storage block groups in an alternating manner, and generate the corresponding available storage block node in sequence according to the selected starting sequence number; wherein, the alternating manner includes first selecting the starting sequence number of a storage block from the group with the smallest block sequence number, then selecting the starting sequence number of a storage block from the group with the largest block sequence number, and so on, until all storage block groups have been processed.
[0138] S1230. Store the available storage block nodes into the initial empty queue in sequence to obtain the available storage queue.
[0139] For example, if the hard drive is a mechanical hard drive, the method can group 10485700 into 1024 blocks, then first obtain the starting sequence number of the block corresponding to the group from the smaller sequence number end and store it in the available storage queue, and then obtain the starting sequence number of the block corresponding to the group from the larger sequence number end and store it in the available storage queue.
[0140] This method, after storing all groups, can balance the fast read / write speed of blocks corresponding to smaller sequence numbers and the slow read / write speed of blocks corresponding to larger sequence numbers when caching large amounts of data.
[0141] In the above embodiments, the method enables the storage blocks of subsequent actual written data to be distributed as dispersed as possible in terms of physical address, thereby effectively reducing read-write conflicts and seek latency caused by frequent concentration in the same track or adjacent area during continuous writing of mechanical hard disks, and thus improving the parallel writing efficiency and overall cache throughput of network storage hosts under high-load data burst scenarios.
[0142] In some embodiments, step S1200 may further include: S1240. When the hard disk type is mechanical hard disk, construct the first empty queue and the second empty queue according to the business data transmission requirements.
[0143] S1250. Select the starting sequence number of the storage block from the group with the smallest block sequence number in sequence, generate the corresponding first node according to the selected starting sequence number, and store the first node into the first empty queue to obtain the first queue; wherein, the first queue is used to cache the storage location of data with high transmission rate requirements.
[0144] S1260. Select the starting sequence number of the storage block from the group with the largest block sequence number in sequence, generate the corresponding second node according to the selected starting sequence number, and store the second node in the second empty queue to obtain the second queue; wherein, the second queue is used to cache the storage location of data with low transmission rate requirements.
[0145] In this embodiment, the available storage block queue includes a first queue and a second queue. That is, the first queue and the second queue constitute the available storage block queue.
[0146] For example, this method can construct queue 1-1 and queue 1-2 according to business needs. (In queue NM, N represents disk N, and M represents queue M.) Queue 1-1 is used to store the starting number of the block in the small sequence group and is used to cache data with fast transmission speed; Queue 1-2 caches data with slow transmission speed.
[0147] Based on this, the proxy engine of the receiving host can obtain available storage blocks from queue 1-1 or queue 1-2 according to the fast or slow demand of business data.
[0148] In the above embodiments, the method can construct independent first and second queues for small sequence number groups and large sequence number groups respectively according to the physical characteristics of the hard disk, so that data with high transmission rate requirements can obtain storage resources from both the inner and outer physical areas of the hard disk at the same time, thereby realizing parallel processing of the two areas during writing or reading, effectively reducing seek latency and congestion caused by concentrated reading and writing in a single area of the hard disk.
[0149] In some embodiments, step S1200 may include: S1270. When the hard disk type is solid-state drive, create an initial empty queue; and select multiple storage block starting numbers from multiple storage block groups in ascending order of block number.
[0150] S1280. Generate multiple available storage block nodes in sequence according to the starting sequence number of multiple storage blocks.
[0151] S1290. Store multiple available storage block nodes sequentially into multiple initial empty queues to obtain an available storage block queue.
[0152] For example, if the hard disk type is an SSD, the method obtains the corresponding block start number from the little-endian sequence of the group and stores it in the queue node.
[0153] In the above embodiments, this method can take advantage of the characteristics of solid-state drives (SSDs) having no physical seek latency and excellent random read / write performance. It adopts a strategy of selecting the starting sequence number of storage blocks in ascending order of block number and enqueuing them sequentially, avoiding complex alternation or group isolation logic. This allows the construction of the available storage block queue to be completed with minimal management overhead. At the same time, sequential address mapping also helps the SSD controller to perform internal wear leveling and garbage collection, thereby reducing system complexity while ensuring efficient cache performance. This improves the initialization and data writing efficiency of network storage hosts in pure SSD scenarios.
[0154] In some embodiments, the method may execute step S1300 in parallel during the execution of steps S100 to S500. Step S1300 may include: S1310. Detect the current cache status information; wherein, the cache status information includes the number of available storage block nodes in the available storage block queue or the queue length of the cache queue.
[0155] S1320. When the number of nodes is lower than the first threshold or the queue length exceeds the second threshold, a rate reduction request is sent to the client so that the client can adjust the data transmission rate of single-line data packets according to the rate reduction request.
[0156] In this embodiment, step S1300 is executed by the receiving host in the data transmission system.
[0157] In this embodiment, when the data transmission system needs to be shut down, the proxy engine of the receiving host can call the cleanup interface to close the connection with the network storage host and release resources such as queues and nodes.
[0158] In the above embodiments, the method can realize reverse flow control from the data receiving end to the sending end, thereby avoiding the situation where the cache is completely full due to continuous high-speed writing and forced packet loss, thereby improving the self-healing ability and stability of the entire cross-domain data transmission system under extreme load.
[0159] like Figure 2 As shown, some embodiments of this application provide a schematic diagram of the architecture of a data transmission system. It should be understood that this system is related to... Figure 1 The method executed in the middle corresponds to the steps involved in the aforementioned method. The specific functions and effects of the system can be found in the description above. To avoid repetition, detailed descriptions are omitted here.
[0160] The data transmission system comprises a client, a sending host, a receiving host, a network storage host, and a server. The client is used to send one-way data packets to the receiving host; The sending host is used to receive one-way data packets and forward them to the receiving host. The receiving host is used to detect the receiving rate of one-way data packets and the sending rate of data to the server. The receiving host is also used to determine whether the receiving rate is greater than the sending rate; if so, it stores the one-way data packet in the network storage host and records the storage location of the one-way data packet through a buffer queue. The receiving host is also used to read historical cached data from the network storage host according to the cache queue and send the historical cached data to the server. like Figure 3 As shown, this application provides an electronic device 1400, which includes a processor 1401 and a memory 1402. The processor 1401 and the memory 1402 are interconnected and communicate with each other through a communication bus 1403 and / or other forms of connection mechanism (not shown). The memory 1402 stores a computer program that can be executed by the processor 1401. When the computing device is running, the processor 1401 executes the computer program to perform the method in any of the aforementioned optional implementations.
[0161] This application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the method in any of the aforementioned optional implementations.
[0162] The computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0163] This application provides a computer program product, which includes a computer program that, when run by a processor, executes the method in any of the aforementioned optional implementations.
[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and they should all be covered within the scope of the claims and specification of this application. In particular, as long as there is no conflict, the various technical features mentioned in the embodiments can be combined in any way. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.
Claims
1. A data transmission method, characterized in that, include: Receive one-way data packets sent by the client; Detect the receiving rate of the one-way data packet and the sending rate of data to the server; Determine whether the receiving rate is greater than the sending rate; If so, the one-way data packet is stored in the network storage host, and the storage location of the one-way data packet is recorded through a cache queue; Historical cache data is read from the network storage host according to the cache queue, and the historical cache data is sent to the server.
2. The data transmission method according to claim 1, characterized in that, Before receiving the one-way data packet sent by the receiving client, the method further includes: The initialization interface is invoked to send SPDK parameters to the network storage host; wherein, the SPDK parameters are used for the initialization of the network storage host; Call the discovery server interface to search for the SCSI protocol server of the network storage host in the network; If the SCSI protocol server is found, call the connection server interface to establish a communication connection with the network storage host; The communication connection is used to transmit one-way data packets to be cached and / or historical cached data.
3. The data transmission method according to claim 2, characterized in that, After establishing a communication connection between the call connection server interface and the network storage host, the method further includes: Call the server information retrieval interface to obtain the hard disk information of the network storage host; wherein, the hard disk information includes hard disk identifier, hard disk block size, total number of hard disk blocks, and total hard disk capacity; The hard drive type of the network storage host is obtained based on the hard drive information; wherein, the hard drive type includes mechanical hard drive or solid-state drive; The network storage host's hard drives are grouped according to the total number of hard drive blocks to obtain multiple storage block groups; Based on the hard disk information, the hard disk type, and the multiple storage block groups, construct an available storage block queue; The available storage block queue includes multiple available storage block nodes, and each available storage block node stores storage block information; the storage block information includes hard disk identifier, storage block start number, and actual stored data size; The available storage block queue is used to provide storage block nodes to the cache queue, where the nodes in the cache queue are storage block nodes that have been written with data.
4. The data transmission method according to claim 3, characterized in that, The step of constructing an available storage block queue based on the hard disk information, the hard disk type, and the multiple storage block groups includes: When the hard drive type is a mechanical hard drive, create an initial empty queue; The starting sequence number of the storage block is selected sequentially from the multiple storage block groups in an alternating manner, and the corresponding available storage block node is generated sequentially according to the selected starting sequence number. The available storage block nodes are sequentially stored into the initial empty queue to obtain the available storage queue; The alternation method includes first selecting the starting sequence number of a storage block from the group with the smallest block number, then selecting the starting sequence number of a storage block from the group with the largest block number, and so on, until all storage block groups have been processed.
5. The data transmission method according to claim 3, characterized in that, The step of constructing an available storage block queue based on the hard disk information, the hard disk type, and the multiple storage block groups includes: When the hard drive type is a mechanical hard drive, a first empty queue and a second empty queue are constructed according to the business data transmission requirements; The starting sequence number of the storage block is selected sequentially from the group with the smallest block sequence number, and the corresponding first node is generated according to the selected starting sequence number. The first node is then stored in the first empty queue to obtain the first queue. The first queue is used to cache the storage location of data with high transmission rate requirements. The starting sequence number of the storage block is selected sequentially from the group with the largest block sequence number, and the corresponding second node is generated according to the selected starting sequence number. The second node is then stored in the second empty queue to obtain the second queue. The second queue is used to cache the storage location of data with low transmission rate requirements. The available storage block queue includes the first queue and the second queue.
6. The data transmission method according to claim 3, characterized in that, The step of constructing an available storage block queue based on the hard disk information, the hard disk type, and the multiple storage block groups includes: When the hard drive type is a solid-state drive, create an initial empty queue; Multiple storage block start numbers are selected sequentially from the multiple storage block groups in ascending order of block number; Multiple available storage block nodes are generated sequentially based on the starting sequence numbers of the multiple storage blocks; The multiple available storage block nodes are sequentially stored into the multiple initial empty queues to obtain the available storage block queue.
7. The data transmission method according to claim 1, characterized in that, The step of storing the one-way data packet in the network storage host includes: Establish a first network connection with the client; Select a target node from the available storage block queue and read the target storage block information from the target node; Using the target storage block information as a parameter, the write data interface is called to write the one-way data packet to the hard disk of the network storage host; Update the actual stored data size in the target node according to the actual data size of the one-way data packet; The target node is written into the cache queue corresponding to the first network connection.
8. The data transmission method according to claim 1, characterized in that, The step of reading historical cached data from the network storage host according to the cache queue includes: Read the target node from the cache queue corresponding to the first network connection; Obtain the storage block information in the target node; Using the storage block information as a parameter, the data read interface is invoked to read historical cached data from the hard disk of the network storage host.
9. The data transmission method according to claim 8, characterized in that, After sending the historical cache data to the server, the process also includes: The actual data size in the target node is reset to 0, and the target node is written back to the available storage block queue.
10. The data transmission method according to claim 1, characterized in that, The method further includes: Detect the current cache status information; wherein, the cache status information includes the number of available storage block nodes in the available storage block queue or the queue length of the cache queue; When the number of nodes is lower than a first threshold or the queue length exceeds a second threshold, a rate reduction request is sent to the client so that the client can adjust the data transmission rate of single-line data packets according to the rate reduction request.
11. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the data transmission method according to any one of claims 1 to 10.
12. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, performs the data transmission method according to any one of claims 1 to 10.
13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, performs the data transmission method according to any one of claims 1 to 10.