A TCP / IP-based data caching and transmission method

Abstract

The application relates to the technical field of data processing, in particular to a data caching and transmission method based on TCP / IP, which comprises the following steps: constructing a layered architecture of a transmission task management layer, a data structure layer and a TCP / IP data transmission layer; the transmission task management layer manages transmission tasks in the unit of a whole machine; the data structure layer performs fragmentation processing on source machine data; the TCP / IP data transmission layer realizes data transmission based on a TCP / IP protocol; the transmission task management layer records the relevant information and states of reading tasks, writing tasks and recovery tasks of each copying task, controls the transmission process, and realizes transit transmission, caching transmission, cascade transmission and parallel transmission. The application realizes multiple modes such as transit transmission, cascade transmission and parallel transmission through layered design, supports data caching, transit, cascade and parallel transmission, and meets the data transmission requirements in different scenes.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a TCP / IP-based data caching and transmission method. Background Technology

[0002] In the multi-cloud scenarios of the cloud computing era, traditional data synchronization and transmission methods can no longer meet the requirements, mainly due to the following technical problems:

[0003] Low transmission efficiency: Existing technologies mostly transmit data via files or file bytes, resulting in high resource consumption and low efficiency during data acquisition and transmission.

[0004] Poor adaptability to complex networks: It cannot meet the transmission needs of scenarios such as the Internet, VPN, and leased lines in multi-cloud environments, and it is especially difficult to achieve data transmission in unidirectional network environments.

[0005] Lack of relay transmission capability: It is impossible to set up a relay cache server on the user side to cache data using the high-speed intranet environment before transmitting it through external lines.

[0006] Cascading transmission is difficult to implement: During the transmission process from source A to storage point B and then to storage node C, it is difficult to define the data increment from B to C, making long-term transmission impossible.

[0007] Insufficient support for parallel transmission: Traditional methods cannot enable source A to transmit data to storage nodes B and C simultaneously.

[0008] Lack of multi-host concurrent replication capability: Unable to efficiently handle data replication tasks performed simultaneously on multiple hosts.

[0009] Currently, the main technologies for solving data transmission in multi-cloud scenarios include inter-host file transfer technology and host-to-cloud platform image import technology. Among them, inter-host file transfer technology achieves file synchronization through difference detection, file fragmentation, and compressed transmission, but its operation based on the file system layer still has efficiency bottlenecks and cannot meet the diverse transmission needs in complex network environments. Summary of the Invention

[0010] The purpose of this invention is to provide a TCP / IP-based data caching and transmission method to solve problems such as low transmission efficiency, poor network adaptability, and inability to achieve relay, cascading and parallel transmission in the prior art, thereby improving the flexibility and efficiency of data transmission in a multi-cloud environment.

[0011] To achieve the above objectives, the present invention provides the following technical solution:

[0012] A TCP / IP-based data caching and transmission method, characterized by the following steps:

[0013] Construct a layered architecture consisting of a transmission task management layer, a data structure layer, and a TCP / IP data transmission layer;

[0014] The transmission task management layer manages transmission tasks on a per-machine basis.

[0015] The data structure layer performs data segmentation processing on the source machine;

[0016] The TCP / IP data transmission layer implements data transmission based on the TCP / IP protocol;

[0017] The transmission task management layer records the relevant information and status of each copy task's read, write, and restore tasks, thereby controlling the transmission process to complete relay transmission, buffered transmission, cascaded transmission, and parallel transmission.

[0018] Preferably, in the transmission task management layer, each task is marked with a unique task UUID, and each source machine is marked with a unique host UUID. The relevant information and status of the task are stored in the form of files in specific task directories of the data gateway and storage nodes, and the task information includes information about the source machine, data gateway and target storage node.

[0019] Preferably, in the data structure layer, the data organization structure used for fragmented transmission of source machine data includes metadata and data, wherein the metadata records source machine information and the blocks corresponding to the data.

[0020] Preferably, the data structure layer adopts block-level data processing technology, and reads data in blocks of 2MB to 32MB.

[0021] Preferably, the TCP / IP data transmission layer supports bidirectional and unidirectional transmission. In bidirectional transmission, the network between the two nodes is bidirectional, and in unidirectional transmission, the network between the two nodes is unidirectional. A reverse transmission tunnel is established through HTTPS to realize the unidirectional transmission function between the source and the destination.

[0022] Preferably, the relay transmission is implemented as follows: a data gateway is set up on the user side, the source machine data is cached to the data gateway using the high-speed intranet network environment, and then the cached data is transmitted to the target storage node via the Internet, VPN or dedicated line. The cached data is stored in a specified directory in the form of data cache files, and the data cache files are deleted after the transmission is completed.

[0023] Preferably, the cascading transmission is implemented as follows: the source data is transmitted to data storage node B, and data storage node B transmits the data to data storage node C while writing the data to the disk, wherein the data increment from data storage node B to data storage node C is defined by the transmission task management layer.

[0024] Preferably, the parallel transmission is implemented as follows:

[0025] The source data is transmitted simultaneously to data storage node B and data storage node C through the data gateway. After confirming that the transmission between data storage node B and data storage node C is complete, the data gateway deletes the data cache file.

[0026] Preferably, the method further includes a copy task creation process, specifically:

[0027] Register the source machine, data gateway, and storage node information;

[0028] Prepare the target instance, disk, and other relevant resources;

[0029] Confirm connectivity between the storage node and the data gateway;

[0030] Check the disk status and correspondence between the source and target ends;

[0031] Submit the task to the management console, which will then distribute the task to the data gateway and storage nodes.

[0032] Start the source-side read task and confirm that the task UUID is consistent with the data gateway;

[0033] Calculate the amount of data at the source end;

[0034] Read and transfer data;

[0035] Complete data transfer and write.

[0036] A data processing system, comprising:

[0037] The transmission task management module is used to implement the functions of the transmission task management layer;

[0038] The data structure processing module is used to implement the functions of the data structure layer;

[0039] The TCP / IP data transmission module is used to implement the functions of the TCP / IP data transmission layer.

[0040] Compared with the prior art, the beneficial effects of the present invention are:

[0041] This invention employs block-level data processing technology, bypassing the file system hierarchy, reducing metadata operations and random access overhead, and enabling continuous block data to be processed in a single I / O operation. This reduces the number of operations, improves data reading efficiency, and increases the RPO (Recovery Point Objective) from one day to the minute level.

[0042] This invention designs bidirectional and unidirectional transmission technologies, supporting various network scenarios such as VPN, leased lines, and the Internet, meeting the unidirectional transmission needs from office networks to the Internet, and adapting to complex network architectures in multi-cloud environments.

[0043] This invention implements multiple modes such as relay transmission, cascade transmission and parallel transmission through a layered design, supports data caching, relay, cascade and parallel transmission, and meets the data transmission needs in different scenarios.

[0044] This invention, based on the design of the task management layer, enables concurrent replication of tasks across multiple hosts, ensuring orderly and efficient data transmission and improving the overall transmission efficiency of the system.

[0045] This invention provides precise control over the buffer size for block-level read / write operations, reducing memory usage and CPU load, decreasing data transfer volume, and synchronizing only the changing blocks to save transmission bandwidth and resource consumption. Attached Figure Description

[0046] Figure 1 This is a flowchart of the method steps of the present invention;

[0047] Figure 2 A flowchart for creating a copy task in this invention;

[0048] Figure 3 This is a schematic diagram of the system architecture of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Example 1:

[0051] like Figure 1-3 As shown, a TCP / IP-based data caching and transmission method includes the following steps:

[0052] Construct a layered architecture consisting of a transmission task management layer, a data structure layer, and a TCP / IP data transmission layer;

[0053] The transmission task management layer manages transmission tasks on a per-machine basis.

[0054] The data structure layer performs data fragmentation processing on the source machine;

[0055] The TCP / IP data transmission layer implements data transmission based on the TCP / IP protocol;

[0056] By recording the relevant information and status of each copy task's read, write, and restore tasks through the transmission task management layer, the transmission process can be controlled to complete relay transmission, buffered transmission, cascaded transmission, and parallel transmission.

[0057] The transmission task management layer dynamically allocates system resources, and the calculation formula is as follows:

[0058]

[0059] in, Allocate resources to the task.

[0060] This refers to the current system resource capacity, specifically the amount of resources that the system can actually allocate at this moment, and is affected by the system's used resources and temporary load fluctuations. For example, if the total system resources are 100, and 30 are currently used, then... It might be 70; if a task suddenly releases resources, It will increase dynamically;

[0061] Task priority is a quantitative description of the "importance and urgency" of a transmission task. It does not have a fixed physical unit and is similar to a "ranking system." For example, high-priority tasks (like urgent business data transmission) The value is set to 5, and for ordinary tasks it is set to 3. The higher the value, the more priority the task needs to be guaranteed; it determines the "bias" of resource allocation. The higher the priority of the task, the more likely it is to occupy more system resources in the formula.

[0062] The total system resources enable intelligent scheduling of relay transmission, buffered transmission, cascaded transmission, and parallel transmission; the maximum resource limit that the system can theoretically support is relatively fixed (unless the system is expanded). For example, the total resources of a transmission system, such as hardware and bandwidth, are determined at the factory or during planning, and represent the "total ceiling" of resource allocation.

[0063] In the transmission task management layer, each task is marked with a unique task UUID, and each source machine is marked with a unique host UUID. The relevant information and status of the task are stored in the specific task directory of the data gateway and storage node in the form of files, and the task information includes information about the source machine, data gateway and target storage node.

[0064] In the transmission task management layer, each task is marked with a globally unique identifier, and a distributed hash table is used to store the task state. The consistent hash calculation formula is as follows:

[0065]

[0066] in, and It is a random number. The "key" refers to the value to be hashed; it is the input data for the hash operation. Size of the hash space:

[0067]

[0068] in, This is the updated state.

[0069] The old state

[0070] This represents the state increment.

[0071] In the data structure layer, the data organization structure used for fragmented transmission of source machine data includes metadata and data. Metadata records source machine information and the corresponding data blocks. Data consists of the data stored in the source machine blocks, compressed, and stored as files. The file naming format is...<disk_id> - <number><snapshot_id>.

[0072] In the data structure layer, the data organization structure used for the sharded transmission of source machine data includes metadata and data, as follows:

[0073] Metadata structure: contains multi-level index information, at least including:

[0074] Source machine system information: operating system type, version, host identifier (Host UUID);

[0075] Disk topology: number of disks, disk identifier (Disk ID), disk capacity, partition information;

[0076] Block mapping table: physical address (sector offset), logical address (relative offset), timestamp (creation / modification time) of each data block;

[0077] Verification information: block hash value (such as SHA-256), CRC checksum, data integrity flag.

[0078] The data structure layer adopts block-level data processing technology, specifically including:

[0079] Dynamic block size adaptive mechanism:

[0080] Automatically adjust block size based on source machine IOPS performance, ranging from 2MB to 32MB;

[0081] Data prefetching strategy:

[0082] Implement block pre-reading based on the locality principle, predicting the probability of adjacent block access;

[0083] Use double buffer mechanism to asynchronously prefetch the next block while processing the current block;

[0084] Support multi-thread parallel reading, with thread number dynamically adjusted according to CPU core number.

[0085] Error recovery mechanism:

[0086] Each block contains forward error correction code (FEC) to support single-bit error automatic correction;

[0087] Use redundancy verification mechanism to trigger block retransmission when CRC verification fails;

[0088] Record the block position of failed reading, supporting breakpoint resume.

[0089] Data storage format:

[0090] The source machine block data is processed by the LZ4 / Zstandard compression algorithm to generate binary data blocks.

[0091] Each data block is attached with header information, including:

[0092] Metadata pointer: points to the associated block mapping table entry;

[0093] Compression parameters: compression algorithm type, compression level, original data size;

[0094] Transmission control information: priority flag, retransmission strategy, timeout threshold.

[0095] Data file naming rules: use <disk_id>-<block_number>_<snapshot_id> format, where:

[0096] <disk_id> is the unique identifier of the disk (such as disk01, nvme0n1);

[0097] <block_number> is a 6-bit zero-padded serial number (such as 000001, 000002), which is incremented in reading order;

[0098] <snapshot_id> is a timestamp or version number (such as 20250617T120000Z).

[0099] Data indexing mechanism:

[0100] Build a global index file to record the metadata digest of all shard files;

[0101] Use B + tree structure to realize fast retrieval of index, support multi-dimensional query such as time range, disk ID, hash value, etc.

[0102] The index file is updated synchronously with the data file, and the differential incremental index is generated regularly.

[0103] The storage structure of metadata and data files also includes:

[0104] Multi-level cache strategy:

[0105] Level 1 cache: memory LRU cache, stores recently accessed metadata and hot data blocks;

[0106] Level 2 cache: SSD cache, stores high-frequency access complete files;

[0107] Level 3 cache: HDD storage, archives low-frequency access historical version data.

[0108] Data life cycle management:

[0109] Automatically migrate data to the appropriate storage tier based on access frequency;

[0110] Set a data retention period; expired data will automatically trigger an archiving or deletion process.

[0111] It supports a tiered data storage strategy, and critical data retains multiple versions.

[0112] Concurrency control mechanisms:

[0113] The read-write lock separation mechanism allows multiple read operations to be executed concurrently;

[0114] Write operations use exclusive locks to ensure data consistency;

[0115] Implement transactional operations to ensure atomic updates of metadata and data files.

[0116] The data structure layer also includes data encryption and security mechanisms, specifically:

[0117] Encryption of transmission:

[0118] End-to-end encryption of data blocks in transit is performed using the TLS 1.3 protocol;

[0119] Session keys are generated using the ECDH key exchange algorithm, with a key length of no less than 256 bits;

[0120] Supports certificate verification mechanisms to prevent man-in-the-middle attacks.

[0121] Static encryption:

[0122] Data files are encrypted using the AES-256-GCM algorithm during storage;

[0123] Key Management System (KMS) generates and manages encryption keys;

[0124] It supports a key rotation mechanism to update the encryption key periodically.

[0125] Access control:

[0126] The role-based access control (RBAC) model defines roles such as administrator, operator, and auditor.

[0127] Implement fine-grained access control, with different roles having different operation permissions for metadata and data files;

[0128] Record complete operation audit logs and support security event backtracking.

[0129] The data structure layer uses block-level data processing technology, reading data in blocks ranging from 2MB to 32MB.

[0130] The data structure layer uses an adaptive block size algorithm, and the optimal block size is calculated using the following formula:

[0131]

[0132] in,

[0133] The optimal block size refers to the block size calculated using this formula that allows for optimal efficiency in data read and write operations (considering disk access and network transmission scenarios).

[0134] Disk access time,

[0135] Total data volume

[0136] This refers to network transmission time.

[0137] The TCP / IP data transmission layer supports both bidirectional and unidirectional transmission. In bidirectional transmission, the network between the two nodes can pass through each other in both directions. In unidirectional transmission, the network between the two nodes can pass through each other in one direction only, and the receiver does not need to establish a reverse link. The unidirectional transmission function between the source and the destination can be achieved by establishing a reverse transmission tunnel through HTTPS.

[0138] Table 1: Layered Architecture of Transmission Task Management Layer, Data Structure Layer, and TCP / IP Data Transmission Layer

[0139] Transmission Task Management Layer Transmission is managed on a per-machine basis, with each source machine having a corresponding transmission task manager. Data structure layer This refers to the data organization structure used for fragmented transmission of data from the source machine, namely metadata and data. TCP / IP data transmission layer TCP / IP-based data transmission methods include peer-to-peer full-duplex transmission as well as one-way transmission.

[0140] The relay transmission is implemented as follows: a data gateway is set up on the user side, and the source machine data is cached to the data gateway using the high-speed intranet network environment. The cached data is then transmitted to the target storage node via the Internet, VPN or dedicated line. The cached data is stored in a specified directory in the form of data cache files. The data cache files are deleted after the transmission is completed.

[0141] The relay transmission employs a multi-level caching strategy, and the cache hit rate is calculated using the following formula:

[0142]

[0143] in, To cache the hit count, This represents the total number of visits.

[0144] The cascading transmission is implemented as follows: the source data is transmitted to data storage node B, and data storage node B transmits the data to data storage node C while writing the data to the disk. The data increment from data storage node B to data storage node C is defined by the transmission task management layer.

[0145] Table 2: Definitions related to data transmission tasks

[0146] jobs .prt Read task .prt.status Read task status .wrt Write task .wrt.status Write task status .wrt.progress Write task progress .prv Recovery Task .prv.status Restore task status

[0147] Cascaded transmission employs pipelined parallel processing, and the data processing delay is calculated using the following formula:

[0148]

[0149] in, Let i be the processing time for stage i. Let n be the parallelism of the i-th stage, and n be the number of processing stages.

[0150] The parallel transmission is implemented as follows:

[0151] The source data is transmitted simultaneously to data storage node B and data storage node C through the data gateway. After confirming that the transmission between data storage node B and data storage node C is complete, the data gateway deletes the data cache file.

[0152] The method also includes copying the task creation process, specifically:

[0153] Register the source machine, data gateway, and storage node information;

[0154] Prepare the target instance, disk, and other relevant resources;

[0155] Confirm connectivity between the storage node and the data gateway;

[0156] Check the disk status and correspondence between the source and target ends;

[0157] Submit the task to the management console, which will then distribute the task to the data gateway and storage nodes.

[0158] Start the source-side read task and confirm that the task UUID is consistent with the data gateway;

[0159] Calculate the amount of data at the source end;

[0160] Read and transfer data;

[0161] Complete data transfer and write.

[0162] A data processing system, comprising:

[0163] The transmission task management module is used to implement the functions of the transmission task management layer;

[0164] The data structure processing module is used to implement the functions of the data structure layer;

[0165] The TCP / IP data transmission module is used to implement the functions of the TCP / IP data transmission layer.

[0166] Specific Implementation Example 1: Relay Transmission Scenario

[0167] In a certain enterprise multi-cloud architecture, the local data center needs to migrate business data to a public cloud platform. Due to the limited transmission bandwidth from the local data center to the public network, the relay transmission architecture of this invention is adopted:

[0168] Deploy a data gateway in the local data center to quickly read data from the source server using the high-speed intranet environment (10Gbps), generate data cache files in chunks, and store them on the data gateway.

[0169] The data gateway transmits the cached files to the public cloud storage node step by step via a public dedicated line (100Mbps), and deletes the local cached files after the transmission is complete.

[0170] After receiving the cache file, the storage node writes the data to the target disk and then deletes the cache file.

[0171] In this embodiment, by leveraging the caching function of the data gateway, the high bandwidth requirement that originally needed to be transmitted directly from the source end through the public network is transformed into a combination of high-speed internal network caching and low-speed public network transmission, reducing the impact on the source end and improving transmission efficiency.

[0172] Specific Implementation Example 2: Cascading Transmission Scenario

[0173] A financial institution needs to transfer core business data from its local data center (source A) to a local disaster recovery center (storage node B), and then to a remote disaster recovery center (storage node C):

[0174] Data from source A is transmitted to the data gateway via the intranet, where it is cached to generate a data cache file.

[0175] The data gateway transmits the cached file to storage node B. After receiving it, B writes it to its local disk and simultaneously transmits the cached file to storage node C.

[0176] After receiving the cached file, storage node C writes it to the local disk, completing the cascading transfer.

[0177] By recording and managing the task status of each node through the task management layer, the data increment from B to C can be accurately identified, ensuring the long-term stable operation of cascaded transmission.

[0178] Specific Implementation Example 3: Parallel Transmission Scenario

[0179] An internet company needs to back up user data to two different cloud service providers simultaneously (storage nodes B and C):

[0180] Source data is cached through a data gateway to generate a data cache file.

[0181] The data gateway simultaneously transmits cached files to storage nodes B and C.

[0182] The two storage nodes receive the cached files and write them to their local disks, and the data gateway deletes the cached files after confirming that the transmission is complete.

[0183] This embodiment enables parallel data transmission, improves backup efficiency, and ensures the security of multiple copies of data.

[0184] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.< / number>

Claims

1. A data caching and transmission method based on TCP / IP, characterized in that, Includes the following steps: Construct a layered architecture consisting of a transmission task management layer, a data structure layer, and a TCP / IP data transmission layer; The transmission task management layer manages transmission tasks on a per-machine basis. The data structure layer performs data segmentation processing on the source machine; The TCP / IP data transmission layer implements data transmission based on the TCP / IP protocol; The transmission task management layer records the relevant information and status of each copy task's read, write, and restore tasks, thereby controlling the transmission process to complete relay transmission, buffered transmission, cascaded transmission, and parallel transmission. The data structure layer employs block-level data processing technology, specifically including: Dynamic block size adaptive mechanism: The block size is automatically adjusted based on the source machine's IOPS performance, ranging from 2MB to 32MB; Data prefetching strategy: Block prefetching is implemented based on the principle of locality to predict the access probability of adjacent blocks; A double-buffer mechanism is used to asynchronously prefetch the next block while processing the current block; It supports multi-threaded parallel reading, and the number of threads is dynamically adjusted according to the number of CPU cores; The method also includes copying the task creation process, specifically: Register the source machine, data gateway, and storage node information; Prepare the target instance, disk, and other relevant resources; Confirm connectivity between the storage node and the data gateway; Check the disk status and correspondence between the source and target ends; Submit the task to the management console, which will then distribute the task to the data gateway and storage nodes. Start the source-side read task and confirm that the task UUID is consistent with the data gateway; Calculate the amount of data at the source end; Read and transfer data; Complete data transfer and write.

2. The TCP / IP-based data caching and transmission method according to claim 1, characterized in that, In the transmission task management layer, each task is marked with a unique task UUID, and each source machine is marked with a unique host UUID. The relevant information and status of the task are stored in the form of files in specific task directories of the data gateway and storage nodes, and the task information includes information about the source machine, data gateway and target storage node.

3. The TCP / IP-based data caching and transmission method according to claim 1, characterized in that, In the data structure layer, the data organization structure used for fragmented transmission of source machine data includes metadata and data. The metadata records source machine information and the blocks corresponding to the data.

4. The TCP / IP-based data caching and transmission method according to claim 3, characterized in that, The data structure layer employs block-level data processing technology, reading data in blocks ranging from 2MB to 32MB.

5. The TCP / IP-based data caching and transmission method according to claim 1, characterized in that, The TCP / IP data transmission layer supports both bidirectional and unidirectional transmission. In bidirectional transmission, the network between the two nodes can communicate in both directions, while in unidirectional transmission, the network between the two nodes can communicate in only one direction. A reverse transmission tunnel is established via HTTPS to achieve unidirectional transmission between the source and destination.

6. The TCP / IP-based data caching and transmission method according to claim 1, characterized in that, The relay transmission is implemented as follows: a data gateway is set up on the user side, and the source machine data is cached to the data gateway using the high-speed intranet network environment. The cached data is then transmitted to the target storage node via the Internet, VPN, or dedicated line. The cached data is stored in a specified directory in the form of a data cache file, and the data cache file is deleted after the transmission is completed.

7. The TCP / IP-based data caching and transmission method according to claim 1, characterized in that, The cascading transmission is implemented as follows: the source data is transmitted to data storage node B, and data storage node B transmits the data to data storage node C while writing the data to the disk. The data increment from data storage node B to data storage node C is defined by the transmission task management layer.

8. The TCP / IP-based data caching and transmission method according to claim 1, characterized in that, The parallel transmission is implemented as follows: The source data is transmitted simultaneously to data storage node B and data storage node C through the data gateway. After confirming that the transmission between data storage node B and data storage node C is complete, the data gateway deletes the data cache file.

9. A data processing system, characterized in that, The method for implementing the TCP / IP-based data caching and transmission method according to any one of claims 1 to 8 includes: The transmission task management module is used to implement the functions of the transmission task management layer; The data structure processing module is used to implement the functions of the data structure layer; The TCP / IP data transmission module is used to implement the functions of the TCP / IP data transmission layer.

Images