Data caching and transmission method based on TCP / IP (Transmission Control Protocol / Internet Protocol)

Abstract

The invention relates to the technical field of data processing, in particular to a TCP / IP (Transmission Control Protocol / Internet Protocol)-based data caching and transmission method, which comprises the following steps of: constructing a layered architecture of a transmission task management layer, a data structure layer and a TCP / IP data transmission layer; the transmission task management layer manages transmission tasks by taking a whole machine as a unit; the data structure layer carries out fragmentation processing on source machine data; the TCP / IP data transmission layer realizes data transmission based on a TCP / IP protocol; and recording related information and states of a reading task, a writing task and a recovery task of each copy task through the transmission task management layer to realize control of a transmission process so as to complete transfer transmission, cache transmission, cascade transmission and parallel transmission. According to the invention, multiple modes such as transit transmission, cascade transmission and parallel transmission are realized through hierarchical design, data caching, transit, cascade and parallel transmission are supported, and data transmission requirements in different scenes are met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data caching and transmission method based on TCP / IP. BACKGROUND

[0002] In the multi-cloud scenario of the cloud computing era, the traditional data synchronization and transmission method cannot meet the needs, and the following technical problems exist: Low transmission efficiency: the existing technology mainly transmits through files or file bytes, and the resource consumption is high and the efficiency is low during data acquisition and transmission.

[0003] Poor adaptability to complex networks: unable to meet the transmission needs in the Internet, VPN, dedicated line and other scenarios in the multi-cloud environment, especially in one-way network environment.

[0004] Lack of relay transmission capability: unable to set up a relay caching server on the user side, and cache data in the internal network high-speed environment before transmission through external lines.

[0005] Difficult to implement cascading transmission: in the transmission process from source A to storage point B to storage node C, it is difficult to define the data increment from B to C, and long-term transmission cannot be realized.

[0006] Insufficient support for parallel transmission: the traditional method cannot realize the transmission of data from source A to storage nodes B and C at the same time.

[0007] Multiple host concurrent replication capability is missing: unable to efficiently handle the task of data replication by multiple hosts at the same time.

[0008] The current technology for solving the data transmission in the multi-cloud scenario mainly includes the file transmission technology between hosts and the mirror import technology from host to cloud platform. Among them, the file transmission technology between hosts realizes file synchronization through difference detection, file fragmentation and compression transmission, but its operation based on the file system layer still has efficiency bottleneck, and cannot meet the transmission needs in complex network environment. SUMMARY

[0009] The purpose of the present application is to provide a data caching and transmission method based on TCP / IP, to solve the problems of low transmission efficiency, poor network adaptability, inability to realize relay, cascading and parallel transmission in the prior art, and to improve the flexibility and efficiency of data transmission in the multi-cloud environment.

[0010] To achieve the above purpose, the present application provides the following technical scheme: A data caching and transmission method based on TCP / IP, characterized in that it comprises the following steps: Constructing a layered architecture of a transmission task management layer, a data structure layer and a TCP / IP data transmission layer; The transmission task management layer manages transmission tasks in the unit of the whole machine. The data structure layer performs sharding processing on source machine data. The TCP / IP data transmission layer realizes data transmission based on the TCP / IP protocol. The transmission task management layer records the relevant information and state of the reading task, the writing task and the recovery task of each replication task, realizes the control of the transmission process, and completes the relay transmission, the cache transmission, the cascade transmission and the parallel transmission.

[0011] Preferably, in the transmission task management layer, each task is marked by a unique task UUID, and each source machine is marked by a unique host UUID. The relevant information and state of the task are stored in the form of a file in a specific task directory of the data gateway and the storage node, and the task information includes the information of the source machine, the data gateway and the target storage node.

[0012] Preferably, in the data structure layer, the data organization structure used for sharding transmission of the source machine data includes metadata and data. The metadata records the source machine information and the corresponding block of the data.

[0013] Preferably, the data structure layer adopts block-level data processing technology and reads according to a block size of 2MB to 32MB when reading data.

[0014] Preferably, the TCP / IP data transmission layer supports bidirectional transmission and unidirectional transmission. In the bidirectional transmission, the network between the two nodes for transmission is bidirectional. In the unidirectional transmission, the network between the two nodes for transmission is unidirectional. The unidirectional transmission function of the source end and the target end is realized by establishing a reverse transmission tunnel through the https mode.

[0015] Preferably, the implementation mode of the relay transmission is as follows: a data gateway is set on the user side, the source machine data is cached to the data gateway by using the intranet high-speed network environment, and the cached data is transmitted to the target storage node through the Internet, VPN or dedicated line. The cached data is stored in the form of a data cache file in a specified directory, and the data cache file that completes the transmission is deleted.

[0016] Preferably, the implementation mode of the cascade transmission is as follows: the source end data is transmitted to the data storage node B, the data storage node B transmits the data to the data storage node C while writing the data to the disk, and the data increment from the data storage node B to the data storage node C is defined by the transmission task management layer.

[0017] Preferably, the implementation mode of the parallel transmission is as follows: The source data is transmitted to the data storage node B and the data storage node C through the data gateway, and the data gateway deletes the data cache file after confirming that the transmission of the data storage node B and the data storage node C is completed.

[0018] Preferably, the method further comprises a replication task creation process, specifically: Register the source machine, data gateway and storage node information; Prepare target instance, disk and other related resources; Confirm the connectivity of the storage node to the data gateway; Check the disk status and corresponding relationship of the source and target ends; Submit the task to the management console, and the management console issues the task to the data gateway and the storage node; Start the source reading task, and confirm that the task UUID is consistent with the data gateway; Calculate the amount of source data; Read and transmit data; Complete data transmission and writing.

[0019] A data processing system comprises: A transmission task management module for realizing the function of the transmission task management layer; A data structure processing module for realizing the function of the data structure layer; A TCP / IP data transmission module for realizing the function of the TCP / IP data transmission layer.

[0020] Compared with the prior art, the present application has the following advantages: The present application adopts block-level data processing technology, bypasses the file system level, reduces metadata operations and random access overhead, processes continuous block data in a single I / O, reduces the number of operations, improves data reading efficiency, and improves the RPO index from one day to minutes.

[0021] The present application designs bidirectional and unidirectional transmission technology, supports VPN, dedicated line, Internet and other network scenarios, meets the unidirectional transmission needs of office networks to the Internet, and adapts to complex network architectures in a multi-cloud environment.

[0022] The present application realizes multiple modes such as relay transmission, cascade transmission and parallel transmission through hierarchical design, supports data caching, relay, cascade and parallel transmission, and meets the data transmission needs in different scenarios.

[0023] The present application is based on the design of the task management layer, realizes the ability of concurrent replication tasks of multiple hosts, ensures the order of transmission data, and improves the overall transmission efficiency of the system.

[0024] The application block-level read-write precision control buffer size, reduces memory occupation and CPU load, reduces data transmission volume, synchronizes only the changed block, saves transmission bandwidth and resource consumption. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The method step flowchart of the application; Figure 2 The replication task creation flowchart of the application; Figure 3 The system architecture schematic diagram of the application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work belong to the protection scope of the application.

[0027] Embodiment 1: As shown in the figure, a data caching and transmission method based on TCP / IP includes the following steps: Figures 1-3 Constructing a layered architecture of a transmission task management layer, a data structure layer and a TCP / IP data transmission layer; The transmission task management layer manages transmission tasks in units of whole machines; The data structure layer performs fragmentation processing on source machine data; The TCP / IP data transmission layer realizes data transmission based on the TCP / IP protocol; The transmission task management layer records the relevant information and states of the read task, the write task and the recovery task of each replication task, realizes the control of the transmission process, and completes the relay transmission, the caching transmission, the cascade transmission and the parallel transmission. The transmission task management layer dynamically allocates system resources, and the calculation formula is:

[0028] Among them, is the resource allocation amount of the task, is the current system resource capacity, which refers to the actual allocatable resource amount of the system at this moment, and is affected by the used system resources and temporary load fluctuations. For example, the total system resources are 100, 30 are used, and may be 70; if a task suddenly releases resources, it will be dynamically increased; ​Priority is a quantitative description of the "importance and urgency" of a transmission task, without fixed physical units, similar to "rank division". For example, high-priority tasks (such as emergency service data transmission) The value is set to 5, and the ordinary task is set to 3. The higher the value, the more important the task needs to be prioritized; The priority of the task determines the "tendency" of resource allocation. The higher the priority of the task, the more system resources it is likely to occupy in the formula. Total system resources, intelligent scheduling of transit transmission, cache transmission, cascade transmission and parallel transmission; The maximum resource limit that the system can theoretically support is relatively fixed (unless the system is expanded). For example, the total sum of hardware, bandwidth and other resources of a set of transmission systems is determined at the time of factory planning, which is the "total ceiling" of resource allocation.

[0029] In the transmission task management layer, each task is marked by a unique task UUID, and each source machine is marked by a unique host UUID. The relevant information and status of the task are stored in the form of a file in the specific task directory of the data gateway and storage node, and the task information includes the information of the source machine, data gateway and target storage node.

[0030] In the transmission task management layer, each task is marked by a globally unique identifier, and a distributed hash table is used to store the task status. The consistency hash calculation formula is: Wherein, and is a random number, refers to the "key value" to be hashed, which is the input data of the hash operation, is the size of the hash space: Wherein, is the updated state, is the old state, is the state increment.

[0031] In the data structure layer, the data organization structure used for sharded transmission of source machine data includes metadata and data. The metadata records the source machine information and the corresponding block of the data. The data is the data stored by the source machine block and is stored in the form of a file after compression. The file naming format is <disk_id>- <number><snapshot_id>.

[0032] In the data structure layer, the data organization structure used for the sharded transmission of source machine data includes metadata and data, as follows: Metadata structure: contains multi-level index information, at least including: Source machine system information: operating system type, version, host identifier (Host UUID); Disk topology: number of disks, disk identifier (Disk ID), disk capacity, partition information; Block mapping table: physical address (sector offset), logical address (relative offset), timestamp (creation / modification time) of each data block; Verification information: block hash value (such as SHA-256), CRC checksum, data integrity flag.

[0033] The data structure layer adopts block-level data processing technology, including: Dynamic block size adaptive mechanism: Automatically adjust block size based on source machine IOPS performance, ranging from 2MB to 32MB; Data prefetching strategy: Implement block pre-reading based on the locality principle, predict the probability of adjacent block access; Use double buffer mechanism to asynchronously prefetch the next block while processing the current block; Support multi-thread parallel reading, the number of threads is dynamically adjusted according to the number of CPU cores.

[0034] Error recovery mechanism: Each block contains forward error correction code (FEC) to support single-bit error automatic correction; Use redundancy verification mechanism, trigger block retransmission when CRC verification fails; Record the block position of failed reading, support breakpoint resume.

[0035] Data storage format: After the source machine block data is processed by the LZ4 / Zstandard compression algorithm, binary data blocks are generated; Each data block is attached with header information, including: Metadata pointer: points to the associated block mapping table entry; Compression parameters: compression algorithm type, compression level, original data size; Transmission control information: priority flag, retransmission strategy, timeout threshold.

[0036] Data file naming rule: Use the format <disk_id>-<block_number>_<snapshot_id>, where: <disk_id> is the unique identifier of the disk (e.g., disk01, nvme0n1); <block_number> is a 6-digit zero-padded serial number (e.g., 000001, 000002) that increments in reading order; <snapshot_id> is a timestamp or version number (e.g., 20250617T120000Z).

[0037] Data indexing mechanism: Build a global index file to record the metadata summary of all shard files; Use B + tree structure to realize fast retrieval of index, support multi-dimensional query by time range, disk ID, hash value, etc.; Index file and data file are updated synchronously, and differential incremental index is generated regularly.

[0038] The storage structure of metadata and data files also includes: Multi-level cache strategy: Level 1 cache: Memory LRU cache, stores recently accessed metadata and hot data blocks; Level 2 cache: SSD cache, stores frequently accessed complete files; Level 3 cache: HDD storage, archives low-frequency access historical version data.

[0039] Data lifecycle management: Based on access frequency, automatically migrate data to the appropriate storage level; Set data retention period, expired data automatically triggers archiving or deletion process; Support data tiered storage strategy, key data retention multiple versions.

[0040] Concurrent control mechanism: Use read-write lock separation mechanism, allow multiple read operations to execute concurrently; Write operation uses exclusive lock to ensure data consistency; Implement transactional operations to ensure atomic update of metadata and data files.

[0041] The data structure layer also includes data encryption and security mechanisms, specifically: Transmission encryption: Use TLS 1.3 protocol to encrypt data blocks end-to-end during transmission; Use ECDH key exchange algorithm to generate session key, key length not less than 256 bits; Supports certificate verification mechanisms to prevent man-in-the-middle attacks.

[0042] Static encryption: Data files are encrypted using the AES-256-GCM algorithm during storage; Key Management System (KMS) generates and manages encryption keys; It supports a key rotation mechanism to update the encryption key periodically.

[0043] Access control: The role-based access control (RBAC) model defines roles such as administrator, operator, and auditor. Implement fine-grained access control, with different roles having different operation permissions for metadata and data files; Record complete operation audit logs and support security event backtracking.

[0044] The data structure layer uses block-level data processing technology, reading data in blocks ranging from 2MB to 32MB.

[0045] The data structure layer uses an adaptive block size algorithm, and the optimal block size is calculated using the following formula: in, The optimal block size refers to the block size calculated using this formula that allows for optimal efficiency in data read and write operations (considering disk access and network transmission scenarios). Disk access time, Total data volume This refers to network transmission time.

[0046] The TCP / IP data transmission layer supports both bidirectional and unidirectional transmission. In bidirectional transmission, the network between the two nodes can pass through each other in both directions. In unidirectional transmission, the network between the two nodes can pass through each other in one direction only, and the receiver does not need to establish a reverse link. The unidirectional transmission function between the source and the destination can be achieved by establishing a reverse transmission tunnel through HTTPS.

[0047] Table 1: Layered Architecture of Transmission Task Management Layer, Data Structure Layer, and TCP / IP Data Transmission Layer Transmission task management layer According to the whole machine as a unit, the transmission management is carried out when the transmission is carried out, and each source machine will have a corresponding transmission task management. Data structure layer The data organization structure used for the data transmission of the source machine, that is, the metadata and data. TCP / IP data transmission layer TCP / IP-based data transmission mode, including peer-to-peer duplex transmission, also including one-way transmission. The implementation mode of the relay transmission is that a data gateway is arranged at the user side, source machine data is cached to the data gateway by using the intranet high-speed network environment, and the cached data is transmitted to the target storage node through the Internet, VPN or dedicated line, wherein the cached data is stored in the form of data cache files in a specified directory, and the data cache files completed transmission are deleted.

[0048] The relay transmission adopts a multi-level cache strategy, and the cache hit rate calculation formula is: wherein, is the cache hit times, is the total access times.

[0049] The implementation mode of the cascade transmission is that the source end data is transmitted to the data storage node B, the data storage node B transmits the data to the data storage node C while writing the data to the disk, wherein the data increment from the data storage node B to the data storage node C is defined by the transmission task management layer.

[0050] Table 2: Definition table related to data transmission task jobs .prt Reading task .prt.status Reading task status .wrt Writing task .wrt.status Writing task status .wrt.progress Writing task progress .prv Recovery task .prv.status Recovery task status The cascade transmission adopts a pipeline parallel processing, and the data processing delay calculation formula is: wherein, is the i-th stage processing time, is the i-th stage parallelism, and n is the number of processing stages.

[0051] The implementation mode of the parallel transmission is: The source end data is transmitted to the data storage node B and the data storage node C through the data gateway at the same time, and the data gateway deletes the data cache file after confirming the completion of the transmission of the data storage node B and the data storage node C.

[0052] The method further includes a copy task creation process, specifically: Registering the source machine, data gateway and storage node information; Preparing target instances, disks and other related resources; Confirming the connectivity of the storage node to the data gateway; Checking the disk state and corresponding relationship of the source end and the target end; Submitting the task to the management console, and the management console issues the task to the data gateway and the storage node; Starting the source end reading task, and confirming that the task UUID is consistent with the data gateway; Calculating the source end data volume; Reading and transmitting data; Complete data transfer and write.

[0053] A data processing system, comprising: The transmission task management module is used to implement the functions of the transmission task management layer; The data structure processing module is used to implement the functions of the data structure layer; The TCP / IP data transmission module is used to implement the functions of the TCP / IP data transmission layer.

[0054] Specific Implementation Example 1: Relay Transmission Scenario In a certain enterprise multi-cloud architecture, the local data center needs to migrate business data to a public cloud platform. Due to the limited transmission bandwidth from the local data center to the public network, the relay transmission architecture of this invention is adopted: Deploy a data gateway in the local data center to quickly read data from the source server using the high-speed intranet environment (10Gbps), generate data cache files in chunks, and store them on the data gateway.

[0055] The data gateway transmits the cached files to the public cloud storage node step by step via a public dedicated line (100Mbps), and deletes the local cached files after the transmission is complete.

[0056] After receiving the cache file, the storage node writes the data to the target disk and then deletes the cache file.

[0057] In this embodiment, by leveraging the caching function of the data gateway, the high bandwidth requirement that originally needed to be transmitted directly from the source end through the public network is transformed into a combination of high-speed internal network caching and low-speed public network transmission, reducing the impact on the source end and improving transmission efficiency.

[0058] Specific Implementation Example 2: Cascading Transmission Scenario A financial institution needs to transfer core business data from its local data center (source A) to a local disaster recovery center (storage node B), and then to a remote disaster recovery center (storage node C): Data from source A is transmitted to the data gateway via the intranet, where it is cached to generate a data cache file.

[0059] The data gateway transmits the cached file to storage node B. After receiving it, B writes it to its local disk and simultaneously transmits the cached file to storage node C.

[0060] After receiving the cached file, storage node C writes it to the local disk, completing the cascading transfer.

[0061] By recording and managing the task status of each node through the task management layer, the data increment from B to C can be accurately identified, ensuring the long-term stable operation of cascaded transmission.

[0062] Specific Implementation Example 3: Parallel Transmission Scenario An internet company needs to backup user data to two different cloud service providers (storage nodes B and C) simultaneously: The source data generates a data cache file through the data gateway cache.

[0063] The data gateway transmits the cache file to storage nodes B and C simultaneously.

[0064] The two storage nodes receive the cache file and write it to the local disk respectively, and the data gateway deletes the cache file after confirming the transmission is completed.

[0065] This embodiment realizes parallel transmission of data, improves backup efficiency, and ensures the safety of multiple copies of data.

[0066] Although embodiments of the present application have been shown and described, it is to be understood that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the present application, the scope of which is defined by the appended claims and their equivalents.< / number>

Claims

1. A data caching and transmission method based on TCP / IP, characterized in that, Includes the following steps: Construct a layered architecture consisting of a transmission task management layer, a data structure layer, and a TCP / IP data transmission layer; The transmission task management layer manages transmission tasks on a per-machine basis. The data structure layer performs data segmentation processing on the source machine; The TCP / IP data transmission layer implements data transmission based on the TCP / IP protocol; The transmission task management layer records the relevant information and status of each copy task's read, write, and restore tasks, thereby controlling the transmission process to complete relay transmission, buffered transmission, cascaded transmission, and parallel transmission.

2. The TCP / IP-based data caching and transmission method according to claim 1, characterized in that, In the transmission task management layer, each task is marked with a unique task UUID, and each source machine is marked with a unique host UUID. The relevant information and status of the task are stored in the form of files in specific task directories of the data gateway and storage nodes, and the task information includes information about the source machine, data gateway and target storage node.

3. The TCP / IP-based data caching and transmission method according to claim 1, characterized in that, In the data structure layer, the data organization structure used for fragmented transmission of source machine data includes metadata and data. The metadata records source machine information and the blocks corresponding to the data.

4. The TCP / IP-based data caching and transmission method according to claim 3, characterized in that, The data structure layer employs block-level data processing technology, reading data in blocks ranging from 2MB to 32MB.

5. The TCP / IP-based data caching and transmission method according to claim 1, characterized in that, The TCP / IP data transmission layer supports both bidirectional and unidirectional transmission. In bidirectional transmission, the network between the two nodes can communicate in both directions, while in unidirectional transmission, the network between the two nodes can communicate in only one direction. A reverse transmission tunnel is established via HTTPS to achieve unidirectional transmission between the source and destination.

6. The TCP / IP-based data caching and transmission method according to claim 1, characterized in that, The relay transmission is implemented as follows: a data gateway is set up on the user side, and the source machine data is cached to the data gateway using the high-speed intranet network environment. The cached data is then transmitted to the target storage node via the Internet, VPN, or dedicated line. The cached data is stored in a specified directory in the form of a data cache file, and the data cache file is deleted after the transmission is completed.

7. The TCP / IP-based data caching and transmission method according to claim 1, characterized in that, The cascading transmission is implemented as follows: the source data is transmitted to data storage node B, and data storage node B transmits the data to data storage node C while writing the data to the disk. The data increment from data storage node B to data storage node C is defined by the transmission task management layer.

8. The TCP / IP-based data caching and transmission method according to claim 1, characterized in that, The parallel transmission is implemented as follows: The source data is transmitted simultaneously to data storage node B and data storage node C through the data gateway. After confirming that the transmission between data storage node B and data storage node C is complete, the data gateway deletes the data cache file.

9. The TCP / IP-based data caching and transmission method according to any one of claims 1 to 8, characterized in that, The method also includes a copy task creation process, specifically: Register the source machine, data gateway, and storage node information; Prepare the target instance, disk, and other relevant resources; Confirm connectivity between the storage node and the data gateway; Check the disk status and correspondence between the source and target ends; Submit the task to the management console, which will then distribute the task to the data gateway and storage nodes. Start the source-side read task and confirm that the task UUID is consistent with the data gateway; Calculate the amount of data at the source end; Read and transfer data; Complete data transfer and write.

10. A data processing system, characterized in that, The method for implementing the TCP / IP-based data caching and transmission method according to any one of claims 1 to 9 includes: The transmission task management module is used to implement the functions of the transmission task management layer; The data structure processing module is used to implement the functions of the data structure layer; The TCP / IP data transmission module is used to implement the functions of the TCP / IP data transmission layer.

Images