Data transmission platform and method

Abstract

The present invention discloses a data transmission platform and method. The data transmission platform is deployed in each cluster, and each data transmission platform includes a scheduler and at least one node; the scheduler realizes a singleton through the election of each node, which is used to perform unified flow control and priority scheduling on the data transmission task; each node includes a receptionist, a transmission client and a transmission server; the receptionist is used to receive the data transmission task issued by the scheduling client and forward it to the scheduler; the transmission client is used to receive and execute the data transmission task issued by the scheduler; the transmission server is used to receive the data transmitted by the transmission client. The scheduler can perform unified flow control and priority scheduling on the data transmission task, ensuring that the network bandwidth between clusters will not be fully occupied when multiple users transmit files at the same time, and ensuring that files with higher business priority are transmitted first; the scheduler realizes a singleton through the election of multiple nodes, and will re-elect when the node fails to achieve high-availability scheduling.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data transmission platform and method. Background Art

[0002] Enterprises often need to transfer file data across multiple groups during production and big data business processing, and they all have the following pain points:

[0003] (1) The data volume is large and difficult to process;

[0004] (2) Cross-region transmission, unstable network, and high requirements for the reliability and consistency of file data transmission;

[0005] (3) Bandwidth is limited. If data transmission takes a long time and the amount of data is large, it is easy to squeeze the network bandwidth between clusters, causing other services that require real-time communication to be affected. Even if a limited transmission is set for a single task, when multiple users transmit at the same time, it will still cause the network bandwidth between clusters to be squeezed;

[0006] (4) Existing tools generally do not have centralized bandwidth management and priority control. When multiple users are involved, they will occupy limited bandwidth resources and affect each other.

[0007] (5) For security reasons, inter-cluster network access is limited to one-way access, and even networks cannot communicate directly and can only be accessed indirectly through a jump server;

[0008] (6) Across file systems, files in multiple file systems such as S3 / FTP / SFTP / HDFS need to be able to be transferred to each other.

[0009] When transferring file data, data transfer based on FTP / SFTP protocol is the simplest transmission method, but there is a single point problem. Faced with terabyte-level data volume and unreliable network environment, single nodes are often stretched, and any network fluctuations may cause file transfer interruption and affect reliability. And it does not support cross-file system transmission. DistCp is an inter-cluster file transfer tool for Hadoop ecosystem provided by Apache, which can achieve large-scale data migration. However, this tool does not support transmission bandwidth restrictions, and needs to ensure network interconnection between clusters. It does not support inter-cluster migration with Kerberos enabled but asynchronous clocks. In addition, support for file systems outside the Hadoop ecosystem is also a problem.

[0010] Therefore, the inventors realized that there is an urgent need for a high-availability data transmission platform that can support reliable transmission of large amounts of data, has complete current limiting and priority management, supports proxy transmission, and supports multiple file systems. Summary of the invention

[0011] Based on this, in order to solve the above technical problems, a high-availability data transmission platform that can support flow limiting and priority management is provided to solve the problem that the existing data transmission platform in the prior art has no centralized bandwidth management and priority control.

[0012] In order to achieve the above objectives, this application provides the following technical solutions:

[0013] In a first aspect, a data transmission platform is deployed in each cluster, and the data transmission platform of each cluster includes a scheduler and at least one node; the scheduler implements a single instance through election of each node, and is used to perform unified flow control and priority scheduling on received data transmission tasks; each node includes:

[0014] A receptionist is used to communicate with the scheduling client and the scheduler, receive the data transmission task issued by the scheduling client and forward it to the scheduler; the scheduling client is created on the local client or on the data transmission platform;

[0015] The transmission client is used to communicate with the scheduler, receive and execute the data transmission task issued by the scheduler, transmit the data to the transmission server of the data transmission platform of the target cluster, and return the execution result to the scheduler;

[0016] The transmission server is used to communicate with the transmission client, receive data transmitted by the transmission client, and automatically verify the file after the data transmission is completed, and return the verification result to the transmission client.

[0017] Optionally, the transmission client is further used to create a transmission channel with a transmission server of the data transmission platform of the target cluster, and transmit the data to the transmission server of the data transmission platform of the target cluster through the transmission channel.

[0018] Further optionally, the transmission channel is used to automatically limit the transmission according to the flow limit set by the scheduler, and supports multiple data compression formats.

[0019] Further optionally, the transmission channel includes a back pressure controller, a ring buffer, a back end monitor, a compressor, a decompressor, a frame encoder, a frame decoder, a front end monitor, a write buffer and a read buffer.

[0020] Further optionally, the message writing process in the transmission channel includes:

[0021] The external system writes data to the transmission channel through data streams;

[0022] The back pressure controller determines whether the write buffer is in a writable state based on the write buffer usage provided by the front-end monitor; if the write buffer is not writable, the write-end back pressure is triggered and blocked waiting is performed;

[0023] The backend monitor counts the amount of data read and written by the message;

[0024] The compressor compresses the message;

[0025] The compressed message is encoded by a frame encoder, and the compressed message is encapsulated into data frames one by one;

[0026] The front-end monitor counts the actual amount of data transmitted over the network and the usage of the write buffer;

[0027] The message is written to the write buffer and is ready to be sent;

[0028] The reading process of the message in the transmission channel includes:

[0029] Network messages are written into the read buffer;

[0030] The front-end monitor counts the amount of data received by the network and the usage of the read buffer;

[0031] The frame decoder decodes the network message and generates a data frame;

[0032] The decompressor decompresses the data frame to obtain the data actually transmitted;

[0033] The backend monitoring area obtains the actual message size read and the usage of the ring buffer;

[0034] The back pressure controller determines whether the ring buffer is full. If it is full, it triggers the read-side back pressure and waits for the buffer to have free space before writing the data to the ring buffer.

[0035] The external system reads the data in the ring buffer through data stream for subsequent processing.

[0036] Optionally, the data transmission platform adopts the GOSSIP protocol.

[0037] Optionally, the transmission client and the transmission server support two data transmission modes: PUSH and PULL.

[0038] Further optionally, the receptionist is also used to query the client for the file information that needs to be migrated, so as to facilitate scheduling the client to create a data migration task.

[0039] In a second aspect, a data transmission method is applied to the data transmission platform provided in the first aspect, the method comprising:

[0040] The receptionist of cluster A receives the first data transmission task issued by the first scheduling client created on the local client, and forwards it to the scheduler of cluster A;

[0041] The scheduler of cluster A performs task verification and priority scheduling on the received first data transmission task, registers occupied bandwidth for the first data transmission task, and then sends the first data transmission task to the transmission client of cluster A;

[0042] The transmission client of cluster A establishes a connection with the transmission server of target cluster B, and transmits data to the transmission server of target cluster B according to the bandwidth size registered by the scheduler; the transmission client of cluster A pushes the transmission progress to the first scheduling client through the receptionist of cluster A in real time;

[0043] When the transmission is completed, the transmission client of cluster A receives the verification result returned by the transmission server of target cluster B;

[0044] After the transmission is completed, the transmission client of cluster A returns the task result to the scheduler of cluster A. After the scheduler of cluster A releases the bandwidth occupied by the first data transmission task, it returns the final result of the first data transmission task to the first scheduling client through the receptionist of cluster A.

[0045] Optionally, the method further comprises:

[0046] When the transmission server of the target cluster B receives the data sent by the transmission client of cluster A, it creates a second scheduling client on the target cluster B, and the second scheduling client sends a second data transmission task to the receptionist of the target cluster B;

[0047] The receptionist of the target cluster B receives the second data transmission task sent by the second scheduling client, and forwards it to the scheduler of the target cluster B;

[0048] The scheduler of the target cluster B performs task verification and priority scheduling on the received second data transmission task, registers the occupied bandwidth for the second data transmission task, and then sends the second data transmission task to the transmission client of the target cluster B;

[0049] The transmission client of target cluster B establishes a connection with the transmission server of target cluster C, and transmits data to the transmission server of target cluster C according to the bandwidth size registered by the scheduler; the transmission client of target cluster B pushes the transmission progress to the second scheduling client through the receptionist of cluster B in real time;

[0050] When the transmission is completed, the transmission client of target cluster B receives the verification result returned by the transmission server of target cluster C;

[0051] After the transmission is completed, the transmission client of the target cluster B returns the task result to the scheduler of the target cluster B. After the scheduler of the target cluster B releases the bandwidth occupied by the second data transmission task, it returns the final result of the second data transmission task to the second scheduling client through the receptionist of the target cluster B.

[0052] The present invention has at least the following beneficial effects:

[0053] The embodiment of the present invention provides a data transmission platform, which is deployed in each cluster, and the data transmission platform of each cluster includes a scheduler and at least one node; the scheduler realizes a singleton through the election of each node, and is used for unified flow control and priority scheduling of the received data transmission tasks; each node includes a receptionist, a transmission client and a transmission server; the receptionist is used to receive the data transmission task issued by the scheduling client and forward it to the scheduler; the transmission client is used to receive and execute the data transmission task issued by the scheduler, and transmit the data to the transmission server of the data transmission platform of the target cluster; the transmission server is used to receive the data transmitted by the transmission client, and automatically verify the file after the data transmission is completed, and return the verification result to the transmission client; the scheduler in the data transmission platform can perform unified flow control and priority scheduling for the received data transmission tasks, and the scheduler uniformly allocates the flow resources to ensure that the network bandwidth between clusters will not be fully occupied when multiple users transmit files at the same time, and ensure that the normal operation of other services that require network access is not affected; the scheduler performs unified scheduling according to the priority of the transmission task to ensure that the files with higher service priority are transmitted first; the scheduler realizes a singleton through the election of multiple nodes, and when the node fails, each node will re-elect a new scheduler node to achieve high-availability scheduling.

[0054] The data transmission platform adopts a distributed implementation solution based on the GOSSIP protocol, which makes the data of all nodes in the system consistent, and has the characteristics of decentralization, high scalability, strong fault tolerance, and eventual consistency. Based on this protocol, the output transmission platform can achieve dynamic scaling during runtime, and automatically eliminate faulty nodes when running, to ensure high availability and efficient operation of the system.

[0055] The transmission client and transmission server support two data transmission modes: PUSH and PULL. That is, they support both pushing data from the local cluster to the target cluster and pulling data from the target cluster. They can also realize two-way data transmission in a network environment with only one-way access. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A logical architecture diagram of a data transmission platform provided for one embodiment of the present invention;

[0057] Figure 2 A structural diagram of a transmission channel in one embodiment of the present invention;

[0058] Figure 3 A timing diagram of a data transmission process in one embodiment of the present invention;

[0059] Figure 4 A schematic diagram of a general file system protocol in one embodiment of the present invention;

[0060] Figure 5 A schematic diagram of the process of PUSH data transmission in proxy mode in one embodiment of the present invention;

[0061] Figure 6 A schematic diagram of a process flow of PULL data transmission in proxy mode in one embodiment of the present invention;

[0062] Figure 7 A schematic diagram of a data transmission method according to an embodiment of the present invention;

[0063] Figure 8 A task execution flow sequence diagram of a data transmission method in one embodiment of the present invention;

[0064] Fig. 9 This is a proxy mode transmission logic diagram in one embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] In one embodiment, Figure 1 As shown, a data transmission platform is provided. A set of the data transmission platform is deployed in each cluster. The data transmission platform of each cluster includes a scheduler and at least one node (1 or more nodes); each node includes roles such as receptionist / transmission client / transmission server, and the scheduler role exists as a singleton in the transmission platform.

[0067] Specifically, the data migration platform is distributed, and the scheduler is implemented as a singleton through elections among the nodes in the platform, and is responsible for unified flow control and priority scheduling of the received data transmission tasks. To ensure the consistency of flow control and priority scheduling, each data transmission platform has only one scheduler role; when the node where the scheduler is located fails, a new scheduler role will be re-elected to achieve high-availability task scheduling.

[0068] Each node consists of:

[0069] The receptionist is used to communicate with the scheduling client and the scheduler, receive the data transmission tasks issued by the scheduling client and forward them to the scheduler; the scheduling client is created on the local client or on the data transmission platform;

[0070] The transmission client is used to communicate with the scheduler, receive and execute the data transmission tasks issued by the scheduler, transmit the data to the transmission server of the data transmission platform of the target cluster, and return the execution results to the scheduler;

[0071] The transmission server is used to communicate with the transmission client, receive data transmitted by the transmission client, and automatically verify the file after the data transmission is completed, and return the verification result to the transmission client.

[0072] That is to say, users communicate with the data transmission platform of the current cluster through the scheduling client to submit data transmission tasks, apply for bandwidth resources, check the task running status, etc. The data transmission platform will also create a scheduling client to communicate with the data transmission platform of other clusters. The file transfer task of the scheduling client will first be submitted to the scheduler, and then sent to the transmission client for actual task transmission, or the task will be rejected due to flow control or priority restrictions.

[0073] The receptionist is the receptionist of the cluster. When the scheduling client establishes a connection with the cluster for the first time, it will first communicate with the receptionist. The receptionist can obtain the address information of the scheduler in real time, so when the scheduling client wants to communicate with the scheduler, the receptionist can forward the communication message to the scheduler. In addition, the receptionist is also responsible for querying the file information in the file system that needs to be migrated for the client, so as to facilitate the scheduling client to create data migration tasks.

[0074] Furthermore, the transmission client will also create a transmission channel with the transmission server of the data transmission platform of the target cluster, and transmit the data to the transmission server of the data transmission platform of the target cluster through the transmission channel to perform file transfer. After the transmission server and the transmission client establish a transmission channel, the file transfer task will begin. In other words, the transmission client and the server use the transmission channel to achieve two-way data transmission and message communication. The transmission channel will automatically limit the transmission according to the flow limit set by the scheduler, and support multiple data compression formats to achieve efficient data transmission.

[0075] Furthermore, the transmission channel includes a back pressure controller, a ring buffer, a back end monitor, a compressor, a decompressor, a frame encoder, a frame decoder, a front end monitor, a write buffer and a read buffer. Figure 2 The internal structure design of the transmission channel is demonstrated, and users can read and write data to the transmission channel in the form of data streams.

[0076] The message writing process is as follows:

[0077] 1. The external system writes data to the transmission channel through data stream;

[0078] 2. The back pressure controller determines whether the write buffer is in a writable state based on the write buffer usage provided by the front-end monitor; if the write buffer is not writable, the write-end back pressure is triggered and blocked waiting is performed;

[0079] 3. The backend monitor counts the amount of data read and written by the message;

[0080] 4. The compressor compresses the message;

[0081] 5. The compressed message is encoded by a frame encoder, and the compressed message is encapsulated into data frames one by one to prevent network packet sticking and unpacking from causing decompression failure;

[0082] 6. The front-end monitor counts the actual amount of data transmitted over the network and the usage of the write buffer;

[0083] 7. The message is written to the write buffer and is ready to be sent.

[0084] The message reading process is as follows:

[0085] 1. Network messages are first written into the read buffer;

[0086] 2. The front-end monitor counts the amount of data received by the network and the usage of the read buffer;

[0087] 3. The frame decoder decodes the network message and generates a data frame;

[0088] 4. The decompressor decompresses the data frame to obtain the actual transmitted data;

[0089] 5. The backend monitoring area obtains the actual message size read and the usage of the ring buffer;

[0090] 6. The back pressure controller determines whether the ring buffer is full. If it is full, it means that the data reading speed is slow, so the read-end back pressure is triggered, and the data is written to the ring buffer when there is remaining space in the buffer in a round-robin manner;

[0091] 7. The external system reads the data in the ring buffer through data stream for subsequent processing.

[0092] Furthermore, if Figure 3 As shown, the transmission client and the transmission server support two data transmission modes: PUSH and PULL.

[0093] Furthermore, if Figure 4As shown in the figure, the data transmission platform provides a "universal file system protocol" that provides common file system operations such as query, copy, create directory, delete, upload, download, etc. It also provides implementations of this protocol for a variety of common file systems, including S3, HDFS, FTP, SFTP, local file system, etc. The data transmission platform uses this protocol to achieve high-availability and high-performance data transmission across file systems and clusters.

[0094] The general file system protocol also provides the implementation of the proxy mode. The purpose of data transmission in the proxy mode is to achieve data transmission in an isolated network environment. That is, clusters A and C are not interconnected. Through a proxy cluster B, the data of cluster A is transmitted to cluster B, and then from cluster B to cluster C, thereby achieving proxy-based data transmission.

[0095] When users use the proxy mode to transfer data, it is like sending the data to a new file system. In other words, the implementation of the proxy mode is based on Figure 4 The “general file system protocol” shown in the figure means that cluster B abstracts the cluster it proxies into an independent file system.

[0096] In proxy mode, the transmission client and the transmission server also support PUSH and PULL data transmission modes. The flow diagram of PUSH data transmission in proxy mode is as follows: Figure 5 As shown, the flow chart of PULL data transmission in proxy mode is as follows Figure 6 shown.

[0097] In general, the data transmission platform provided by the present invention has the following technical features:

[0098] 1. High Availability

[0099] The data transmission platform adopts a distributed implementation solution based on the GOSSIP protocol, which can dynamically and quickly scale nodes. When a node fails, it can automatically remove abnormal nodes to ensure that the platform can provide services normally 24 hours a day and the data transmission tasks can run normally.

[0100] The scheduler implements a singleton through multi-node election. That is, at a certain time, only one node has the scheduler role. When the node fails, each node will re-elect a new scheduler node to achieve high-availability scheduling.

[0101] GOSSIP ensures dynamic scaling of nodes: GOSSIP is a common consistency protocol in distributed systems. It uses random and infectious methods to spread information to the entire network and make all node data in the system consistent within a certain period of time. It has the characteristics of decentralization, high scalability, strong fault tolerance, and final consistency. Based on this protocol, the output transmission platform can achieve dynamic scaling at runtime, automatically remove faulty nodes when running nodes, and ensure high availability and efficient operation of the system.

[0102] Vector clocks ensure the consistency of each node state: Vector clocks are a data structure and a technology that generates partial order values ​​for various operations or events in a distributed environment. They can detect parallel conflicts and causal violations of operations or events to maintain system consistency. The data transmission platform uses vector clocks to ensure that there is only one scheduler instance at a certain moment, and thereby achieve unified priority scheduling and unified flow control.

[0103] (ii) High performance

[0104] At the platform architecture level, dynamic scaling and multi-channel parallel data transmission can be achieved, greatly improving the efficiency and performance of data transmission;

[0105] At the bottom layer, the data transmission platform supports virtual memory mapping technology (mmap). During the data transmission process, it can reduce operating system context switching, improve transmission performance, and reduce the CPU and memory resource usage of the process. It also supports multiple data compression formats to reduce bandwidth usage while transmitting the same amount of data.

[0106] (III) High reliability

[0107] It supports automatic retry of tasks that fail in the network or time out, and automatic comparison and verification after the file transfer is completed to ensure the reliability and consistency of file transfer.

[0108] (IV) High Practicality

[0109] Unified traffic control: The scheduler uniformly allocates traffic resources to ensure that the network bandwidth between clusters is not fully occupied when multiple users transfer files at the same time, ensuring that the normal operation of other services that require network access is not affected.

[0110] Unified priority management: the scheduler performs unified scheduling based on the priority of the transmission task, ensuring that files with higher business priority are transmitted first;

[0111] The transmission client and transmission server support two data transmission modes: PUSH and PULL. That is, they support both pushing data from the local cluster to the target cluster and pulling data from the local cluster to the target cluster. They can also realize two-way data transmission in a network environment with only one-way access.

[0112] Support proxy mode. When a cluster is not interconnected and can only be accessed through a jump server, the data transmission platform can still achieve two-way data transmission through proxy mode.

[0113] It supports submitting and managing data transmission tasks through multiple interactive methods such as visual pages, Shell command lines, and JavaAPI, making it easy to use and conduct secondary development.

[0114] In summary, the transmission platform provided by the present invention has high availability and high reliability, which greatly improves the stability of large-scale transmission tasks and reduces the possibility of file transmission failure due to network or hardware problems; supports multi-channel parallel data transmission, virtual memory mapping and data compression, which greatly improves the efficiency of data transmission and reduces the occupation of server resources; unified flow control and unified priority management provide support for the use of multi-tenants in the data transmission platform, and higher priority transmission tasks can be allocated higher transmission bandwidth and transmitted first, and prevent the bandwidth resources of other services from being squeezed out; supports data transmission in various complex network environments, based on the PUSH / PULL data transmission modes, even if the network between clusters can only be accessed unidirectionally, the data transmission platform can still realize two-way data transmission, and based on the agent, it can realize data transmission between clusters that are not interconnected in the network; supports submitting and managing data transmission tasks in multiple ways through visual pages and Shell command lines, reduces the user's operation threshold, and also provides a JavaAPI for task submission and management to provide support for integrating other big data systems.

[0115] In one embodiment, Figure 7 As shown, a data transmission method is provided to perform a data transmission task, which is applied to the data transmission platform provided in the above embodiment. The method includes:

[0116] S701, a receptionist of cluster A receives a first data transmission task issued by a first scheduling client created on a local client, and forwards it to a scheduler of cluster A;

[0117] S702, the scheduler of cluster A performs task verification and priority scheduling on the received first data transmission task, registers the occupied bandwidth for the first data transmission task, and then sends the first data transmission task to the transmission client of cluster A;

[0118] S703, the transmission client of cluster A establishes a connection with the transmission server of target cluster B, and transmits data to the transmission server of target cluster B according to the bandwidth size registered by the scheduler; the transmission client of cluster A pushes the transmission progress to the first scheduling client through the receptionist of cluster A in real time;

[0119] S704, when the transmission is completed, the transmission client of cluster A receives the verification result returned by the transmission server of target cluster B;

[0120] S705, after the transmission is completed, the transmission client of cluster A returns the task result to the scheduler of cluster A. After the scheduler of cluster A releases the bandwidth occupied by the first data transmission task, it returns the final result of the first data transmission task to the first scheduling client through the receptionist of cluster A.

[0121] Figure 8 The data transmission task execution logic of this method is demonstrated. In other words, when the user starts data transmission, he first creates a scheduling client role locally, connects to the receptionist role of the transmission platform of the current cluster, and sends the transmission task to the receptionist, who then forwards it to the scheduler. After receiving the task, the scheduler will perform task verification and priority scheduling, register the occupied bandwidth for the task, and then send the task to the transmission client. The transmission client will create a connection with the transmission server of the target cluster and start data transmission according to the bandwidth size registered by the scheduler. During the transmission process, the transmission progress will be pushed to the scheduling client through the receptionist in real time. When completed, the transmission server will return the verification result to the transmission client. After the transmission is completed, the transmission client will return the task result to the scheduler. After the scheduler releases the bandwidth occupied by the task, it will return the final result of the task to the client through the receptionist, and the entire transmission task is completed.

[0122] Furthermore, the method further comprises:

[0123] When the transmission server of the target cluster B receives the data sent by the transmission client of cluster A, it creates a second scheduling client on the target cluster B, and the second scheduling client sends a second data transmission task to the receptionist of the target cluster B;

[0124] The receptionist of the target cluster B receives the second data transmission task sent by the second scheduling client, and forwards it to the scheduler of the target cluster B;

[0125] The scheduler of the target cluster B performs task verification and priority scheduling on the received second data transmission task, registers the occupied bandwidth for the second data transmission task, and then sends the second data transmission task to the transmission client of the target cluster B;

[0126] The transmission client of target cluster B establishes a connection with the transmission server of target cluster C, and transmits data to the transmission server of target cluster C according to the bandwidth registered by the scheduler; the transmission client of target cluster B pushes the transmission progress to the second scheduling client through the receptionist of cluster B in real time;

[0127] When the transmission is completed, the transmission client of target cluster B receives the verification result returned by the transmission server of target cluster C;

[0128] After the transmission is completed, the transmission client of target cluster B returns the task result to the scheduler of target cluster B. After the scheduler of target cluster B releases the bandwidth occupied by the second data transmission task, it returns the final result of the second data transmission task to the second scheduling client through the receptionist of target cluster B.

[0129] This method can realize data transmission in proxy mode, that is, Fig. 9 As shown in the figure, if cluster A and cluster C are not interconnected, the data transmission platforms of the two clusters cannot achieve direct data transmission. However, if cluster B communicates with clusters A and C respectively, a data transmission platform can be deployed in cluster B and used as a springboard to achieve data transmission between clusters A and C.

[0130] In proxy mode, cluster A will follow Figure 8 The task execution logic shown in the figure communicates with cluster B and transmits data. When cluster B's transmission server receives data from cluster A, it will create a scheduling client and also follow the Figure 8 The task execution logic shown communicates with cluster C and transmits the data received from cluster A to cluster C, thereby realizing data transmission in proxy mode.

[0131] In other words, the process is real-time. There is no need to wait for cluster A to complete the transmission to cluster B before cluster B transmits the data to cluster C. Instead, the data of cluster A is transferred through cluster B and then transmitted to cluster C in real time.

[0132] For cluster A, only Figure 4 The "Common File System Protocol" provided and Figure 8 The process shown in the figure is to transmit data with cluster B, while cluster B is in Figure 5 and Figure 6 During the initialization process, cluster C's address information and file system information are parsed from the protocol, and cluster B then follows Figure 8 The process establishes a connection with cluster C and performs data transmission.

[0133] It should be understood that although Figure 7 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 7At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0134] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, which involves all or part of the processes in the above-mentioned embodiment method.

[0135] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which involves all or part of the processes in the above-mentioned embodiment method.

[0136] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0137] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0138] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A data transmission platform, characterized in that: Deployed in each cluster, the data transmission platform of each cluster includes a scheduler and at least one node; the scheduler realizes a single instance through node election, which is used to perform unified flow control and priority scheduling for received data transmission tasks; each node includes: A receptionist is used to communicate with the scheduling client and the scheduler, receive the data transmission task issued by the scheduling client and forward it to the scheduler; the scheduling client is created on the local client or on the data transmission platform; The transmission client is used to communicate with the scheduler, receive and execute the data transmission task issued by the scheduler, transmit the data to the transmission server of the data transmission platform of the target cluster, and return the execution result to the scheduler; The transmission server is used to communicate with the transmission client, receive the data transmitted by the transmission client, and automatically verify the file after the data transmission is completed, and return the verification result to the transmission client; The transmission client is also used to create a transmission channel with the transmission server of the data transmission platform of the target cluster, and transmit the data to the transmission server of the data transmission platform of the target cluster through the transmission channel; The transmission channel includes a back pressure controller, a ring buffer, a back end monitor, a compressor, a decompressor, a frame encoder, a frame decoder, a front end monitor, a write buffer and a read buffer; The message writing process in the transmission channel includes: The external system writes data to the transmission channel through data streams; The back pressure controller determines whether the write buffer is in a writable state based on the write buffer usage provided by the front-end monitor; if the write buffer is not writable, the write-end back pressure is triggered and blocked waiting is performed; The backend monitor counts the amount of data read and written by the message; The compressor compresses the message; The compressed message is encoded by a frame encoder, and the compressed message is encapsulated into data frames one by one; The front-end monitor counts the actual amount of data transmitted over the network and the usage of the write buffer; The message is written to the write buffer and is ready to be sent; The reading process of the message in the transmission channel includes: Network messages are written into the read buffer; The front-end monitor counts the amount of data received by the network and the usage of the read buffer; The frame decoder decodes the network message and generates a data frame; The decompressor decompresses the data frame to obtain the data actually transmitted; The backend monitoring area obtains the actual message size read and the usage of the ring buffer; The back pressure controller determines whether the ring buffer is full. If it is full, it triggers the read-side back pressure and waits for the buffer to have free space before writing the data to the ring buffer. The external system reads the data in the ring buffer through data stream for subsequent processing.

2. The data transmission platform according to claim 1, characterized in that: The transmission channel is used to automatically limit the transmission according to the flow limit set by the scheduler, and supports multiple data compression formats.

3. The data transmission platform according to claim 1, characterized in that: The data transmission platform adopts the GOSSIP protocol.

4. The data transmission platform according to claim 1, characterized in that: The transmission client and the transmission server support two data transmission modes: PUSH and PULL.

5. The data transmission platform according to claim 1, characterized in that: The receptionist is also used to query the client for the file information that needs to be migrated, so as to facilitate scheduling the client to create a data migration task.

6. A data transmission method, characterized in that: Applied to the data transmission platform according to any one of claims 1 to 5, the method comprises: The receptionist of cluster A receives the first data transmission task issued by the first scheduling client created on the local client, and forwards it to the scheduler of cluster A; The scheduler of cluster A performs task verification and priority scheduling on the received first data transmission task, registers occupied bandwidth for the first data transmission task, and then sends the first data transmission task to the transmission client of cluster A; The transmission client of cluster A establishes a connection with the transmission server of target cluster B, and transmits data to the transmission server of target cluster B according to the bandwidth size registered by the scheduler; the transmission client of cluster A pushes the transmission progress to the first scheduling client through the receptionist of cluster A in real time; When the transmission is completed, the transmission client of cluster A receives the verification result returned by the transmission server of target cluster B; After the transmission is completed, the transmission client of cluster A returns the task result to the scheduler of cluster A. After the scheduler of cluster A releases the bandwidth occupied by the first data transmission task, it returns the final result of the first data transmission task to the first scheduling client through the receptionist of cluster A.

7. The data transmission method according to claim 6, characterized in that: The method further comprises: When the transmission server of the target cluster B receives the data sent by the transmission client of cluster A, it creates a second scheduling client on the target cluster B, and the second scheduling client sends a second data transmission task to the receptionist of the target cluster B; The receptionist of the target cluster B receives the second data transmission task sent by the second scheduling client, and forwards it to the scheduler of the target cluster B; The scheduler of the target cluster B performs task verification and priority scheduling on the received second data transmission task, registers the occupied bandwidth for the second data transmission task, and then sends the second data transmission task to the transmission client of the target cluster B; The transmission client of target cluster B establishes a connection with the transmission server of target cluster C, and transmits data to the transmission server of target cluster C according to the bandwidth size registered by the scheduler; the transmission client of target cluster B pushes the transmission progress to the second scheduling client through the receptionist of cluster B in real time; When the transmission is completed, the transmission client of target cluster B receives the verification result returned by the transmission server of target cluster C; After the transmission is completed, the transmission client of the target cluster B returns the task result to the scheduler of the target cluster B. After the scheduler of the target cluster B releases the bandwidth occupied by the second data transmission task, it returns the final result of the second data transmission task to the second scheduling client through the receptionist of the target cluster B.

Images