A cross-node asynchronous data transmission method based on message system

Abstract

The present application provides a cross-node asynchronous data transmission method based on a message system for use in a high-performance computing environment, comprising a first proxy server receiving a file request sent by a client, sending the file request to a message system, the message system storing the file request and the data to be transmitted, and a second proxy server pulling the file request from a logical container of the message system and establishing a file transmission channel with a target HPC server based on the client information. The data transmission system of the present invention can expand the number of HPC servers by simply adding front-end nodes, back-end nodes, and logical containers in the message system, thereby achieving low coupling and high scalability of the system, while improving the reliability and fault tolerance of data transmission. If a node fails during the transmission process, the system can quickly update and recover the node to ensure a stable transmission process.

Description

Technical Field

[0001] One or more embodiments of the present disclosure relate to the field of network technology, and more particularly, to a cross-node asynchronous data transmission method based on a message system. Background Art

[0002] Currently, files can be transferred between clients and HPC servers using traditional file transfer protocols such as FTP, SCP, SFTP, and HTTP. While these protocols and the file transfer systems built on them offer certain advantages for point-to-point data transfer, in high-performance computing service clusters, where users cannot directly access the HPC servers and data must be transferred across nodes, these protocols suffer from numerous disadvantages. First, transfer efficiency is low and path complexity is high. Manual configuration and transfer operations between multiple nodes are cumbersome, increasing system coupling and resulting in poor scalability. Second, point-to-point transfers lack reliability and fault tolerance, making effective recovery difficult if a failure occurs during the transfer process. Alternatively, files can be transferred between clients and HPC servers using GridFTP or the Bulk Data Copy Program. GridFTP relies on a grid computing environment and related software packages, such as the Globus toolkit, which complicates system configuration and maintenance. Furthermore, as GridFTP is based on the FTP protocol, it inherits some of FTP's inherent drawbacks, such as firewall and NAT incompatibilities. The Bulk Data Copy Program (BBCP) requires the deployment of executable files on the HPC server, which contradicts the design goal of avoiding the deployment of additional software on the HPC server.

[0003] Application Contents

[0004] This application describes a cross-node asynchronous data transmission method based on a message system, which can solve the above technical problems.

[0005] According to a first aspect, a cross-node asynchronous data transmission method based on a message system is provided, for use in a high-performance computing environment, the method comprising:

[0006] At least one first proxy server receives a file application request sent by a client, and sends the file application request to a message system, wherein the file application request includes a file upload request and / or a file download request and carries login information of the client and information of a target HPC server;

[0007] The message system includes at least one logical container, receives the file application request, and stores the file application request in the logical container;

[0008] At least one second proxy server pulls the file application request from the logical container of the message system, and establishes a file transfer channel with the target HPC server according to the file application request, wherein the file transfer channel is used to transfer the data to be transferred between the second proxy server and the target HPC server;

[0009] The first proxy server forwards data to be transmitted between the client and the message system;

[0010] The message system receives the data to be transmitted and stores the data to be transmitted in a logical container;

[0011] The second proxy server forwards the data to be transmitted between the message system and the target HPC server.

[0012] In some embodiments, the method further comprises:

[0013] When the first proxy server receives a file download request sent by the client, it starts a first consumer thread;

[0014] The first consumer thread is used to wait for pulling data blocks of downloaded files and metadata of downloaded files from the logical container of the message system;

[0015] The metadata of the downloaded file is obtained by the second proxy server according to information of the downloaded file, and is used by the client to assemble data blocks of the downloaded file according to the metadata of the downloaded file to obtain the downloaded file;

[0016] The first proxy server pulls the data blocks of the downloaded file and the metadata of the downloaded file from the logical container of the message system through the first consumer thread, and sends the data blocks of the downloaded file and the metadata of the downloaded file to the client.

[0017] In some embodiments, the method further comprises:

[0018] When the second proxy server pulls the file upload request from the logical container of the message system, it starts a second consumer thread;

[0019] The second consumer thread is configured to wait for pulling data blocks of an uploaded file from a logical container of the message system, wherein the file upload request carries metadata of the uploaded file, and the metadata of the uploaded file is obtained by the client according to information of the uploaded file, and is used by the second proxy server to transmit the data blocks of the uploaded file to the HPC server according to the metadata of the uploaded file;

[0020] The second proxy server pulls the data blocks of the uploaded file from the logical container in the message system through the second consumer thread, and transmits the data blocks of the uploaded file to the target HPC server through the file transmission channel.

[0021] In some embodiments, the method further comprises:

[0022] The first proxy server receives a file upload end request sent by the client;

[0023] The file upload end request is used to identify all data blocks of the uploaded file sent by the client, carries the first verification code of the uploaded file, and sends the file upload end request to the message system;

[0024] The second proxy server pulls the file upload completion request from the logical container of the message system, obtains a second verification code of the uploaded data based on all data blocks of the uploaded file forwarded to the target HPC server, and determines whether the uploaded file has been successfully uploaded based on the first verification code and the second verification code.

[0025] In some embodiments, the method further comprises:

[0026] The client slices the uploaded file to obtain data blocks of the uploaded file, saves the slice information of the uploaded file into the metadata of the uploaded file, and assembles the metadata of the uploaded data into the file upload request;

[0027] The client assembles data blocks of the downloaded file according to metadata of the downloaded file to obtain the downloaded file.

[0028] In some more specific embodiments, the method further comprises:

[0029] The second proxy server obtains a third verification code based on all data blocks of the downloaded file downloaded from the target HPC server, and sends the third verification code to the message system;

[0030] The first proxy server pulls the third verification code from the logical container of the message system, and forwards the third verification code to the corresponding client;

[0031] The client obtains a fourth verification code of the downloaded file based on all received data blocks of the downloaded file, and determines whether the downloaded file has been successfully downloaded based on the third verification code and the fourth verification code.

[0032] In some embodiments, the method further comprises:

[0033] When the second proxy server pulls the file upload request from the corresponding logical channel in the message system and establishes a file transmission channel with the target HPC server, it establishes a target file directory on the target HPC server according to the metadata in the file upload request, and the target file directory is used to store the data blocks of the uploaded file.

[0034] When the second proxy server pulls the file download request from the corresponding logical channel in the message system and establishes a file transmission channel with the target HPC server, it obtains the information of the downloaded file from the HPC server, assembles the metadata of the downloaded file using the information of the downloaded file, and sends the metadata of the downloaded file to the message system.

[0035] In some embodiments, the method further comprises:

[0036] The first proxy server sends the file application request to the logical container of the message system corresponding to the information of the target HPC server, and sends the data to be transmitted to the logical container of the message system corresponding to the information of the target HPC server.

[0037] In some embodiments, the method further comprises:

[0038] The second proxy server sends the data to be transmitted to a target HPC server corresponding to the logical container of the message system.

[0039] According to a second aspect, a cross-node asynchronous data transmission system based on a message system is provided. It is used in a high-performance computing environment and is characterized in that the cross-node asynchronous data transmission system includes:

[0040] At least one first proxy server, configured to receive a file request sent by a client and send the file request to a message system, wherein the file request includes a file upload request and / or a file download request and carries client login information and information about a target HPC server; the first proxy server is further configured to forward data to be transmitted between the client and the message system;

[0041] The message system is configured to receive the file application request and store the file application request and the data to be transmitted in a logical container stored in the message system;

[0042] At least one second proxy server is used to pull the file application request from the logical container of the message system, and establish a file transfer channel with the target HPC server based on the file application request, wherein the file transfer channel is used to transmit the data to be transmitted between the second proxy server and the target HPC server; the second proxy server is also used to forward the data to be transmitted between the message system and the target HPC server.

[0043] In the above-mentioned system and method provided in the embodiments of this specification, the number of HPC servers can be expanded by simply adding front-end and back-end nodes and corresponding logical container Topics, thereby achieving low coupling and high scalability of the system, while improving the reliability and fault tolerance of data transmission. If a node fails during the transmission process, the system can quickly update and restore the node to ensure the stability of the transmission process. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0045] Figure 1 A schematic diagram of a system in a cross-node asynchronous data transmission method based on a message system provided in an embodiment of this specification is shown;

[0046] Figure 2 A schematic diagram showing the process of uploading files in the cross-node asynchronous data transmission method based on the message system provided by the embodiment of this specification is shown;

[0047] Figure 3 A schematic diagram showing the process of downloading files in the cross-node asynchronous data transmission method based on the message system provided by an embodiment of this specification;

[0048] Figure 4 A schematic diagram of the architecture of a cross-node asynchronous data transmission system based on a message system provided in an embodiment of this specification is shown;

[0049] Figure 5 A flow chart of a cross-node asynchronous data transmission method based on a message system provided in an embodiment of this specification is shown. DETAILED DESCRIPTION

[0050] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0052] In the description of the embodiments of the present application, words such as "exemplary," "for example," or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary," "for example," or "for example" in the embodiments of the present application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "for example," or "for example" is intended to present the relevant concepts in a concrete manner.

[0053] In the description of the embodiments of this application, the term "and / or" is merely a description of an association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can represent the following three situations: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "plurality" means two or more.

[0054] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly identifying the technical features being referred to. Thus, features specified as "first" or "second" may explicitly or implicitly include one or more of such features. The terms "include," "comprising," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0055] It's important to note that in high-performance computing (HPC) environments, file transfers between clients and HPC servers can be performed using traditional file transfer protocols such as FTP, SCP, SFTP, and HTTP. However, when multiple service nodes, such as proxy servers, firewalls, load balancers, and other middleware, exist between the client and the HPC server, understanding the network structure and configuring these service nodes accordingly are essential to ensure smooth file transfers. Furthermore, failures during transfer can be difficult to quickly recover from.

[0056] To address these issues, the GridFTP protocol can be used. It supports enhanced features such as parallel data transmission and data striping, making it suitable for large-scale data transmission over wide area networks. However, GridFTP relies on a grid computing environment and related software packages, such as the Globus toolkit, which increases the complexity of system configuration and maintenance. Furthermore, GridFTP is based on the FTP protocol and inherits some of FTP's inherent shortcomings, such as its incompatibility with firewalls and NAT.

[0057] In high-performance computing environments, BBCP (Bulk Data Copy Program) can also be used as a file transfer tool. BBCP achieves higher transfer speeds than traditional file transfer tools through multi-threaded transfers, breakpoint resume, and direct I / O technology. However, using BBCP requires deploying executable files on the HPC server, which contradicts the design goal of avoiding the deployment of additional software on the HPC server.

[0058] Messaging systems, as asynchronous communication protocols, and particularly their publish / subscribe model, have gained widespread adoption due to their low coupling and high scalability. However, due to limitations on message size, these publish / subscribe-based messaging systems are currently rarely used for large file transfers. Therefore, how to utilize messaging systems to transfer large files in a way that meets the requirements for efficient and reliable file data transfer in high-performance computing environments remains a technical challenge.

[0059] like Figure 1 As shown, in this application, a message system is used as the middleware for file data transmission. The data transmission system includes: a client, a front-end proxy server FrontendProxy, a message system MessageQueue, a back-end processing server BackendProcessor and an HPC server.

[0060] The client obtains the user's upload file request and download file request, determines the corresponding front-end proxy server according to the target HPC server in the upload file request and download file request, and sends the upload file request and download file request to the corresponding front-end proxy server.

[0061] The client also sends the uploaded files to the front-end proxy server and receives the downloaded files sent by the front-end proxy server.

[0062] The front-end proxy server has two services enabled. One service is used to receive the data blocks of uploaded files sent by the client and send the data blocks of uploaded files to the logical container Topic specified by the message system. The other service is used to pull data from the logical container Topic specified by the message system, including the data blocks and metadata of downloaded files.

[0063] The front-end proxy server can determine the corresponding logical container in the message system according to the configuration file and the information of the target HPC server.

[0064] The message system is used to receive and store uploaded and downloaded data. There are multiple logical container topics in the message system. The topics of different logical containers are connected to different backend processing servers.

[0065] Specifically, the logical container of the message system can be a message queue, middleware, and topic, etc. The message queue is a data structure that temporarily stores messages so that consumers can extract messages from the logical container for processing.

[0066] The backend processing server manages multiple HPC servers and determines a target HPC server based on information about the target HPC server.

[0067] The backend processing server also enables two services. One service can read the downloaded files in blocks from the target HPC server through the SFTP protocol and send the read data blocks of the downloaded files to the topic of the logical channel specified by the message system. The other service pulls the data blocks of the uploaded files from the topic specified by the message system and writes the data blocks of the uploaded files obtained from the message system to the target HPC server through the SFTP protocol.

[0068] The SFTP service is enabled on the HPC server. Backend processing servers can use the SFTP service to read and write local files on the HPC server.

[0069] When the client needs to upload a file to the target HPC server, it sends a file upload request to the corresponding front-end proxy server, and the request carries the metadata of the uploaded file.

[0070] In this example, metadata uses the FileMetadata structure to store file metadata. Each field has its own specific purpose. The following is a detailed explanation of each field:

[0071] Fid string: The ID of the operation file, usually a randomly generated UUID, used to uniquely identify the file

[0072] Filesize int64: the size of the file in bytes

[0073] Filename string: the name of the file

[0074] SliceNumint: The number of slices into which the file is divided

[0075] RemoteDir string: remote path where uploaded files are saved

[0076] ModifyTimetime.Time: The last modification time of the file

[0077] User string: User name

[0078] RemoteIP string: IP address of the target HPC server

[0079] The client establishes a WebSocket connection with the front-end proxy server, which is used to monitor the transmission progress.

[0080] The client reads the uploaded file in chunks and uploads them in parallel to the front-end proxy server via HTTP. The client then calculates the MD5 hash value of the uploaded file. After all chunks have been transferred, the client sends a notification indicating that all chunks have been uploaded. The notification includes the previously calculated MD5 hash value. MD5 (Message-Digest Algorithm 5) is a commonly used hashing algorithm that converts data of any length into a fixed-length string (usually 128 bits, or 32 hexadecimal characters).

[0081] The front-end proxy server receives the client's file upload request and forwards the metadata in the file upload request and the data blocks of the uploaded file received later to the specified Topic in the message system.

[0082] The backend processing server starts multiple consumer threads, which form a consumer group (ConsumerGroup). The consumer group receives data blocks of uploaded files from the Topic multi-thread specified in the message system.

[0083] The consumer group of the backend processing server receives the file upload request. The backend processing server uses the SSH protocol to connect to the HPC server, sends an SFTP upload command to the HPC server, and creates a new target file directory based on the file name and target path in the metadata of the uploaded file.

[0084] The backend processing server pulls the data blocks of the uploaded file from the specified Topic of the message system and temporarily stores the data blocks of the uploaded file in the memory buffer of the backend processing server. If the memory buffer is full, the data blocks of the uploaded file are temporarily stored in the hard disk of the backend processing server.

[0085] At the same time, the back-end processing server extracts the data blocks of the uploaded file from the memory buffer or hard disk in sequence, writes the data blocks of the uploaded file to the target file directory of the HPC server, calculates the MD5 of the data blocks written to the uploaded file, and deletes the data blocks of the uploaded file from the memory buffer and hard disk.

[0086] The backend processing server pulls the file upload completion notification from the specified topic in the messaging system, sends all data blocks of the uploaded file to the HPC server, and then recalculates the MD5 hash value of all data blocks of the uploaded file sent to the HPC server. The MD5 hash value in the file upload completion notification is compared with the calculated MD5 hash value to see if they are consistent. If they are consistent, the file upload is successful.

[0087] The backend processing server also sets up a 4G memory buffer to temporarily store the received data.

[0088] During the processing, the backend processing server calculates and receives the progress of the data blocks of the uploaded file, sends the progress and error information to the specified Topic in the message system, and the front-end proxy server pulls the progress and error information and forwards it to the client through WebSocket.

[0089] like Figure 3 It shows that the client needs to download the file from the target HPC server.

[0090] The client sends a download request to the front-end proxy server, with the request parameters including the target file, target HPC, etc., and then waits for the front-end proxy server to return the metadata of the target file.

[0091] After receiving the download request, the front-end proxy server forwards the download request to the specified Topic in the message system, and then enables a group of consumer threads (ConsumerGroup) to wait for receiving the metadata and data blocks of the target file.

[0092] After receiving a download request from the specified Topic in the messaging system, the backend proxy server establishes an SSH connection with the HPC, obtains the target file's metadata, and sends it to the specified Topic in the messaging system. It then sends an SFTP download command to the HPC, reads the requested file in chunks via SFTP, and sends the read data chunks in parallel to the specified Topic in the messaging system, calculating the MD5 of the file. After all data chunks have been sent, the calculated MD5 is sent to the specified Topic in the messaging system.

[0093] The consumer group on the front-end proxy server receives file metadata, data blocks, and MD5 from the specified topic multi-threaded in the messaging system. The metadata is returned to the client, the data blocks are stored in the memory buffer or on disk, and the MD5 is saved to a local file.

[0094] After the client receives the target file metadata from the front-end proxy server, it concurrently requests data blocks from the front-end proxy server based on the information in the metadata, writes the data blocks into a new file, and calculates the MD5.

[0095] After receiving the data block request from the client, the front-end proxy server immediately takes the target data block out of the memory buffer or hard disk, ensuring that the client deletes the data block from the memory buffer or hard disk after receiving the data block.

[0096] After receiving all the data blocks, the client requests the MD5 from the front-end proxy server and verifies whether the MD5 is consistent with the MD5 calculated by itself. If the MD5 is consistent, the file download is successful; otherwise, an error is reported.

[0097] The specific steps are as follows:

[0098] 1. The client sends a download request to the front-end proxy server FS1. The request parameters include the target file, target HPC, etc., and then waits for the front-end proxy server FS1 to return the metadata of the target file.

[0099] 2. After receiving the download request, the front-end proxy server FS1 sends the download request to the FS1Topic of the message system. It then enables a group of consumer threads (ConsumerGroup) to listen to the FS1Topic and wait to receive the metadata and data blocks of the target file.

[0100] 3. After receiving the download request from the FS1Topic of the message system, the backend processing server BS1 establishes an SSH connection with the target HPC, sets metadata for the target file, generates FID and ModifyTime, calculates the file size Filesize, calculates the number of slices SliceNum based on the slice size SliceSize, sets Filename, and then sends the file metadata to the FS1Topic of the message system.

[0101] 4. Backend processing server BS1 sends an SFTP download command to the HPC. It then reads the requested file in chunks via SFTP, reading index*SliceSize (index initialized to 0) each time. It then sends the read data chunks in parallel to the FS1 topic in the messaging system, while calculating the file's MD5. After all chunks have been sent, the calculated MD5 is sent to the FS1 topic in the messaging system.

[0102] 5. The front-end proxy server FS1 sets a 4G memory buffer to temporarily store the received data.

[0103] 6. The consumer group enabled on the front-end proxy server FS1 receives file metadata, data blocks, and MD5 from the FS1Topic multi-threaded system of the messaging system, then returns the metadata to the client, stores the data blocks in the memory buffer or on the hard disk, and saves the MD5 to a local file.

[0104] 7. After the client receives the target file metadata returned by the front-end proxy server FS1, it concurrently requests data blocks from the front-end proxy server FS1 based on the information in the metadata, writes the data blocks into a new file, and calculates the MD5.

[0105] 8. After receiving the data block request from the client, the front-end proxy server FS1 immediately takes the target data block out of the memory buffer or hard disk, ensuring that the client deletes the data block from the memory buffer or hard disk after receiving the data block.

[0106] After receiving all the data blocks, the client requests the MD5 from the front-end proxy server FS1. After receiving the MD5, it verifies whether it is consistent with the MD5 calculated by itself. If the MD5 is consistent, it means that the file has been downloaded successfully; otherwise, an error is reported.

[0107] like Figure 2 As shown, the client needs to upload files to the target HPC server.

[0108] The client first sends a start request to inform the front-end proxy server that it is about to upload a file. The request content contains the file's metadata.

[0109] The client then establishes a WebSocket connection with the front-end proxy server to monitor the transmission progress.

[0110] Next, the client reads the file to be uploaded in blocks and uploads the data blocks in parallel to the front-end proxy server via HTTP, while calculating the MD5 of the file. After all data blocks are transferred, the client sends a completion notification to inform the front-end proxy server that all data blocks have been uploaded. The content of the notification contains the MD5 calculated previously.

[0111] After receiving the client's start request, the front-end proxy server forwards the file metadata in the request content and the subsequent data blocks to the specified Topic in the message system. After receiving the client's end notification, it also forwards the MD5 value in the end notification to the specified Topic in the message system.

[0112] The backend processing server enables multiple consumer threads, which form a consumer group (ConsumerGroup). The consumer group receives data from the Topic multi-thread specified in the message system.

[0113] The backend processing server also sets up a 4G memory buffer to temporarily store the received data.

[0114] After the consumer group of the backend processing server receives the start request, the backend processing server uses the SSH protocol to connect to the HPC server. After the connection is established, it sends an SFTP upload command to the HPC server and creates a new target file based on the file name and target path in the file metadata.

[0115] The backend processing server then continues to receive data blocks from the file and temporarily stores them in the memory buffer. If the memory buffer is full, the data blocks are temporarily stored in the hard disk's file system. Simultaneously, the backend processing server sequentially extracts data blocks from the memory buffer or hard disk, writes them to the target file, calculates the MD5 of the data blocks, and deletes them from the memory buffer or hard disk to save space.

[0116] After receiving the end notification and processing all data blocks, the backend processing server compares the MD5 calculated by itself with the MD5 in the end notification. If the two MD5s are consistent, it means that the file upload is successful, otherwise an error is reported.

[0117] During the entire process of receiving and processing data, the backend processing server will calculate the processing progress and send the progress and error information to the specified Topic in the message system. The front-end proxy server will inform the client of the received progress and error information through WebSocket.

[0118] The specific steps are as follows:

[0119] 1. The client sets metadata for the file to be uploaded, generates FID and ModifyTime, calculates the file size Filesize, calculates the number of slices SliceNum based on the slice size SliceSize in the system configuration, sets Filename, RemoteDir, User, and RemoteIP, and sends a start request startRequest to the front-end proxy server FS1. The request content contains the file metadata.

[0120] 2. The client establishes a WebSocket connection with the front-end proxy server FS1.

[0121] 3. The client reads the file in blocks, each time reading index*SliceSize (index is initialized to 0), generates file slices and uploads them to the front-end proxy server FS1 in parallel, while calculating MD5.

[0122] 4. After uploading all file segments, the client sends a finishNotice to the front-end proxy server FS1, and the notification content includes MD5.

[0123] 5. After receiving the startRequest from the client, the front-end proxy server FS1 forwards the file metadata in the request content and the subsequent data blocks to the specified Topic in the message system: FS1Topic.

[0124] 6. The front-end proxy server FS1 notifies the client of the progress and errors in the subsequent transmission process through WebSocket.

[0125] 7. After receiving the finishNotice from the client, the front-end proxy server FS1 forwards the MD5 value to the FS1Topic of the message system.

[0126] 8. The backend processing server BS1 enables multiple consumer threads, which form a consumer group (ConsumerGroup). The consumer group receives data from the FS1Topic multi-thread of the message system.

[0127] 9. The backend processing server BS1 also sets a 4G memory buffer for temporarily storing the received data.

[0128] 10. After the consumer group receives the startRequest, the backend processing server BS1 uses the SSH protocol to connect to the target HPC server. After the connection is established, it sends an SFTP upload command to the HPC server and creates a new target file based on the Filename and RemoteDir in the file metadata.

[0129] 11. The back-end processing server BS1 continuously receives data blocks of the file and temporarily stores the data blocks in the memory buffer. If the memory buffer is full, the data blocks are temporarily stored in the file system of the hard disk.

[0130] 12. Simultaneously, the backend processing server BS1 sequentially extracts data blocks from the memory buffer or hard disk, writes the data blocks to the HPC via SFTP, and calculates the MD5 of the data blocks. It then deletes the data blocks from the memory buffer or hard disk to save space.

[0131] 13. After receiving the finishNotice and processing all data blocks, the backend processing server BS1 compares the MD5 calculated by itself with the MD5 in the finishNotice. If the two MD5s are consistent, it means that the file has been uploaded successfully. Otherwise, an error is reported.

[0132] 14. During the entire process of receiving and processing data, the backend processing server BS1 calculates the processing progress and sends progress and error information to the FS1 Topic of the messaging system.

[0133] Figure 5 The following is a flow chart of a cross-node asynchronous data transmission method based on a message system provided in an embodiment of this specification. The method includes:

[0134] S1. At least one first proxy server receives a file application request sent by a client and sends the file application request to a message system. The file application request includes a file upload request and / or a file download request and carries the client's login information and information of a target HPC server.

[0135] S2. The message system includes at least one logical container, receives a file application request, and stores the file application request in the logical container.

[0136] S3. At least one second proxy server pulls a file application request from the logical container of the message system, and establishes a file transfer channel with the target HPC server according to the file application request. The file transfer channel is used to transfer the data to be transferred between the second proxy server and the target HPC server.

[0137] S4. The first proxy server forwards the data to be transmitted between the client and the message system.

[0138] S5. The message system receives the data to be transmitted and stores the data to be transmitted in a logical container.

[0139] S6. The second proxy server forwards the data to be transmitted between the message system and the target HPC server.

[0140] In some embodiments, upon receiving a file download request sent by a client, the first proxy server starts a first consumer thread.

[0141] The first consumer thread waits to pull the data blocks of the downloaded file and the metadata of the downloaded file from the logical container of the message system.

[0142] The metadata of the downloaded file is obtained by the second proxy server according to the information of the downloaded file, and is used by the client to assemble data blocks of the downloaded file according to the metadata of the downloaded file to obtain the downloaded file.

[0143] The first proxy server is further configured to pull the data blocks of the downloaded file and the metadata of the downloaded file from the logical container of the message system through the first consumer thread, and send the data blocks of the downloaded file and the metadata of the downloaded file to the client.

[0144] In some embodiments, when the second proxy server pulls the file upload request from the logical container of the messaging system, it starts the second consumer thread.

[0145] The second consumer thread is used to wait for the data blocks of the uploaded file to be pulled from the logical container of the message system. The file upload request carries the metadata of the uploaded file. The metadata of the uploaded file is obtained by the client based on the information of the uploaded file, and is used by the second proxy server to transmit the data blocks of the uploaded file to the HPC server based on the metadata of the uploaded file.

[0146] The second proxy server pulls the data blocks of the uploaded file from the logical container of the message system through the second consumer thread, and transmits the data blocks of the uploaded file to the target HPC server through the file transmission channel.

[0147] In some embodiments, it further includes:

[0148] The first proxy server receives the file upload completion request sent by the client.

[0149] The file upload end request is used to identify that the client has sent all data blocks of the uploaded file, carries the first verification code of the uploaded file, and sends the file upload end request to the message system.

[0150] The second proxy server pulls the file upload completion request from the logical container of the message system, obtains the second verification code of the uploaded data based on all the data blocks of the uploaded file forwarded to the target HPC server, and determines whether the uploaded file has been uploaded successfully based on the first verification code and the second verification code.

[0151] In some embodiments, it further includes:

[0152] The client slices the uploaded file to obtain data blocks of the uploaded file, saves the slice information of the uploaded file in the metadata of the uploaded file, and assembles the metadata of the uploaded data into the file upload request.

[0153] In some embodiments, it further includes:

[0154] The client assembles the data blocks of the downloaded file according to the metadata of the downloaded file to obtain the downloaded file.

[0155] In some embodiments, it further includes:

[0156] The second proxy server obtains a third verification code based on all data blocks of the downloaded file obtained from the target HPC server, and sends the third verification code to the message system.

[0157] The first proxy server pulls the third verification code from the logical container of the message system, and forwards the third verification code to the corresponding client.

[0158] The client obtains a fourth check code of the downloaded file according to all received data blocks of the downloaded file, and determines whether the downloaded file has been successfully downloaded according to the third check code and the fourth check code.

[0159] In some embodiments, it further includes:

[0160] When the second proxy server pulls the file upload request from the corresponding logical channel in the message system and establishes a file transmission channel with the target HPC server, it creates a target file directory on the target HPC server based on the metadata in the file upload request. The target file directory is used to store the data blocks of the uploaded file.

[0161] When the second proxy server pulls the file download request from the corresponding logical channel in the message system and establishes a file transfer channel with the target HPC server, it obtains the information of the downloaded file from the HPC server, assembles the metadata of the downloaded file using the information of the downloaded file, and sends the metadata of the downloaded file to the message system.

[0162] In some embodiments, it further includes:

[0163] The first proxy server sends the file application request to the logical container of the message system corresponding to the information of the target HPC server, and sends the data to be transmitted to the logical container of the message system corresponding to the information of the target HPC server.

[0164] In some embodiments, it further includes:

[0165] The second proxy server sends the data to be transmitted to the target HPC server corresponding to the logical container in the message system.

[0166] When processing data to be transmitted between servers, this application can temporarily store it in the memory buffer. If the memory buffer space is insufficient, it will be saved to the hard disk, which reduces the hard disk I / O time and improves transmission efficiency.

[0167] The concurrent and parallel technologies are used in the process of transmitting data blocks to improve the transmission speed.

[0168] The front-end proxy server and back-end processing server establish connections to designated topics in the messaging system. The back-end processing server can manage multiple HPC nodes. The system can expand the number of HPC servers by simply adding new front-end and back-end nodes and corresponding topics, and updating the configuration file, thus achieving a low-coupling and high-scalability system. Furthermore, this architecture improves data transmission reliability and fault tolerance. If a node fails during transmission, the system can quickly update and restore the node, ensuring a stable transmission process.

[0169] Corresponding to the above method provided by the present invention, Figure 4 A schematic diagram of a cross-node asynchronous data transmission system based on a message system provided by an embodiment of this specification is shown, which is used in a high-performance computing environment, including:

[0170] At least one first proxy server, configured to receive a file request from a client and send the file request to the message system. The file request includes a file upload request and / or a file download request and carries client login information and information about a target HPC server. The first proxy server is further configured to forward data to be transmitted between the client and the message system.

[0171] A message system is used to receive file application requests and store the file application requests and data to be transferred into a logical container in the message system;

[0172] At least one second proxy server is used to pull file application requests from the logical container of the message system, establish a file transfer channel with the target HPC server based on the file application request, and the file transfer channel is used to transmit the data to be transmitted between the second proxy server and the target HPC server; the second proxy server is also used to forward the data to be transmitted between the message system and the target HPC server.

[0173] Specifically, the data to be transmitted includes files to be uploaded and files to be downloaded.

[0174] In some embodiments, the first proxy server is further configured to start the first consumer thread upon receiving a file download request sent by the client;

[0175] The first consumer thread is used to wait for the data blocks and metadata of the downloaded file to be pulled from the logical container of the message system;

[0176] The metadata of the downloaded file is obtained by the second proxy server according to the information of the downloaded file, and is used by the client to assemble the data blocks of the downloaded file according to the metadata of the downloaded file to obtain the downloaded file;

[0177] The first proxy server pulls the data blocks of the downloaded file and the metadata of the downloaded file from the logical container of the message system through the first consumer thread, and sends the data blocks of the downloaded file and the metadata of the downloaded file to the client.

[0178] In some embodiments, the second proxy server is further configured to pull the file upload request from the logical container of the messaging system and start a second consumer thread;

[0179] The second consumer thread is used to wait for the data blocks of the uploaded file to be pulled from the logical container of the message system. The file upload request carries the metadata of the uploaded file. The metadata of the uploaded file is obtained by the client based on the information of the uploaded file. The second proxy server transmits the data blocks of the uploaded file to the HPC server based on the metadata of the uploaded file.

[0180] The second proxy server pulls the data blocks of the uploaded file from the logical container of the message system through the second consumer thread, and transmits the data blocks of the uploaded file to the target HPC server through the file transmission channel.

[0181] In some embodiments, the first proxy server is further configured to receive a file upload end request sent by the client;

[0182] The file upload end request is used to identify that the client has sent all data blocks of the uploaded file, carries the first verification code of the uploaded file, and sends the file upload end request to the message system;

[0183] The second proxy server pulls the file upload completion request from the logical container of the message system, obtains the second verification code of the uploaded data based on all the data blocks of the uploaded file forwarded to the target HPC server, and determines whether the uploaded file has been uploaded successfully based on the first verification code and the second verification code.

[0184] In some embodiments, the client is configured to slice the uploaded file to obtain data blocks of the uploaded file, save the slice information of the uploaded file into metadata of the uploaded file, and assemble the metadata of the uploaded data into a file upload request;

[0185] The client is further used to assemble data blocks of the downloaded file according to the metadata of the downloaded file to obtain the downloaded file.

[0186] In some embodiments, the second proxy server is further configured to obtain a third verification code based on all data blocks of the downloaded file obtained from the target HPC server, and send the third verification code to the message system;

[0187] The first proxy server is further configured to pull a third verification code from the logical container of the message system and forward the third verification code to a corresponding client;

[0188] The client is further configured to obtain a fourth check code of the downloaded file according to all received data blocks of the downloaded file, and determine whether the downloaded file has been successfully downloaded according to the third check code and the fourth check code.

[0189] In some embodiments, the second proxy server is further used to pull a file upload request from a corresponding logical container in the message system, establish a file transfer channel with a target HPC server, and then establish a target file directory on the target HPC server based on the metadata in the file upload request. The target file directory is used to store data blocks of the uploaded file.

[0190] The second proxy server is also used to pull the file download request from the corresponding logical container in the message system, establish a file transmission channel with the target HPC server, obtain the information of the downloaded file from the HPC server, assemble the metadata of the downloaded file using the information of the downloaded file, and send the metadata of the downloaded file to the message system.

[0191] The first proxy server is further configured to send the file application request to the logical container of the message system corresponding to the information of the target HPC server, and send the data to be transmitted to the logical container of the message system corresponding to the information of the target HPC server.

[0192] The second proxy server is further configured to send the data to be transmitted to a target HPC server corresponding to the logical container of the message system.

[0193] The above embodiment allows the number of HPC servers to be expanded by simply adding new front-end and back-end nodes and corresponding topics and updating the configuration files, thereby achieving a low coupling and high scalability of the system. This also improves the reliability and fault tolerance of data transmission. If a node fails during transmission, the system can quickly update and restore the node, ensuring a stable transmission process.

[0194] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described herein may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0195] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this application. It should be understood that the above description is only the specific implementation methods of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of this application should be included in the scope of protection of this application.

Claims

1. A cross-node asynchronous data transmission method based on a message system, used in a high-performance computing environment, characterized in that: The method comprises: At least one first proxy server receives a file application request sent by a client, and sends the file application request to a message system, wherein the file application request includes a file upload request and / or a file download request and carries login information of the client and information of a target HPC server; The message system includes at least one logical container, receives the file application request, and stores the file application request in the logical container; At least one second proxy server pulls the file application request from the logical container of the message system, and establishes a file transfer channel with the target HPC server according to the file application request, wherein the file transfer channel is used to transfer the data to be transferred between the second proxy server and the target HPC server; The first proxy server forwards the data to be transmitted between the client and the message system; The message system receives the data to be transmitted and stores the data to be transmitted in a logical container; The second proxy server forwards the data to be transmitted between the message system and the target HPC server.

2. The cross-node asynchronous data transmission method according to claim 1, characterized in that: The method further comprises: When the first proxy server receives a file download request sent by the client, it starts a first consumer thread; The first consumer thread is used to wait for pulling data blocks of downloaded files and metadata of downloaded files from the logical container of the message system; The metadata of the downloaded file is obtained by the second proxy server according to information of the downloaded file, and is used by the client to assemble data blocks of the downloaded file according to the metadata of the downloaded file to obtain the downloaded file; The first proxy server pulls the data blocks of the downloaded file and the metadata of the downloaded file from the logical container of the message system through the first consumer thread, and sends the data blocks of the downloaded file and the metadata of the downloaded file to the client.

3. The cross-node asynchronous data transmission method according to claim 1, characterized in that: The method further comprises: When the second proxy server pulls the file upload request from the logical container of the message system, it starts a second consumer thread; The second consumer thread is configured to wait for pulling data blocks of an uploaded file from a logical container of the message system, wherein the file upload request carries metadata of the uploaded file, and the metadata of the uploaded file is obtained by the client according to information of the uploaded file, and is used by the second proxy server to transmit the data blocks of the uploaded file to the HPC server according to the metadata of the uploaded file; The second proxy server pulls the data blocks of the uploaded file from the logical container of the message system through the second consumer thread, and transmits the data blocks of the uploaded file to the target HPC server through the file transmission channel.

4. The cross-node asynchronous data transmission method according to claim 1, characterized in that: The method further comprises: The first proxy server receives a file upload end request sent by the client; The file upload end request is used to identify all data blocks of the uploaded file sent by the client, carries the first verification code of the uploaded file, and sends the file upload end request to the message system; The second proxy server pulls the file upload completion request from the logical container of the message system, obtains a second verification code of the uploaded file based on all data blocks of the uploaded file forwarded to the target HPC server, and determines whether the uploaded file has been successfully uploaded based on the first verification code and the second verification code.

5. The cross-node asynchronous data transmission method according to claim 1, characterized in that: The method further comprises: The client slices the uploaded file to obtain data blocks of the uploaded file, saves the slice information of the uploaded file into the metadata of the uploaded file, and assembles the metadata of the uploaded file into the file upload request; The client assembles data blocks of the downloaded file according to metadata of the downloaded file to obtain the downloaded file.

6. The cross-node asynchronous data transmission method according to claim 5, characterized in that: The method further comprises: The second proxy server obtains a third verification code based on all data blocks of the downloaded file downloaded from the target HPC server, and sends the third verification code to the message system; The first proxy server pulls the third verification code from the logical container of the message system, and forwards the third verification code to the corresponding client; The client obtains a fourth verification code of the downloaded file based on all received data blocks of the downloaded file, and determines whether the downloaded file has been successfully downloaded based on the third verification code and the fourth verification code.

7. The cross-node asynchronous data transmission method according to claim 1, characterized in that: The method further comprises: When the second proxy server pulls the file upload request from the corresponding logical channel in the message system and establishes a file transmission channel with the target HPC server, a target file directory is created on the target HPC server according to the metadata in the file upload request, and the target file directory is used to store data blocks of the uploaded file; When the second proxy server pulls the file download request from the corresponding logical channel in the message system and establishes a file transmission channel with the target HPC server, it obtains the information of the downloaded file from the HPC server, assembles the metadata of the downloaded file using the information of the downloaded file, and sends the metadata of the downloaded file to the message system.

8. The cross-node asynchronous data transmission method according to claim 1, wherein: The method further comprises: The first proxy server sends the file application request to the logical container of the message system corresponding to the information of the target HPC server, and sends the data to be transmitted to the logical container of the message system corresponding to the information of the target HPC server.

9. The cross-node asynchronous data transmission method according to claim 1, characterized in that: The method further comprises: The second proxy server sends the data to be transmitted to a target HPC server corresponding to the logical container of the message system.

10. A cross-node asynchronous data transmission system based on a message system, used in a high-performance computing environment, characterized in that: The cross-node asynchronous data transmission system includes: At least one first proxy server, configured to receive a file request sent by a client and send the file request to a message system, wherein the file request includes a file upload request and / or a file download request and carries client login information and information about a target HPC server; the first proxy server is further configured to forward data to be transmitted between the client and the message system; The message system is configured to receive the file application request and store the file application request and the data to be transmitted in a logical container stored in the message system; At least one second proxy server is used to pull the file application request from the logical container of the message system, and establish a file transfer channel with the target HPC server based on the file application request, wherein the file transfer channel is used to transmit the data to be transmitted between the second proxy server and the target HPC server; the second proxy server is also used to forward the data to be transmitted between the message system and the target HPC server.

Images