High-performance computing cloud platform parallel data transmission method and system and medium

A data transmission method and high-performance computing technology, applied in the field of high-performance computing, can solve problems such as low transmission efficiency, achieve the effects of solving transmission efficiency, avoiding overall retransmission, and solving serial inefficiency

Pending Publication Date: 2022-07-29
深圳北鲲云计算有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0010] The main purpose of the present invention is to provide a high-performance computing cloud platform parallel data transmission m

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • High-performance computing cloud platform parallel data transmission method and system and medium
  • High-performance computing cloud platform parallel data transmission method and system and medium
  • High-performance computing cloud platform parallel data transmission method and system and medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0040] It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

[0041] refer to figure 1 , the present invention proposes a high-performance computing cloud platform parallel data transmission method, the method includes the following steps:

[0042] Step S1, encoding the file to be transmitted at the sender to obtain the encoded data block;

[0043] Step S2, fragment and queue each encoded data block to form a parallel transmission queue;

[0044] Step S3, compressing and encrypting the data of each transmission queue;

[0045] Step S4, using multiple transmission channels to upload data of multiple transmission queues to the recipient in parallel.

[0046] Wherein, the step of encoding the file to be transmitted at the sender to obtain the encoded data block includes:

[0047] Select the list of files to be transferred on the sender;

[0048] Encode the file to be tra...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a parallel data transmission method and system for a high-performance computing cloud platform and a medium, and the method comprises the steps: carrying out the End i ng coding of a to-be-transmitted file, generating a plurality of data blocks with fixed sizes according to a preset BlockS i ze parameter, carrying out the fragmentation and queuing of the data blocks, forming a parallel transmission queue, and carrying out the data transmission of the parallel transmission queue. The method comprises the following steps of: compressing and encrypting data of each queue item, then uploading the data of a plurality of queues in parallel by using a plurality of transmission channels, receiving the data of the transmission queues in parallel at a receiving side, decrypting and decompressing to obtain an original data block, and finally, directly writing file contents according to header information in the data block. According to the invention, the problem of low efficiency of serial transmission is solved, and the serial transmission efficiency of massive small files during transmission is improved; in the case of transmission failure, due to parallel multi-data-block transmission, only the failed data blocks need to be uploaded again, the whole file does not need to be uploaded again, and the overall retransmission requirement is avoided.

Description

technical field [0001] The invention relates to the technical field of high-performance computing, and in particular, to a method, system and medium for parallel data transmission on a high-performance computing cloud platform. Background technique [0002] High-performance computing HPC will use a large number of files for computing input and computing results. The input file needs to be uploaded from the local to the high-performance computing platform, and the result file needs to be downloaded from the platform to the user's local. Therefore, file transfer is very important for high-performance computing. The computing platform is crucial; current high-performance computing platforms generally use the following methods for data transmission: [0003] 1. Transmission methods based on the http protocol, such as web pages, clients, etc.; [0004] 2. Use scp, rsync and other transmission methods based on Linux command line; [0005] 3. Use some standard transmission tools ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): H04L67/06H04L9/40G06F16/174G06F3/06
CPCH04L67/06H04L63/0428G06F16/1744G06F3/064G06F3/0643
Inventor 冯建新李青松
Owner 深圳北鲲云计算有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap