Data transmission method, device, equipment and storage medium

Abstract

The present disclosure provides a data transmission method, apparatus, device and storage medium, which relate to the field of artificial intelligence technology, specifically cloud computing, cloud storage and online compression technology, and can be applied in intelligent cloud scenarios. The data transmission method includes: using a first processing resource to divide the file to be transmitted into at least one block, compressing the at least one block to obtain at least one compressed block, and storing the at least one compressed block in a first queue; using a second processing resource to read the at least one compressed block from the first queue, obtaining at least one transmission block based on the at least one compressed block, and storing the at least one transmission block in a second queue; using a third processing resource to read the at least one transmission block from the second queue, and transmitting the at least one transmission block to the destination. The present disclosure can improve data transmission efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to cloud computing, cloud storage, and online compression technology, which can be applied in intelligent cloud scenarios, and in particular to a data transmission method, apparatus, device, and storage medium. Background Art

[0002] Users have a need to download files in batches. If the batch of files to be downloaded is too large, the batch files are usually compressed and then downloaded.

[0003] In related technologies, compression and downloading are usually performed in a serial manner, resulting in low data transmission efficiency. Summary of the Invention

[0004] The present disclosure provides a data transmission method, apparatus, device, and storage medium.

[0005] According to one aspect of the present disclosure, a data transmission method is provided, comprising: using a first processing resource to divide a file to be transmitted into at least one block, compressing the at least one block to obtain at least one compressed block, and storing the at least one compressed block in a first queue; using a second processing resource to read the at least one compressed block from the first queue, obtaining at least one transmission block based on the at least one compressed block, and storing the at least one transmission block in a second queue; using a third processing resource to read the at least one transmission block from the second queue, and transmitting the at least one transmission block to a destination; wherein the first processing resource, the second processing resource, and the third processing resource are executed concurrently.

[0006] According to another aspect of the present disclosure, a data transmission device is provided, including: a compression module, used to use a first processing resource to divide a file to be transmitted into at least one block, compress the at least one block to obtain at least one compressed block, and store the at least one compressed block in a first queue; a conversion module, used to use a second processing resource to read the at least one compressed block from the first queue, obtain at least one transmission block based on the at least one compressed block, and store the at least one transmission block in a second queue; a transmission module, used to use a third processing resource to read the at least one transmission block from the second queue, and transmit the at least one transmission block to a destination; wherein the first processing resource, the second processing resource and the third processing resource are executed concurrently.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any one of the methods described in any one of the above aspects.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods according to any one of the above aspects.

[0009] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of the above aspects.

[0010] According to the technical solution disclosed in the present invention, data transmission efficiency can be improved.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0014] Figure 2 is a schematic diagram of an application scenario corresponding to an embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram according to a second embodiment of the present disclosure;

[0016] Figure 4 is a schematic diagram according to a third embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0018] Figure 6 Schematic diagram of an electronic device for implementing the data transmission method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0020] In related technologies, data is usually transmitted in a compression and serial downloading manner, which has a poor data transmission efficiency.

[0021] In order to improve data transmission efficiency, the present disclosure provides the following embodiments.

[0022] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure, which provides a data transmission method, such as Figure 1 As shown, the method includes:

[0023] 101. Use a first processing resource to divide a file to be transmitted into at least one block, perform compression processing on the at least one block to obtain at least one compressed block, and store the at least one compressed block in a first queue.

[0024] 102. Use a second processing resource to read the at least one compressed block from the first queue, obtain at least one transmission block based on the at least one compressed block, and store the at least one transmission block in a second queue.

[0025] 103. Use a third processing resource to read the at least one transmission block from the second queue, and transmit the at least one transmission block to a destination.

[0026] The first processing resource, the second processing resource, and the third processing resource are executed concurrently.

[0027] The above-mentioned processing resources may be groutines provided by the Go language.

[0028] Go (also known as Golang) is a statically strongly typed, compiled language.

[0029] Groutine (or Goroutine) is a lightweight thread that allows concurrent execution in the Go language. It distributes tasks assigned to groutines to processors (such as CPUs) and allows these tasks to execute concurrently. Groutines communicate with each other through channels (chan).

[0030] Compared with other languages, Go can implement concurrency simply and efficiently based on Groutine.

[0031] The file to be transferred may include a single file or batch files, that is, may include one or more files.

[0032] When dividing a file to be transferred, the file to be transferred may be divided into at least one block of the preset size based on a preset size. For example, if the preset size is 10 MB and the file to be transferred is 100 MB, the file to be transferred may be divided into 10 blocks, each of which is 10 MB in size.

[0033] Each block may be compressed. The compression may be performed using a relevant compression algorithm, such as a zip algorithm.

[0034] The compressed blocks may be referred to as compressed blocks, and the size of compressed blocks is usually smaller.

[0035] Taking the processing resource as a coroutine as an example, the coroutines communicate with each other through a pipe (chan), and the data in the pipe can be managed in the form of a queue. That is, after the first coroutine obtains the compressed block, it can store the compressed block in the first queue of the first pipe.

[0036] The second goroutine can read the compressed block from the first queue and obtain the transmission block based on the compressed block.

[0037] The size of each transmission block may be a preset value. For example, the preset size of each transmission block is 20M. Assuming that the compressed block is 5M, four compressed blocks may be combined into one transmission block.

[0038] It is understandable that, depending on different actual situations, if the size of the compressed block is larger than the size of the transmission block, the compressed block may be divided into blocks to obtain a transmission block that meets the requirements.

[0039] After the second coroutine obtains the at least one transmission block, it may store the at least one transmission block in the second queue of the second pipeline.

[0040] The third coroutine may read at least one transmission block from the second queue and transmit the at least one transmission block to the destination end.

[0041] This embodiment can be applied in upload or download scenarios. For upload, the destination end can be the cloud; for download, the destination end can be a user terminal.

[0042] In this embodiment, since the first processing resource, the second processing resource and the third processing resource are executed concurrently, the generation process of the transmission block and the transmission process of the transmission block can be realized concurrently, that is, it can be understood as compression and transmission, or streaming transmission. Since compression and transmission are concurrent, the data transmission efficiency can be improved compared to the serial method.

[0043] In order to better understand the embodiment of the present disclosure, the application scenario of the embodiment of the present disclosure is described. This embodiment takes uploading a file as an example, that is, a user can upload a file on the user terminal to the server.

[0044] Figure 2 The following is a schematic diagram of an application scenario corresponding to an embodiment of the present disclosure. The data transmission system may include: a user terminal 201 and a server 202. The user terminal 201 may be a personal computer (PC), a mobile device (such as a mobile phone or laptop), a smart home appliance (such as a smart TV or smart speaker), or a wearable device (such as a smart watch or smart bracelet). The server 202 may be a local server or a cloud server.

[0045] The user can start the file upload program on the user terminal side. After starting, the program can automatically start three concurrent processing resources (a first processing resource, a second processing resource, and a third processing resource). Taking the program based on the Go language as an example, the above processing resources can be coroutines.

[0046] A first processing resource is used to compress the file in blocks to obtain at least one compressed block;

[0047] The second processing resource is configured to obtain at least one transport block based on the at least one compressed block.

[0048] The third processing resource is configured to upload at least one transmission block to the server.

[0049] Since the processing resources are executed concurrently, the transmission blocks can be generated and uploaded at the same time, without having to wait for all the transmission blocks to be obtained before uploading, which can improve the transmission efficiency.

[0050] In combination with the above application scenarios, the embodiments of the present disclosure also provide the following data transmission method.

[0051] Figure 3 is a schematic diagram according to a second embodiment of the present disclosure. This embodiment provides a data transmission method. In this embodiment, taking a Goroutine as an example, the processing resource includes:

[0052] 301. Use a first coroutine to divide a file to be transmitted into at least one block, compress the at least one block to obtain at least one compressed block, and store the at least one compressed block in a first queue.

[0053] 302. Use a second coroutine to read the at least one compressed block from the first queue, merge the at least one compressed block to obtain at least one transmission block, and store the at least one transmission block in a second queue, wherein each transmission block includes at least one compressed block.

[0054] 303. Use a third coroutine to read the at least one transmission block from the second queue and transmit the at least one transmission block to the destination end.

[0055] Among them, the first coroutine, the second coroutine and the third coroutine are executed concurrently.

[0056] Groutine (or Goroutine) is a lightweight thread that allows concurrent execution in the Go language. It distributes tasks assigned to groutines to processors (such as CPUs) and allows these tasks to execute concurrently. Groutines communicate with each other through channels (chan).

[0057] Combine Figure 4 The specific process shown uses a user uploading a file as an example. After the user starts the file upload process, they can first initialize relevant information, such as configuring the user's regional information (depending on actual needs, it can be set to only allow users in certain regions to upload files this way) and configuring the file information to be uploaded, such as the file's uniform resource locator (URL). There can be one or more files to be uploaded, and the URLs of one or more files can form a URL list.

[0058] The file upload program is implemented based on the Go language. After the program is started, it can automatically start the first coroutine, the second coroutine, and the third coroutine for concurrent execution based on pre-written code.

[0059] After starting, the first coroutine can traverse the URL list in sequence. If there is an untraversed URL, it calls the untraversed URL to obtain the corresponding file. After obtaining the corresponding file, it can create compression-related information (such as compressed file header information) and compress the file in blocks to obtain compressed file blocks. The compressed file blocks can be called compressed blocks. There are one or more compressed blocks. Due to the large size of the file, there are usually multiple compressed blocks. The above traversal-block compression process can be repeated multiple times until the files corresponding to all URLs are compressed in blocks.

[0060] By obtaining a file based on file information (such as a URL), the user can specify the file to be transferred, thereby achieving reasonable and efficient file acquisition.

[0061] After the first coroutine obtains the compressed block, it stores it in a first queue. The first queue can be located in a first pipe (chan), which is a pipe for the first coroutine and the second coroutine to communicate with each other.

[0062] After the second coroutine is started, it can read compressed blocks from the first queue, merge the read compressed blocks, and determine whether the size of the merged compressed blocks reaches the preset transmission block size. If so, the merged compressed blocks are used as transmission blocks. Otherwise, the compressed blocks can continue to be merged until the set size is reached.

[0063] By merging compressed blocks into transmission blocks, transmission requirements can be met and transmission efficiency can be improved.

[0064] After the second coroutine obtains the transmission block, it can store it in a second queue. The second queue can be located in a second pipe (chan). The second pipe is a pipe for the second coroutine and the third coroutine to communicate with each other.

[0065] After the third coroutine is started, it can read the transmission block from the second queue and upload the transmission to the server.

[0066] The third coroutine can call the block upload interface of the Baidu Object Storage (BOS) service to upload the transmission block. In addition, after the upload is successful, the identification information of the successful upload can be saved.

[0067] In addition, after the file upload program calls the BOS interface to complete the upload, it can close all files (or file streams) and then end.

[0068] In addition, when an error occurs during the execution of the first coroutine, the second coroutine, and the third coroutine, the error information can be stored in the error queue, and the file upload program can also encapsulate the error information.

[0069] For the first coroutine, the block compression task is performed. Generally speaking, the block compression is performed serially, that is, each compressed block is generated serially.

[0070] During compression, a subsequent compressed block may require information from a previous compressed block, so the accuracy of the data can be guaranteed by serially generating compressed blocks.

[0071] For the second coroutine, a merging task is executed. The merging task can be executed in parallel, that is, the at least one compressed block can be merged in a parallel manner to obtain at least one transmission block.

[0072] For example, if the compressed block includes A, B, C, and D, A and B may be merged into one transmission block, and C and D may be merged into another transmission block in parallel.

[0073] By merging compressed blocks in a parallel manner, the generation efficiency of transmission blocks can be improved, thereby improving data transmission efficiency.

[0074] For the third coroutine, a transmission task is executed. The transmission task may be executed in parallel, that is, the at least one transmission block may be transmitted to the destination end in a parallel manner.

[0075] For example, if the transmission block includes E and F, E and F can be transmitted to the destination in parallel.

[0076] By transmitting the transmission blocks in parallel, the data transmission efficiency can be improved.

[0077] In this embodiment, file compression and transmission can be implemented concurrently based on the Go language. Since compression and transmission are concurrent, the data transmission efficiency can be improved. Due to the advantages of Go such as high efficiency and simplicity, the concurrent execution of compression and transmission can be implemented efficiently and concisely based on the Go language's coroutine concurrency method.

[0078] Figure 5 3 is a schematic diagram according to the third embodiment of the present disclosure. This embodiment provides a data transmission device. The device 500 includes: a compression module 501, a conversion module 502 and a transmission module 503.

[0079] The compression module 501 is used to use a first processing resource to divide the file to be transmitted into at least one block, compress the at least one block to obtain at least one compressed block, and store the at least one compressed block in a first queue; the conversion module 502 is used to use a second processing resource to read the at least one compressed block from the first queue, obtain at least one transmission block based on the at least one compressed block, and store the at least one transmission block in a second queue; the transmission module 503 is used to use a third processing resource to read the at least one transmission block from the second queue, and transmit the at least one transmission block to the destination; wherein, the first processing resource, the second processing resource and the third processing resource are executed concurrently.

[0080] In this embodiment, since the first processing resource, the second processing resource and the third processing resource are executed concurrently, the generation process of the transmission block and the transmission process of the transmission block can be realized concurrently, that is, it can be understood as compression and transmission, or streaming transmission. Since compression and transmission are concurrent, the data transmission efficiency can be improved compared to the serial method.

[0081] In some embodiments, the compression module 501 is further configured to:

[0082] Receiving file information of the file to be transferred input by a user;

[0083] Acquire the file to be transferred based on the file information;

[0084] Divide the file to be transmitted into at least one block.

[0085] In this embodiment, by obtaining a file based on file information (such as a URL), the user can specify the file to be transmitted, thereby achieving reasonable and efficient file acquisition.

[0086] In some embodiments, the conversion module 502 is further configured to:

[0087] The at least one compressed block is merged to obtain at least one transmission block, wherein each transmission block includes at least one compressed block.

[0088] In this embodiment, by merging compressed blocks into transmission blocks, transmission requirements can be met and transmission efficiency can be improved.

[0089] In some embodiments, the conversion module 502 is further configured to:

[0090] The at least one compressed block is merged in a parallel manner.

[0091] In this embodiment, by merging compressed blocks in a parallel manner, the generation efficiency of transmission blocks can be improved, thereby improving the data transmission efficiency.

[0092] In some embodiments, the transmission module 503 is further configured to:

[0093] The at least one transmission block is transmitted to the destination end in a parallel manner.

[0094] By transmitting the transmission blocks in parallel, the data transmission efficiency can be improved.

[0095] In some embodiments, the first processing resource is a first coroutine created based on the Go language;

[0096] The second processing resource is a second coroutine created based on the Go language;

[0097] The third processing resource is a third coroutine created based on the Go language.

[0098] In this embodiment, file compression and transmission can be implemented concurrently based on the Go language. Since compression and transmission are concurrent, the data transmission efficiency can be improved. Due to the advantages of Go such as high efficiency and simplicity, the concurrent execution of compression and transmission can be implemented efficiently and concisely based on the Go language's coroutine concurrency method.

[0099] It can be understood that in the embodiments of the present disclosure, the same or similar contents in different embodiments can be referenced to each other.

[0100] It can be understood that the terms “first”, “second”, etc. in the embodiments of the present disclosure are only used for distinction and do not indicate the degree of importance, time sequence, etc.

[0101] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0102] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0103] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. Electronic device 600 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 600 can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0104] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0105] Multiple components in the electronic device 600 are connected to the I / O interface 605, including an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0106] The computing unit 601 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the data transmission method. For example, in some embodiments, the data transmission method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the data transmission method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the data transmission method in any other appropriate manner (e.g., by means of firmware).

[0107] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0108] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable load balancing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0109] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0110] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0111] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0112] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers and establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within a cloud computing service ecosystem that addresses the management difficulties and limited business scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0113] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0114] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A data transmission method, comprising: Using a first processing resource, dividing a file to be transmitted into a plurality of blocks, compressing the plurality of blocks to obtain a plurality of compressed blocks, and storing the plurality of compressed blocks in a first queue; Using a second processing resource, read the plurality of compressed blocks from the first queue, obtain a plurality of transmission blocks based on the plurality of compressed blocks, and store the plurality of transmission blocks in a second queue; Using a third processing resource, read the multiple transmission blocks from the second queue, and transmit the multiple transmission blocks to a destination end; wherein the first processing resource, the second processing resource, and the third processing resource are executed concurrently; The plurality of compressed blocks obtained by the first processing resource are generated serially; The first processing resource is a first coroutine created based on the Go language; The second processing resource is a second coroutine created based on the Go language; The third processing resource is a third coroutine created based on the Go language; The first queue is located in a first pipeline, and the first pipeline is a pipeline for the first coroutine and the second coroutine to communicate with each other; The second queue is located in a second pipeline, and the second pipeline is a pipeline for the second coroutine and the third coroutine to communicate with each other; The first coroutine, the second coroutine, and the third coroutine are automatically started after the user starts the file transfer program and are executed concurrently; The first coroutine is specifically configured to, after startup, sequentially traverse a list of uniform resource locators (URLs), obtain files corresponding to untraversed URLs as the files to be transferred, and perform traversal and compression multiple times until all files corresponding to the URLs are compressed in blocks; the URL list is obtained through an initialization process after the user starts the file transfer program, and the initialization process includes configuring the user's regional information and file information of the files to be uploaded; The second coroutine is specifically configured to, after being started, read compressed blocks from the first queue, merge the compressed blocks until a transmission block of a preset size is obtained, and perform the merging in parallel; The third coroutine is specifically configured to, after startup, read the transmission block from the second queue, call the block transmission interface of the object storage BOS service, transmit the transmission block through the interface, and save identification information of successful transmission after successful transmission; The first coroutine, the second coroutine and the third coroutine are further configured to store error information in an error queue when an error occurs during execution; The file transfer program is further configured to terminate after closing all files after the interface is called to complete the transfer, or terminate after encapsulating the error message.

2. The method according to claim 1, wherein The step of dividing the file to be transferred into a plurality of blocks includes: Receiving file information of the file to be transferred input by a user; Acquire the file to be transferred based on the file information; Divide the file to be transmitted into multiple blocks.

3. The method according to claim 1, wherein The transmitting the plurality of transport blocks to the destination end includes: The multiple transmission blocks are transmitted to the destination end in a parallel manner.

4. A data transmission device, comprising: a compression module, configured to use a first processing resource to divide a file to be transmitted into a plurality of blocks, perform compression processing on the plurality of blocks to obtain a plurality of compressed blocks, and store the plurality of compressed blocks in a first queue; a conversion module, configured to use a second processing resource to read the plurality of compressed blocks from the first queue, obtain a plurality of transmission blocks based on the plurality of compressed blocks, and store the plurality of transmission blocks in a second queue; a transmission module, configured to read the plurality of transmission blocks from the second queue using a third processing resource, and transmit the plurality of transmission blocks to a destination end; wherein the first processing resource, the second processing resource, and the third processing resource are executed concurrently; The plurality of compressed blocks obtained by the first processing resource are generated serially; The plurality of transport blocks are obtained by merging the plurality of compressed blocks in parallel; The first processing resource is a first coroutine created based on the Go language; The second processing resource is a second coroutine created based on the Go language; The third processing resource is a third coroutine created based on the Go language; The first queue is located in a first pipeline, and the first pipeline is a pipeline for the first coroutine and the second coroutine to communicate with each other; The second queue is located in a second pipeline, and the second pipeline is a pipeline for the second coroutine and the third coroutine to communicate with each other; The first coroutine, the second coroutine, and the third coroutine are automatically started after the user starts the file transfer program and are executed concurrently; The first coroutine is specifically configured to sequentially traverse a URL list after startup, obtain files corresponding to URLs that have not been traversed as the files to be transferred, and perform traversal and compression multiple times until all files corresponding to the URLs are compressed in blocks; the URL list is obtained through an initialization process after the user starts the file transfer program, and the initialization process includes configuring the user's regional information and file information of the files to be uploaded; The second coroutine is specifically configured to, after being started, read compressed blocks from the first queue, merge the compressed blocks until a transmission block of a preset size is obtained, and perform the merging in parallel; The third coroutine is specifically configured to, after startup, read the transmission block from the second queue, call the block transmission interface of the BOS service, transmit the transmission block through the interface, and save identification information of successful transmission after successful transmission; The first coroutine, the second coroutine and the third coroutine are further configured to store error information in an error queue when an error occurs during execution; The file transfer program is further configured to terminate after closing all files after the interface is called to complete the transfer, or terminate after encapsulating the error message.

5. The device according to claim 4, wherein The compression module is further configured to: Receiving file information of the file to be transferred input by a user; Acquire the file to be transferred based on the file information; Divide the file to be transmitted into multiple blocks.

6. The device according to claim 4, wherein The transmission module is further configured to: The multiple transmission blocks are transmitted to the destination end in a parallel manner.

7. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 3.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-3.

9. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 3.

Images