File transmission system, method, device, electronic device and storage medium

Abstract

The application provides a file transmission system, method, device, electronic equipment and storage medium. The system comprises a client and a file server. The file server comprises a storage warehouse for storing files and an access point for data transfer between the client and the storage warehouse. The storage warehouse is configured to store the files in the form of fragmented data. The access point is configured to, in response to a first request of the client for downloading fragmented data of a target file, pull and cache all fragmented data of the target file from the storage warehouse, extract the fragmented data requested by the client for the first time from all the fragmented data of the target file and transmit the data to the client; and in response to a second request of the client for downloading fragmented data of the target file, extract the fragmented data requested by the client for the second time from all the fragmented data of the target file and transmit the data to the client. The application can reduce the time consumed in the file downloading process, thereby improving the file downloading speed.

Description

Technical Field

[0001] This application relates to the field of cloud technology, specifically to a file transfer system, method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the field of cloud technology, file transfer is an essential component. Currently, to facilitate file management, massive amounts of files are often centrally stored in large-capacity storage repositories. However, due to the high cost and scarcity of these storage repositories, many clients are physically located far from them, resulting in slow file transfer speeds and significant time consumption for client-side data transfers. Summary of the Invention

[0003] One objective of this application is to provide a file transfer system, method, apparatus, electronic device, and storage medium that can reduce the time spent during file downloading, thereby increasing file download speed.

[0004] According to one aspect of the embodiments of this application, a file transfer system is disclosed, the system including a client and a file server, the file server including a storage repository for storing files and an access point for transferring data between the client and the storage repository;

[0005] The storage warehouse is configured to store files in a data-sharded manner; The access point is configured as follows: In response to the client's first request to download the fragment data of the target file, all fragment data of the target file are pulled from the storage repository and cached, and the fragment data requested by the client in the first request is extracted from all fragment data of the target file and transmitted to the client; In response to the client's subsequent request to download the fragmented data of the target file, the requested fragmented data is extracted from all fragmented data of the target file and transmitted to the client.

[0006] In one exemplary embodiment of this application, the system further includes a master server, and the client is configured as follows: The system requests the creation of a file from the master server and receives the file server address returned by the master server. A Transmission Control Protocol (TCP) connection is established with the access point according to the file server address, and the fragmented data of the created file is uploaded to the access point through the TCP connection. Under the data relay function of the access point, the fragmented data of the created file is transmitted to the storage warehouse, thus obtaining a storage warehouse storing the created file.

[0007] In an exemplary embodiment of this application, the client is configured as follows: Establish at least two TCP connections with the access point, and upload a fragment of the created file to the access point through each of the at least two TCP connections.

[0008] In an exemplary embodiment of this application, the client is configured as follows: The TCP connection that has completed uploading the assigned fragment data is retained, and another fragment of the created file is uploaded to the access point through the TCP connection that has completed uploading the assigned fragment data.

[0009] In an exemplary embodiment of this application, the master server is configured to: return at least two file server addresses to the client in response to the client's request to create a file; The client is configured as follows: Simultaneously connect to the file servers at the addresses of at least two file servers and send a handshake request to each file server; A TCP connection is established between the file server address of the file server that first responds to the handshake request and the access point of the file server that first responds to the handshake request.

[0010] In an exemplary embodiment of this application, the client is configured as follows: The first thread downloads fragmented data from the access point, transmits the downloaded fragmented data to the second thread, and continues to download fragmented data from the access point. The second thread enqueues the fragmented data downloaded by the first thread into a data queue, and writes the fragmented data in the data queue into the storage area of ​​the terminal where the client is located.

[0011] According to one aspect of the embodiments of this application, a file transfer method is disclosed, the method comprising: In response to the client's first request to download the fragmented data of the target file, all fragmented data of the target file are retrieved from the storage repository and cached, and the fragmented data requested by the client in the first request is extracted from all fragmented data of the target file and transmitted to the client; In response to the client's subsequent request to download the fragmented data of the target file, the requested fragmented data is extracted from all fragmented data of the target file and transmitted to the client.

[0012] According to one aspect of the embodiments of this application, a file transfer apparatus is disclosed, the apparatus comprising: The first response module is configured to, in response to a client's request to download target fragment data, retrieve and cache the target file containing the target fragment data from the storage repository, extract the target fragment data from the target file, and transmit it to the client. The second response module is configured to, in response to a request from the client to download other fragments of the target file, extract other fragments of the target file from the target file and transmit them to the client.

[0013] In one exemplary embodiment of this application, the apparatus further includes: The third response module is configured to, in response to the client's first request to download the fragmented data of the target file, query other files associated with the target file, and at least pull and cache the header fragmented data of the other files located at the beginning of the file from the storage repository; The fourth response module is configured to, in response to the client's first request to download the fragmented data of the other files, transmit the header fragmented data of the other files to the client, and pull and cache the remaining fragmented data of the other files from the storage repository; The fifth response module is configured to, in response to a client's subsequent request to download fragment data of other files, extract the fragment data requested by the client from the remaining fragment data of the other files and transmit it to the client.

[0014] In one exemplary embodiment of this application, the third response module is configured as follows: In response to the client's initial request to download the fragmented data of the target file, the system queries other files associated with the target file and determines the degree of association between each of the other files and the target file. A predetermined number of other files are selected in descending order of their correlation, and the header fragment data of at least the selected other files are retrieved from the storage repository and cached.

[0015] In one exemplary embodiment of this application, the apparatus further includes: The receiving module is configured to establish at least two TCP connections with the client and receive a fragment of data from a file created by the client through the at least two TCP connections.

[0016] According to one aspect of the embodiments of this application, an electronic device is disclosed, comprising: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement any of the above embodiments.

[0017] According to one aspect of the embodiments of this application, a computer program medium is disclosed, on which computer-readable instructions are stored, which, when executed by a computer's processor, cause the computer to perform any of the above embodiments.

[0018] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described above.

[0019] The file transfer system provided in this application stores files in fragmented data format in the storage repository, ensuring the stability of each data transfer. Furthermore, the caching mechanism introduced at the access point improves resource utilization during file transfer, thereby increasing the file transfer speed. In response to the client's initial request to download fragmented data of a target file, the access point retrieves and caches all fragmented data of that target file from the storage repository. In this way, during the client's download of the target file, the access point only needs to retrieve data from the storage repository once, reducing the number of times the access point retrieves data from the storage repository, thereby further reducing the time spent during file download and improving the file download speed.

[0020] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0021] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0022] The above and other objectives, features and advantages of this application will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0023] Figure 1 A schematic diagram of the architecture of a file transfer system according to an embodiment of this application is shown.

[0024] Figure 2 A flowchart of a file transfer method according to an embodiment of this application is shown.

[0025] Figure 3 A schematic diagram of a file download process according to an embodiment of this application is shown.

[0026] Figure 4 A schematic diagram illustrating a client multiplexing multiple TCP connections to upload files according to an embodiment of this application is shown.

[0027] Figure 5 A schematic diagram illustrating a client multiplexing multiple TCP connections to upload files according to an embodiment of this application is shown.

[0028] Figure 6 A schematic diagram is shown illustrating a client performing a speed test and selecting the optimal file server address according to an embodiment of this application.

[0029] Figure 7 A schematic diagram illustrating a client downloading and writing a file according to an embodiment of this application is shown.

[0030] Figure 8 A block diagram of a file transfer apparatus according to an embodiment of this application is shown.

[0031] Figure 9 A hardware diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0032] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided to make the description of this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The drawings are merely illustrative of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0033] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of exemplary embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced with one or more of the specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0034] Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0035] Figure 1 A schematic diagram of the architecture of a file transfer system according to an embodiment of this application is shown.

[0036] refer to Figure 1 As shown, the file transfer system proposed in this application relates to the field of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0037] Cloud technology is a collective term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies applied to the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will all require robust system support, which can only be achieved through cloud computing.

[0038] The file transfer system proposed in this application includes a file server and a client 101 running on a terminal. Furthermore, the file server includes an access point 102 and a storage repository 103.

[0039] In this file transfer system, storage warehouse 103 is a server primarily used for long-term storage of massive amounts of file data, while access point 102 is a server that clients can directly connect to. When client 101 uploads a file, the data first travels from client 101 to access point 102 and is then stored in storage warehouse 103. When client 101 downloads a file, the data first travels from storage warehouse 103 to access point 102 and is then transmitted back to client 101. Access point 102 typically has a caching mechanism to temporarily cache data transmitted from storage warehouse 103. This allows client 101 to retrieve the required data directly from the cache of access point 102 when downloading the same file again, without needing access point 102 to fetch data from storage warehouse 103, thus saving time.

[0040] In this embodiment, the terminal can be any one of a smartphone, tablet computer, portable computer, and desktop computer; the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0041] Figure 2A flowchart of the file transfer method proposed in this application is shown. This method is applied to an access point in the file transfer system, and the method includes: Step S210: In response to the client's first request to download the fragmented data of the target file, pull and cache all fragmented data of the target file from the storage repository, and extract the fragmented data requested by the client for the first time from all fragmented data of the target file and transmit it to the client; Step S220: In response to the client's renewed request to download the fragmented data of the target file, extract the fragmented data requested by the client from all the fragmented data of the target file and transmit it to the client.

[0042] In this embodiment, the storage warehouse stores files in a fragmented data manner; that is, the files stored in the storage warehouse are divided into individual fragments. Correspondingly, when a client downloads a file, it requests one fragment of the file from the access point each time. After receiving the fragment returned by the access point, the client then requests another fragment of the file from the access point, and so on, until the client receives all the fragments of the file.

[0043] When a client makes its first request to the access point to download the fragmented data of a target file, the access point responds to the first request by pulling and caching all the fragmented data of the target file from the storage repository, and then extracts the fragmented data requested by the client and transmits it to the client.

[0044] Since a complete file can only be obtained after all its fragments are complete, a client will not only request to download one fragment of the target file in order to download the complete target file. The client will continue to request to download other fragments of the target file until the client has downloaded all the fragments of the target file.

[0045] Meanwhile, since there is usually a certain physical distance between the access point and the storage warehouse, the access point needs to spend a certain amount of time traveling back and forth geographically each time it pulls data from the storage warehouse. Therefore, if the access point pulls data from the storage warehouse every time the client requests to download a piece of data, it will cause the access point to need to pull data from the storage warehouse multiple times to meet the client's need to download a file, thus spending more time traveling back and forth geographically.

[0046] Therefore, in this embodiment, the access point, in response to the client's first request to download the fragment data of the target file, pulls and caches all fragment data of the target file from the storage repository. When the client requests the access point to download the fragment data of the target file again, the access point does not need to pull data from the storage repository again, but can directly transfer the requested fragment data from its cached fragment data of the target file to the client.

[0047] Therefore, the file transfer system provided in this application embodiment stores files in fragmented data format in the storage warehouse, ensuring the stability of single data transmission. Furthermore, the caching mechanism introduced by the access point improves resource utilization during file transfer, thereby increasing file transfer speed. In response to the client's first request to download fragmented data of the target file, the access point retrieves and caches all fragmented data of the target file from the storage warehouse. In this way, during the client's download of the target file, the access point only needs to retrieve data from the storage warehouse once, reducing the number of times the access point retrieves data from the storage warehouse, thereby further reducing the time spent during file download and improving file download speed.

[0048] Figure 3 A schematic diagram of a file download process according to an embodiment of this application is shown.

[0049] In this embodiment, the file requested for download by the client contains multiple data fragments. The fragment requested for download by the client for the first time is denoted as fragment 1, the fragment requested for download by the client for the second time is denoted as fragment 2, and so on.

[0050] After the client requests to download fragment 1 from the access point, the access point pulls the entire file from the storage repository and caches it, then returns fragment 1 data to the client. Since the access point and the storage repository usually use a dedicated network, the data transmission speed is much faster than the network data transmission speed between the client and the access point. Therefore, by the time the client has finished downloading fragment 1 data, the access point has already cached the entire file.

[0051] Therefore, when the client downloads fragment 1 and requests fragment 2 from the access point, the access point can directly retrieve fragment 2 from its cache and return it to the client, avoiding the need to fetch data from the storage repository again. Similarly, the process of the client downloading other fragments of the same file will not be described in detail.

[0052] In one embodiment, in response to the client's first request to download the fragmented data of the target file, the access point not only pulls and caches all the fragmented data of the target file, but also pulls and caches at least the header fragmented data of other files associated with the target file.

[0053] In this embodiment, after the access point receives the first request from the client to download the fragmented data of the target file, it responds to the client's first request to download the fragmented data of the target file by querying other files associated with the target file and at least pulling and caching the header fragmented data of other files located in the file header from the storage repository.

[0054] If the access point receives the first request from the client to download the fragmented data of other files, it will respond to the client's first request to download the fragmented data of other files by transmitting the header fragmented data of other files to the client, and pulling and caching the remaining fragmented data of other files from the storage repository.

[0055] If the access point receives another request from the client to download fragmented data of other files, it will respond to the client's request by extracting the requested fragmented data from the remaining fragmented data of other files and transmitting it to the client.

[0056] Specifically, multiple files stored in a storage repository can have relationships. These relationships can be described by whether the files are in the same file cluster or whether they were created by the same user. For example, video files belonging to the same TV series are considered to have a relationship; photo files created by the same user are also considered to have a relationship.

[0057] For other files associated with the target file, although the client has not yet requested to download these other files, it is very likely that the client will request to download these other files in some cases. Therefore, when the client first requests to download the fragment data of the target file, it will not only fetch and cache all fragment data of the target file, but also fetch and cache at least the header fragment data of these other files. For example: only fragment 1 of these other files is fetched and cached; or, in addition to fetching and caching fragment 1 of these other files, fragment 2 of these other files is also fetched and cached, and so on.

[0058] Therefore, once a client requests to download the fragment data of another file after requesting to download the fragment data of the target file, the access point can immediately transmit the header fragment data of the other file to the client, and pull and cache the remaining fragment data of the other file to deal with the client's subsequent request to download the fragment data of the other file.

[0059] The advantage of this embodiment is that when a client requests to download other files after requesting to download the target file, the header fragment data of the other files has been cached in the access point, and the access point will continue to pull and cache the remaining fragment data of the other files, so that the client downloads the other files almost without delay.

[0060] In one embodiment, when the access point retrieves other files, it retrieves a certain number of other files according to their degree of association.

[0061] In this embodiment, in response to the client's initial request to download the fragmented data of the target file, the access point queries other files associated with the target file and determines the degree of association between the other files and the target file. A preset number of other files are selected in descending order of association strength, and the header fragmented data of at least the selected other files are retrieved from the storage repository and cached.

[0062] Specifically, while querying other files associated with the target file, the access point determines the degree of association between these other files and the target file. This degree of association can be described by the distance between the files in the file cluster or by the interval between their creation times. For example, the degree of association between video files belonging to the same TV series can be described by the distance between the episode numbers of the corresponding series; the closer the distance, the stronger the association. Similarly, the degree of association between photo files uploaded by the same user can be described by the interval between their creation times; the smaller the interval, the stronger the association.

[0063] Then, N other files are selected in descending order of their correlation strength, and at least the header fragment data of these N other files is fetched and cached. Here, N is a natural number greater than 0. For example, when N is 3, only fragment 1 of the three other files with the highest correlation strength is fetched and cached; or, in addition to fetching and caching fragment 1 of the three other files with the highest correlation strength, fragment 2 of these three other files is also fetched and cached, and so on.

[0064] The advantage of this embodiment is that it retrieves and caches other files according to their correlation, which not only satisfies the client's file download needs but also reduces the caching burden on the access point.

[0065] In one embodiment, the file transfer system also includes a master server. The master server is primarily used for global control of communication within the file transfer system.

[0066] In this embodiment, to upload a file, the client requests the creation of the file from the main server and receives the file server address from the main server. Then, a TCP (Transmission Control Protocol) connection is established with the access point according to the file server address. The fragmented data of the created file is uploaded to the access point via the TCP connection. With the data relay function of the access point, the fragmented data of the created file is transmitted to the storage repository, resulting in a storage repository containing the created file.

[0067] Specifically, the client segments the created file into fragments, obtaining the fragment data. After obtaining the file server address, a TCP connection is established with the corresponding access point based on that file server address. The fragment data of the created file is then transmitted to the access point via this TCP connection, and the access point then transmits the fragment data to the storage repository, thus realizing the upload of the created file.

[0068] In one embodiment, during the file upload process, the client establishes at least two TCP connections with the access point, and uploads a fragment of the created file to the access point through each of the at least two TCP connections.

[0069] Specifically, considering that bandwidth resources are evenly distributed among each TCP connection according to the TCP protocol, the more TCP connections a client has, the more network bandwidth it receives, and the faster the file transfer speed. Therefore, during the file upload process, the client establishes multiple TCP connections with the access point, and then uploads multiple fragments of the created file to the access point in parallel through these multiple TCP connections.

[0070] The advantage of this embodiment is that by establishing multiple TCP connections with the access point to upload files, the network bandwidth utilized during the file upload process is increased, thereby further improving the file upload speed.

[0071] In one embodiment, during the file upload process, the client retains the TCP connection after the uploaded data of the segment it is responsible for has been uploaded, and uploads another segment of the created file to the access point through the TCP connection after the uploaded data of the segment it is responsible for.

[0072] Specifically, considering that each TCP connection establishment requires a three-way handshake, each TCP connection establishment takes more time. Furthermore, if HTTPS (Hypertext Transfer Protocol over Secure Socket Layer) encryption is used during communication, an additional negotiation process is added to each TCP connection establishment, further increasing the time required. Therefore, during file uploads, the client not only establishes multiple TCP connections with the access point but also reuses existing TCP connections. Once a TCP connection has finished uploading its assigned data segment, the client retains that TCP connection and assigns another unuploaded data segment to it, which then uploads that segment to the access point.

[0073] The advantage of this embodiment is that by reusing multiple TCP connections, the overhead of creating new TCP connections is saved, further improving file upload speed.

[0074] Figure 4 as well as Figure 5 This illustration shows a schematic diagram of a client multiplexing multiple TCP connections to upload files according to an embodiment of this application.

[0075] In this embodiment, the client splits the file to be uploaded into multiple chunks of a fixed size. For example, a 10MB file is split into 10 chunks, each 1MB. Then, five TCP connections a, b, c, d, and e are created, with each TCP connection responsible for uploading one chunk.

[0076] Initially, fragments 1 through 5 are uploaded simultaneously. After a period of time, fragment 3 finishes uploading first, leaving TCP connection c temporarily idle. In this case, TCP connection c is retained, and fragment 6 is assigned to TCP connection c for uploading. Similarly, the process for uploading other fragments will not be elaborated further.

[0077] In one embodiment, the master server responds to a client's request to create a file by returning at least two file server addresses to the client.

[0078] The client then connects to file servers at at least two file server addresses simultaneously and sends a handshake request to each file server. The client then establishes a TCP connection with the access point of the file server that first responds to the handshake request, based on the file server address of that file server.

[0079] For details, please refer to Figure 6 The diagram shown illustrates how a client performs speed testing and selection of a file server address according to an embodiment of this application. Considering that the file server address returned by the main server may not be optimal for the client under certain circumstances, the main server returns a set of file server addresses to the client. The client then tests the speed of the file server corresponding to each file server address and selects the fastest file server to establish a TCP connection with, thereby completing the file upload.

[0080] The client-side speed test involves the client simultaneously connecting to the file servers corresponding to these file server addresses and sending handshake requests. The file server that responds to the handshake request first is considered the fastest.

[0081] It should be noted that if all handshake requests for this set of file server addresses fail, the client's outgoing IP address (Internet Protocol Address) will be resolved using DNS (Domain Name System) to obtain a new file server address.

[0082] The advantage of this embodiment is that by testing the speed of multiple file server addresses before determining the actual file server used for file transfer, the response speed of the file server to the client is improved, further increasing the file transfer speed.

[0083] In one embodiment, during the file download process, the client downloads fragmented data from the access point via a first thread, transfers the downloaded fragmented data to a second thread, and continues downloading fragmented data from the access point. The second thread enqueues the fragmented data downloaded by the first thread into a data queue and writes the fragmented data in the data queue into the storage area of ​​the client's terminal.

[0084] Specifically, considering that if a single thread is used to simultaneously download fragmented data from the access point and write the downloaded fragmented data to the terminal's storage area, then downloading and writing can only be performed serially; downloading cannot be done while writing, and vice versa. Therefore, the client delegates downloading and writing to different threads. After downloading the fragmented data, the first thread passes the fragmented data to the second thread, and then the first thread continues to download other fragmented data; the second thread enqueues the fragmented data passed by the first thread into a data queue for temporary storage, and writes the fragmented data stored in the data queue into the storage area.

[0085] The advantage of this embodiment is that by implementing downloading and writing in parallel through multiple threads, conflicts between downloading and writing are avoided, further improving the file download speed.

[0086] Figure 7 A schematic diagram illustrating the client downloading and writing a file according to an embodiment of this application is shown.

[0087] In this embodiment, the client assigns file download processing to thread 1 and file writing processing to thread 2. Thread 1 downloads fragmented data from the access point via the network and passes the downloaded fragmented data to thread 2, thus continuing to download other fragmented data from the access point. Thread 2 enqueues the fragmented data passed from thread 1 into a data queue for temporary storage and writes the data stored in the data queue to the hard drive of the client's terminal, realizing local file writing.

[0088] Figure 8 A block diagram of a file transfer apparatus according to an embodiment of this application is shown, the apparatus comprising: The first response module 310 is configured to, in response to a client's request to download target fragment data, retrieve and cache the target file containing the target fragment data from the storage repository, extract the target fragment data from the target file, and transmit it to the client. The second response module 320 is configured to extract other fragments of the target file from the target file and transmit them to the client in response to a request from the client to download other fragments of the target file.

[0089] In one exemplary embodiment of this application, the apparatus further includes: The third response module is configured to, in response to the client's first request to download the fragmented data of the target file, query other files associated with the target file, and at least pull and cache the header fragmented data of the other files located at the beginning of the file from the storage repository; The fourth response module is configured to, in response to the client's first request to download the fragmented data of the other files, transmit the header fragmented data of the other files to the client, and pull and cache the remaining fragmented data of the other files from the storage repository; The fifth response module is configured to, in response to a client's subsequent request to download fragment data of other files, extract the fragment data requested by the client from the remaining fragment data of the other files and transmit it to the client.

[0090] In one exemplary embodiment of this application, the third response module is configured as follows: In response to the client's initial request to download the fragmented data of the target file, the system queries other files associated with the target file and determines the degree of association between each of the other files and the target file. A predetermined number of other files are selected in descending order of their correlation, and the header fragment data of at least the selected other files are retrieved from the storage repository and cached.

[0091] In one exemplary embodiment of this application, the apparatus further includes: The receiving module is configured to establish at least two TCP connections with the client and receive a fragment of data from a file created by the client through the at least two TCP connections.

[0092] The following is for reference. Figure 9 To describe the electronic device 40 according to an embodiment of this application. Figure 9 The electronic device 40 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0093] like Figure 9As shown, the electronic device 40 is presented in the form of a general-purpose computing device. The components of the electronic device 40 may include, but are not limited to: at least one processing unit 410, at least one storage unit 420, and a bus 430 connecting different system components (including storage unit 420 and processing unit 410).

[0094] The storage unit stores program code that can be executed by the processing unit 410, causing the processing unit 410 to perform the steps described in the exemplary method description section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 410 can perform actions such as... Figure 2 The steps shown are as follows.

[0095] Storage unit 420 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 4201 and / or cache memory 4202, and may further include a read-only memory (ROM) 4203.

[0096] Storage unit 420 may also include a program / utility 4204 having a set (at least one) program module 4205, such program module 4205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0097] Bus 430 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0098] Electronic device 40 can also communicate with one or more external devices 500 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 40, and / or with any device that enables electronic device 40 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 450. Input / output (I / O) interface 450 is connected to display unit 440. Furthermore, electronic device 40 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 460. As shown, network adapter 460 communicates with other modules of electronic device 40 via bus 430. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 40, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0099] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this application.

[0100] In an exemplary embodiment of this application, a computer-readable storage medium is also provided, on which computer-readable instructions are stored, which, when executed by a computer's processor, cause the computer to perform the methods described in the above method embodiments.

[0101] According to one embodiment of this application, a program product for implementing the methods in the above-described method embodiments is also provided. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of this invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0102] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0103] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0104] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0105] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as JAVA and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0106] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0107] Furthermore, although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0108] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0109] The data collection and processing plan outlined in this application must be implemented in strict accordance with the requirements of relevant national laws and regulations, obtaining the informed consent or separate consent of the data subject (or having a legal basis as stipulated by the relevant national laws and regulations), and conducting subsequent data use and processing within the scope authorized by laws and regulations and the data subject.

[0110] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the appended claims.

Claims

1. A file transfer system, characterized in that, The system includes a client and a file server. The file server includes a storage repository for storing files and a server for transferring data between the client and the storage repository. The storage warehouse is configured to store files in a data-sharded manner; The server is configured as follows: In response to the client's first request to download the fragment data of the target file, all fragment data of the target file are pulled from the storage repository and cached, and the fragment data requested by the client in the first request is extracted from all fragment data of the target file and transmitted to the client; In response to the client's subsequent request to download the fragmented data of the target file, the server extracts the fragmented data requested by the client from all the fragmented data of the target file that has been cached by the server and transmits it to the client. The system also includes a master server, and the client is configured as follows: The system requests the creation of a file from the master server and receives the file server address returned by the master server. At least two TCP connections are established with the file server address, and one fragment of the created file is uploaded to the server through each of the at least two TCP connections. The TCP connection that has completed uploading the fragment is reserved, and the other fragment of the created file is uploaded to the server through the TCP connection that has completed uploading the fragment. With the data relay function of the server, the fragment data of the created file is transferred to the storage warehouse, and a storage warehouse storing the created file is obtained. The client is also configured as follows: The first thread downloads fragmented data from the server, transmits the downloaded fragmented data to the second thread, and continues to download fragmented data from the server. The second thread enqueues the fragmented data downloaded by the first thread into a data queue, and writes the fragmented data in the data queue into the storage area of ​​the terminal where the client is located.

2. The file transfer system according to claim 1, characterized in that, The server is also configured to: In response to the client's first request to download the fragment data of the target file, query other files associated with the target file, and at least pull and cache the header fragment data of the other files located at the beginning of the file from the storage repository; In response to the client’s first request to download the fragment data of the other files, the header fragment data of the other files is transmitted to the client, and the remaining fragment data of the other files is retrieved from the storage repository and cached. In response to the client's subsequent request to download fragment data of the other files, the requested fragment data is extracted from the remaining fragment data of the other files and transmitted to the client.

3. The file transfer system according to claim 2, characterized in that, The server is configured as follows: In response to the client's initial request to download the fragmented data of the target file, the system queries other files associated with the target file and determines the degree of association between each of the other files and the target file. A predetermined number of other files are selected in descending order of their correlation, and the header fragment data of at least the selected other files are retrieved from the storage repository and cached.

4. The file transfer system according to claim 1, characterized in that, The master server is configured to return at least two file server addresses to the client in response to the client's request to create a file. The client is configured as follows: Simultaneously connect to the file servers at the addresses of at least two file servers and send a handshake request to each file server; A TCP connection is established with the file server address of the file server that first responds to the handshake request.

5. A file transfer method, characterized in that, The method includes: The client requests the creation of a file from the master server and receives the file server address returned by the master server. The client establishes at least two TCP connections with the server according to the file server address, and uploads one fragment of the created file to the server through the at least two TCP connections respectively. The TCP connection that has completed uploading the fragment is reserved, and the other fragment of the created file is uploaded to the server through the TCP connection that has completed uploading the fragment. With the data relay function of the server, the fragment data of the created file is transferred to the storage warehouse, and a storage warehouse storing the created file is obtained. In response to the client's first request to download the fragmented data of the target file, the server pulls and caches all fragmented data of the target file from the storage repository, and extracts the fragmented data requested by the client for the first time from all fragmented data of the target file and transmits it to the client; In response to a client's subsequent request to download fragmented data of the target file, the requested fragmented data is extracted from all locally cached fragmented data of the target file and transmitted to the client. The client is configured to: download fragmented data from the server via a first thread, transmit the downloaded fragmented data to a second thread, and continue downloading fragmented data from the server; enqueue the fragmented data downloaded by the first thread into a data queue via the second thread, and write the fragmented data in the data queue into the storage area of ​​the client's terminal.

6. A file transfer device, characterized in that, The device includes: The first response module is configured to respond to a client's request to download target fragment data, wherein the server pulls and caches the target file containing the target fragment data from the storage repository, extracts the target fragment data from the target file, and transmits it to the client. The second response module is configured to, in response to a request from the client to download other fragments of the target file, extract other fragments of the target file from the target file cached locally and transmit them to the client; The client is configured as follows: it requests the creation of a file from the master server and receives the file server address returned by the master server; it establishes at least two TCP connections with the server according to the file server address, and uploads one fragment of the created file to the server through the at least two TCP connections respectively; it retains the TCP connection after the fragment of the file it is responsible for has been uploaded, and uploads another fragment of the created file to the server through the TCP connection after the fragment of the file it is responsible for has been uploaded; under the data relay function of the server, the fragment data of the created file is transmitted to the storage warehouse, and a storage warehouse storing the created file is obtained. The client is further configured to: download fragmented data from the server via a first thread, transfer the downloaded fragmented data to a second thread, and continue downloading fragmented data from the server; enqueue the fragmented data downloaded by the first thread into a data queue via the second thread, and write the fragmented data in the data queue into the storage area of ​​the terminal where the client is located.

7. An electronic device, characterized in that, include: Memory, which stores computer-readable instructions; The processor reads computer-readable instructions stored in memory to execute the method of claim 5.

8. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by the computer's processor, cause the computer to perform the method of claim 5.

9. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, wherein a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to cause the computer device to perform the method as described in claim 5.

Images