Traffic balancing-based ftp distributed downloading method and device, medium and equipment

Abstract

The application provides a FTP distributed downloading method and device based on flow balance, a medium and equipment. The method comprises the following steps: S110, generating a file list for each FTP server; S120, obtaining the respective current downloading task history records of each FTP server; S130, filtering out the file information with the file state of distributed and downloaded in the file list to form a current downloading task list; S140, selecting one FTP server with the minimum flow as a current target FTP server; S150, distributing one downloading task in the downloading task list to the target FTP server to enable the target FTP server to execute the downloading task; S160, marking the file state of the downloading task as distributed, and moving the downloading task from the downloading task list to the downloading task history record of the target FTP server; S170, judging whether the current downloading task list is empty; and S180, if yes, ending the method. The embodiment of the application balances the pressure of each FTP server.

Description

Technical Field

[0001] This invention relates to the field of file transfer technology, and in particular to an FTP distributed download method, apparatus, medium, and device based on traffic balancing. Background Technology

[0002] FTP (File Transportation Protocol) is a standard protocol for transferring files over a network, allowing users to communicate with another host via file operations. Downloading files via FTP is the most common application scenario. In practice, multiple FTP servers are typically involved in the download, using a peer-to-peer approach, where one FTP client downloads files from one FTP server. When some FTP servers have too many files, the load on the corresponding FTP client increases, leading to download delays or even failure to complete the download, resulting in a backlog. Summary of the Invention

[0003] To address at least one of the above technical problems, embodiments of the present invention provide an FTP distributed download method, apparatus, medium, and device based on traffic balancing.

[0004] According to a first aspect, the FTP distributed download method based on traffic balancing provided by embodiments of the present invention includes:

[0005] S110. Generate a file list for each FTP server; wherein, the file list includes multiple file information entries, each including a file identifier and a file status;

[0006] S120. Obtain the current download task history of each FTP server;

[0007] S130. Based on the current download task history of each FTP server, filter out the file information in the file list whose file status is distributed and downloaded to form the current download task list; wherein, each file information in the filtered file list is used as a download task in the download task list.

[0008] S140. Based on the download task history of each FTP server, select the FTP server with the lowest traffic as the current target FTP server.

[0009] S150. Distribute a download task from the download task list to the target FTP server so that the target FTP server executes the download task.

[0010] S160. Mark the file status of the download task as distributed, and move the download task from the download task list to the download task history of the target FTP server;

[0011] S170. Determine if the current download task list is empty;

[0012] S180. If yes, then this method ends;

[0013] S190, otherwise, return to S140.

[0014] In one embodiment, generating a file list for each FTP server includes:

[0015] Connect to various FTP servers;

[0016] Obtain the first information of each FTP server among the FTP servers; wherein the first information includes the IP address and directory of the FTP server;

[0017] The directories in the first information of each FTP server are scanned to obtain the second information of each file on that FTP server; wherein, the second information of each file includes the file path and file status of the file;

[0018] The file list is generated based on the first information of each FTP server and the second information of each file on each FTP server; wherein the file list includes multiple file information entries, and each file information entry includes the first information of the FTP server from which the corresponding file comes and the second information of the corresponding file.

[0019] In one embodiment, the first information further includes a username, password, and port; the second information further includes the file size and file generation time.

[0020] In one embodiment, the IP address and file path in each file information entry in the file list form the file identifier for that file information.

[0021] In one embodiment, selecting the FTP server with the lowest download task history as the current target FTP server based on the download task history of each of the FTP servers includes:

[0022] Calculate the total size of historical files for each FTP server based on its download task history.

[0023] Based on the total size of historical files on each FTP server, select the FTP server with the smallest total size of historical files.

[0024] If the number of FTP servers with the smallest total historical file size is 1, then the FTP server with the smallest total historical file size will be used as the current target FTP server.

[0025] If the number of FTP servers with the smallest total historical file size is greater than 1, then one FTP server is randomly selected from the FTP servers with the smallest total historical file size as the current target FTP server.

[0026] In one embodiment, the method provided by this invention further includes:

[0027] After the target FTP server completes the distributed download task, the file status of the download task in the download task history of the target FTP server is updated from "distributed" to "downloaded".

[0028] In one embodiment, distributing a download task from the download task list to the target FTP server so that the target FTP server executes the download task includes:

[0029] The first download task in the download task list is distributed to the target FTP server so that the target FTP server executes the first download task;

[0030] Correspondingly, marking the file status of the download task as distributed and moving the download task from the download task list to the download task history of the target FTP server includes: marking the file status of the first download task as distributed and moving the first download task from the download task list to the download task history of the target FTP server.

[0031] According to a second aspect, the FTP distributed download device based on traffic balancing provided in the embodiments of the present invention includes:

[0032] The list generation module is used to execute S110 and generate a file list for each FTP server; wherein, the file list includes multiple file information entries, and each file information entry includes a file identifier and a file status;

[0033] The history acquisition module is used by S120 to acquire the current download task history of each FTP server.

[0034] The information filtering module is used in S130 to filter out file information with the status of "distributed" and "downloaded" from the file list based on the current download task history of each FTP server, forming the current download task list; wherein, each file information in the filtered file list is used as a download task in the download task list;

[0035] The target selection module is used in S140 to select the FTP server with the lowest traffic as the current target FTP server based on the download task history of each FTP server.

[0036] The task distribution module is used in S150 to distribute a download task from the download task list to the target FTP server so that the target FTP server executes the download task.

[0037] The status marking module is used in S160 to mark the file status of the download task as distributed and move the download task from the download task list to the download task history of the target FTP server;

[0038] The first judgment module is used in S170 to determine whether the current download task list is empty;

[0039] The first processing module is used for S180; if yes, then this method ends.

[0040] The second processing module is used for S190; otherwise, it returns to the target selection module to execute S140.

[0041] According to a third aspect, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method provided in the first aspect.

[0042] According to a fourth aspect, the computing device provided in the embodiments of the present invention includes a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method provided in the first aspect.

[0043] The FTP distributed download method, apparatus, medium, and device based on traffic balancing provided in this invention generate a file list for each FTP server. Then, based on the download task history of each FTP server, files in the file list with the status of "distributed" and "downloaded" are filtered out, forming the current download task list. The FTP server with the lowest traffic is selected as the current target FTP server. A download task from the download task list is distributed to the target FTP server, allowing it to execute the download task. The file status of the download task is then marked as "distributed," and the download task is moved from the download task list to the download task history of the target FTP server. This process continues until all download tasks are completed. It is evident that this invention addresses multiple FTP servers and selects the FTP server with the lowest traffic as the target FTP server to execute the selected download task, achieving traffic balancing across multiple FTP servers. This ensures that the load on each FTP server is evenly distributed, preventing some FTP servers from being overloaded while others are underutilized. Therefore, it reduces download task delays and avoids congestion. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating an embodiment of the FTP distributed download method based on traffic balancing according to the present invention.

[0045] Figure 2 This is a schematic diagram of the first information of multiple FTP servers in one embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of a file list in one embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of the download task history of an FTP server in one embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of a download task list in one embodiment of the present invention;

[0049] Figure 6 This is a structural block diagram of an FTP distributed download device based on traffic balancing in one embodiment of the present invention. Detailed Implementation

[0050] In a first aspect, embodiments of the present invention provide an FTP distributed download method based on traffic balancing, see [link to relevant documentation]. Figure 1 The method includes the following steps S110 to S190:

[0051] S110. Generate a file list for each FTP server; wherein, the file list includes multiple file information entries, each including a file identifier and a file status;

[0052] FTP stands for File Transportation Protocol. It is a standard protocol used for transferring files over a network, allowing users to communicate with another host through file operations.

[0053] The method provided in this embodiment of the invention can be executed by an FTP client.

[0054] In one embodiment, generating a file list for each FTP server may include:

[0055] 1. Connect to each FTP server;

[0056] 2. Obtain the first information of each FTP server among the FTP servers; wherein, the first information includes the IP address and directory of the FTP server;

[0057] 3. Scan the directories in the first information of each FTP server to obtain the second information of each file on that FTP server; wherein, the second information of each file includes the file path and file status of the file;

[0058] 4. Generate the file list based on the first information of each FTP server and the second information of each file on each FTP server; wherein the file list includes multiple file information entries, and each file information entry includes the first information of the FTP server from which the corresponding file comes and the second information of the corresponding file.

[0059] In other words, the FTP client connects to various FTP servers to obtain the initial information of each server, including its IP address and directory information. It then scans the directory of each FTP server to obtain the secondary information of each file. Finally, it combines the initial information from each FTP server with the secondary information of each file to generate a file list. This file list contains multiple entries, each containing the initial information of one FTP server and the secondary information of a file on that server, thus completing the file list generation process.

[0060] The directory of an FTP server can reflect the directory structure formed by the various files stored on that FTP server.

[0061] Among them, the file identifier can uniquely identify a file.

[0062] The file status includes undistributed, distributed, and downloaded. Therefore, the above file list may include file information for undistributed files, distributed files, and downloaded files.

[0063] In one embodiment, the first information may further include a username, password, and port; the second information may further include file size and file generation time.

[0064] In other words, a file information entry will include information such as IP address, username, password, port, file path, file status, file size, and file creation time, which can more comprehensively display the relevant information of each file.

[0065] In one embodiment, the IP address and file path in each file information entry in the file list form the file identifier for that file information.

[0066] That is, the file identifier of the file is formed by the IP address of the FTP server and the file path of the file, thus achieving a unique identifier.

[0067] S120. Obtain the current download task history of each FTP server;

[0068] Each FTP server has its own download task history.

[0069] Each FTP server's current download task history includes both distributed and downloaded download tasks. After a download task is distributed to an FTP server, it is moved to that server's download task history, and the corresponding file's status changes to "distributed." Once the download task is completed, the file's status changes to "downloaded."

[0070] S130. Based on the current download task history of each FTP server, filter out the file information in the file list whose file status is distributed and downloaded to form the current download task list; wherein, each file information in the filtered file list is used as a download task in the download task list.

[0071] In other words, the file information corresponding to the download tasks recorded in the download task history of each FTP server needs to be deleted from the file list to ensure that only the file information of undistributed files remains in the file list. After filtering the file list, a download task list is formed, and each file in the file list is a download task in the download task list.

[0072] S140. Based on the download task history of each FTP server, select the FTP server with the lowest traffic as the current target FTP server.

[0073] Traffic refers to the total size of all files corresponding to the download tasks distributed by the FTP server. Selecting the FTP server with the lowest traffic as the target FTP server can achieve traffic balancing among the FTP servers.

[0074] In one embodiment, the step S140 of selecting the FTP server with the lowest download task history as the current target FTP server may include:

[0075] S141. Calculate the total size of historical files for each FTP server based on its download task history.

[0076] S142. Based on the total size of historical files of each FTP server, select the FTP server with the smallest total size of historical files.

[0077] S143. If the number of FTP servers with the smallest total historical file size is 1, then the FTP server with the smallest total historical file size will be used as the current target FTP server.

[0078] S144. If the number of FTP servers with the smallest total historical file size is greater than 1, then randomly select one FTP server from the FTP servers with the smallest total historical file size as the current target FTP server.

[0079] In other words, based on the download task history of each FTP server, the total size of the historical files corresponding to each download task distributed to that FTP server can be calculated. Then, the FTP server with the smallest total historical file size can be selected from among all the FTP servers. If there is one FTP server with the smallest total historical file size, it will be used as the target FTP server. If there are multiple FTP servers with the smallest total historical file size, one will be randomly selected from among them as the target FTP server. This method ensures that a download task is assigned to only one FTP server.

[0080] S150. Distribute a download task from the download task list to the target FTP server so that the target FTP server executes the download task.

[0081] S160. Mark the file status of the download task as distributed, and move the download task from the download task list to the download task history of the target FTP server;

[0082] In other words, after a download task is distributed to the target FTP server, the target FTP server will execute the download task. Furthermore, the file status of the download task is marked as distributed, and the download task is moved from the download task list to the target FTP server's download task history. Therefore, the download task list will have one less download task, while the target FTP server's download task history will have one more download task record.

[0083] S170. Determine if the current download task list is empty;

[0084] S180. If yes, then this method ends;

[0085] S190, otherwise, return to S140.

[0086] Download tasks are distributed using the above method. If there are still download tasks in the download task list, the process will return to S140 and repeat the above steps until all download tasks in the download task list have been distributed, at which point the method ends.

[0087] For example, see the first information for multiple FTP servers. Figure 2 See the file list. Figure 3 For the download task history of an FTP server, see [link / reference]. Figure 4 See the generated download task list. Figure 5 .

[0088] In one embodiment, the method provided by this invention may further include:

[0089] After the target FTP server completes the distributed download task, the file status of the download task in the download task history of the target FTP server is updated from "distributed" to "downloaded".

[0090] In other words, after the target FTP server completes the distributed download task, the file status of that download task in the target FTP server's download task history will be updated to "downloaded", realizing the state transition from "distributed" to "downloaded".

[0091] In one embodiment, distributing a download task from the download task list to the target FTP server so that the target FTP server executes the download task includes:

[0092] The first download task in the download task list is distributed to the target FTP server so that the target FTP server executes the first download task;

[0093] Correspondingly, marking the file status of the download task as distributed and moving the download task from the download task list to the download task history of the target FTP server includes: marking the file status of the first download task as distributed and moving the first download task from the download task list to the download task history of the target FTP server.

[0094] In other words, when distributing download tasks, the first download task in the download task list is sent to the target FTP server. After distribution is complete, the first download task is marked as distributed and moved from the download task list to the download task history of the target FTP server.

[0095] Using the above methods to select download tasks for distribution is simple and efficient.

[0096] As can be seen, in this embodiment of the invention, by decomposing the FTP download process and distributing download tasks evenly to the FTP server according to the size of the traffic, resources can be fully utilized, data download latency can be reduced, and overall download efficiency can be improved.

[0097] In a real-world scenario, the method provided in this embodiment of the invention reduces the overall download latency of files on 200 FTP servers for a mobile operator in a certain province from 40 minutes to 10 minutes, avoiding the bottleneck effect and improving file download efficiency and the utilization efficiency of the FTP servers.

[0098] In summary, in this embodiment of the invention, a file list is generated for each FTP server. Then, based on the current download task history of each FTP server, files in the file list with the status of "distributed" and "downloaded" are filtered out to form the current download task list. The FTP server with the lowest traffic is selected as the current target FTP server. A download task from the download task list is distributed to the target FTP server so that the target FTP server can execute the download task. The file status of the download task is then marked as "distributed," and the download task is moved from the download task list to the download task history of the target FTP server. This process is repeated until all download tasks are completed. It is evident that this embodiment of the invention targets multiple FTP servers and selects the FTP server with the lowest traffic as the target FTP server to execute the selected download task. This achieves traffic balancing across multiple FTP servers, ensuring that the load on each FTP server is evenly distributed, preventing some FTP servers from being overloaded while others are underutilized. Therefore, it can reduce download task delays and avoid congestion.

[0099] Secondly, embodiments of the present invention provide an FTP distributed download device based on traffic balancing, see [link to relevant documentation]. Figure 6 The device includes:

[0100] The list generation module is used to execute S110 and generate a file list for each FTP server; wherein, the file list includes multiple file information entries, and each file information entry includes a file identifier and a file status;

[0101] The history acquisition module is used by S120 to acquire the current download task history of each FTP server.

[0102] The information filtering module is used in S130 to filter out file information with the status of "distributed" and "downloaded" from the file list based on the current download task history of each FTP server, forming the current download task list; wherein, each file information in the filtered file list is used as a download task in the download task list;

[0103] The target selection module is used in S140 to select the FTP server with the lowest traffic as the current target FTP server based on the download task history of each FTP server.

[0104] The task distribution module is used in S150 to distribute a download task from the download task list to the target FTP server so that the target FTP server executes the download task.

[0105] The status marking module is used in S160 to mark the file status of the download task as distributed and move the download task from the download task list to the download task history of the target FTP server;

[0106] The first judgment module is used in S170 to determine whether the current download task list is empty;

[0107] The first processing module is used for S180; if yes, then this method ends.

[0108] The second processing module is used for S190; otherwise, it returns to the target selection module to execute S140.

[0109] In one embodiment, the list generation module includes:

[0110] The server connection unit is used to connect various FTP servers;

[0111] The first information acquisition unit is used to acquire first information of each FTP server among the FTP servers; wherein, the first information includes the IP address and directory of the FTP server;

[0112] The directory scanning unit is used to scan the directories in the first information of each FTP server to obtain the second information of each file on the FTP server; wherein, the second information of each file includes the file path and file status of the file;

[0113] The list generation unit is used to generate the file list based on the first information of each FTP server and the second information of each file on each FTP server; wherein the file list includes multiple file information entries, and each file information entry includes the first information of the FTP server from which the corresponding file comes and the second information of the corresponding file.

[0114] In one embodiment, the first information further includes a username, password, and port; the second information further includes the file size and file generation time.

[0115] In one embodiment, the IP address and file path in each file information entry in the file list form the file identifier for that file information.

[0116] In one embodiment, the target selection module includes:

[0117] The size calculation unit is used to calculate the total size of the historical files of each FTP server based on the download task history of each FTP server.

[0118] The first selection unit is used to select the FTP server with the smallest total historical file size based on the total historical file size of each FTP server.

[0119] The first determining unit is configured to, if the number of FTP servers with the smallest total historical file size is 1, then select the FTP server with the smallest total historical file size as the current target FTP server.

[0120] The second determining unit is used to randomly select an FTP server from the FTP servers with the smallest total historical file size as the current target FTP server if the number of such servers is greater than 1.

[0121] In one embodiment, the apparatus provided by the present invention may further include:

[0122] The status update module is used to update the file status of the download task in the download task history of the target FTP server from "distributed" to "downloaded" after the target FTP server completes the distributed download task.

[0123] In one embodiment, the task distribution module is specifically used to: distribute the first download task in the download task list to the target FTP server, so that the target FTP server executes the first download task; correspondingly, the status marking module is specifically used to: mark the file status of the first download task as distributed, and move the first download task from the download task list to the download task history of the target FTP server.

[0124] It is understood that explanations of the contents, specific implementation methods, beneficial effects, examples, etc. in the system provided in the embodiments of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0125] Thirdly, embodiments of the present invention provide a computer-readable medium storing computer instructions, which, when executed by a processor, cause the processor to perform the method provided in the first aspect.

[0126] Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0127] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0128] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0129] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0130] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion module connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion module execute some and all of the actual operations, thereby realizing the function of any of the above embodiments.

[0131] It is understood that explanations, specific implementation methods, beneficial effects, examples, etc. of the contents in the computer-readable medium provided in the embodiments of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0132] Fourthly, one embodiment of this specification provides a computing device including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method of any embodiment of the specification.

[0133] It is understood that explanations, specific implementation methods, beneficial effects, examples, etc. of the computing device provided in the embodiments of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0134] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0135] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, widgets, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0136] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A flow balance based FTP distributed download method, characterized in that, The method comprises the following steps: S110, generating a file list for each FTP server; wherein the file list comprises a plurality of file information, and each file information comprises a file identifier and a file state; S120, obtaining a respective current download task history record of each FTP server; S130, filtering out file information with a file state of distributed and downloaded from the file list according to the respective current download task history record of each FTP server, to form a current download task list; wherein each file information in the filtered file list is a download task in the download task list; S140, selecting an FTP server with minimum traffic as a target FTP server according to the respective download task history record of each FTP server; The method of selecting an FTP server with minimum traffic as a target FTP server according to the respective download task history record of each FTP server comprises the following steps: calculating a respective total historical file size of each FTP server according to the respective download task history record of each FTP server; selecting an FTP server with minimum total historical file size; if the number of FTP servers with minimum total historical file size is one, selecting the FTP server with minimum total historical file size as the target FTP server; if the number of FTP servers with minimum total historical file size is more than one, randomly selecting an FTP server from the FTP servers with minimum total historical file size as the target FTP server; S150, distributing a download task in the download task list to the target FTP server, so that the target FTP server executes the download task; S160, marking a file state of the download task as distributed, and moving the download task from the download task list to a download task history record of the target FTP server; S170, judging whether the current download task list is empty; S180, if yes, ending the method; S190, otherwise, returning to S140.

2. The method of claim 1, wherein, The method of generating a file list for each FTP server comprises the following steps: connecting each FTP server; obtaining first information of each FTP server; wherein the first information comprises an IP address and a directory of the FTP server; scanning the directory in the first information of each FTP server to obtain second information of each file on the FTP server; wherein the second information of each file comprises a file path and a file state of the file; generating the file list according to the first information of each FTP server and the second information of each file on each FTP server; wherein the file list comprises a plurality of file information, and each file information comprises first information of a corresponding FTP server and second information of a corresponding file.

3. The method of claim 2, wherein, The first information further comprises a user name, a password and a port; and the second information further comprises a file size and a file generation time.

4. The method of claim 2, wherein, The IP address and the file path in each file information in the file list form a file identifier of the file information.

5. The method of claim 1, wherein, Further comprising: After the target FTP server completes the distributed download task, the file state of the download task in the download task history record of the target FTP server is updated from distributed to downloaded.

6. The method of claim 1, wherein, The distributing of the download task in the download task list to the target FTP server, so that the target FTP server executes the download task, comprises: The distributing of the first download task in the download task list to the target FTP server, so that the target FTP server executes the first download task; Correspondingly, the marking of the file state of the download task as distributed and the moving of the download task from the download task list to the download task history record of the target FTP server, comprises: the marking of the file state of the first download task as distributed and the moving of the first download task from the download task list to the download task history record of the target FTP server.

7. A computer readable storage medium characterized in that, A computer program is stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the method of any one of claims 1-6.

8. A computing device, comprising: A computer program is stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the method of any one of claims 1-6.

Images