A file downloading method, device, apparatus and storage medium

Abstract

The application discloses a file downloading method, device and equipment and a storage medium. In response to a file downloading instruction, a specified file total list on a file server is acquired. File identifiers of at least one file are recorded in the file total list. The file total list is subtracted from a pre-stored processed file list to obtain a total list to be downloaded. The processed file list includes a downloaded file list. Files are downloaded from the file server according to the total list to be downloaded, and the downloaded file list is updated. According to the embodiment of the application, the total list to be downloaded can be determined by designing the file total list and the processed file list, so that the file can be downloaded from the file server without designing an external system, and the architecture complexity is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of file downloading, in particular to a file downloading method, device, equipment and storage medium. BACKGROUND

[0002] In actual application, the files on the file server may need to be downloaded for several times to be completed. In the prior art, the multi-batch downloading of the files is realized by receiving the file ID to be downloaded sent by an external system, but the design of introducing the external system increases the architecture complexity of the whole project. SUMMARY

[0003] Therefore, the present application provides a file downloading method, device, equipment and storage medium, which can determine the file list to be downloaded by designing the file total list and the processed file list, so as to download the files from the file server without designing the external system, thereby reducing the architecture complexity.

[0004] To achieve the above object, the present application provides a file downloading method, comprising:

[0005] In response to a file downloading instruction, a specified file total list on a file server is acquired; wherein the file total list records the file identification of at least one file;

[0006] The file total list is subtracted from a pre-stored processed file list to obtain a total list to be downloaded; wherein the processed file list comprises a downloaded file list;

[0007] According to the total list to be downloaded, the files are downloaded from the file server, and the downloaded file list is updated.

[0008] As an improvement of the above scheme, the processed file list further comprises a large file list; and according to the total list to be downloaded, the files are downloaded from the file server, and the downloaded file list is updated, comprising:

[0009] The file volume of each file corresponding to the file identification in the total list to be downloaded is acquired;

[0010] The file identification with the file volume greater than a preset volume threshold is recorded in the large file list, and the file identification with the file volume greater than the preset volume threshold is deleted from the total list to be downloaded;

[0011] When the large file downloading permission is closed and the total number of the file identifications in the total list to be downloaded is greater than or equal to a preset opening threshold, the files are downloaded from the file server according to the total list to be downloaded;

[0012] when the large file downloading permission is closed and the total number of file identifiers in the total list to be downloaded is less than a preset opening threshold, opening the large file downloading permission;

[0013] when the large file downloading permission is opened, downloading files from the file server according to the total list to be downloaded and the large file list;

[0014] recording file identifiers of files whose downloading is completed in the downloaded file list, and deleting file identifiers of files whose downloading is completed from the total list to be downloaded and the large file list.

[0015] As an improvement of the above-mentioned solution, the downloading of files from the file server according to the total list to be downloaded, the updating of the downloaded file list, comprises:

[0016] splitting the total list to be downloaded into at least two sub lists to be downloaded;

[0017] allocating the at least two sub lists to be downloaded to different file downloading devices respectively; wherein the sub list to be downloaded corresponds to the allocated file downloading device;

[0018] updating the downloaded file list by downloading files from the file server according to the corresponding sub list to be downloaded through the file downloading device.

[0019] As an improvement of the above-mentioned solution, the processed file list further comprises a large file list;

[0020] The step of updating the downloaded file list by downloading files from the file server according to the corresponding sub list to be downloaded through the file downloading device, comprises:

[0021] obtaining file volumes of files to which each file identifier in the sub list to be downloaded belongs;

[0022] recording file identifiers of files in the sub list to be downloaded whose file volumes are greater than a corresponding preset volume threshold in the large file list of the corresponding file downloading device, and deleting file identifiers of files in the sub list to be downloaded whose file volumes are greater than the corresponding preset volume threshold; wherein each file downloading device is respectively configured with a corresponding preset volume threshold;

[0023] when the large file downloading permission of the file downloading device is closed and the total number of file identifiers in the corresponding sub list to be downloaded is greater than or equal to a corresponding preset opening threshold, downloading files from the file server according to the corresponding sub list to be downloaded; wherein each file downloading device is respectively configured with a corresponding preset opening threshold;

[0024] when the large file downloading permission of the file downloading device is closed and the total number of file identifiers in the corresponding sub-list of files to be downloaded is less than a corresponding preset opening threshold, the large file downloading permission is opened;

[0025] when the large file downloading permission of the file downloading device is opened, files are downloaded from the file server according to the corresponding sub-list of files to be downloaded and the large file list;

[0026] file identifiers of files for which file downloading is completed are recorded in the downloaded file list, and file identifiers of files for which file downloading is completed are deleted from the sub-list of files to be downloaded and the large file list.

[0027] As an improvement of the above-mentioned scheme, before the step of downloading files from the file server according to the total list of files to be downloaded and updating the downloaded file list, the method further comprises:

[0028] when the total number of file identifiers in the total list of files to be downloaded is greater than a preset downloading threshold, at least one file identifier in the total list of files to be downloaded is deleted, so that the total number of file identifiers in the total list of files to be downloaded is less than or equal to the preset downloading threshold.

[0029] As an improvement of the above-mentioned scheme, the processed file list further comprises an abnormal file list, and the method further comprises:

[0030] when an abnormal file occurs in the process of downloading files from the file server, a file identifier of the abnormal file is excluded from the total list of files to be downloaded, and the file identifier of the abnormal file is recorded in the abnormal file list stored in the external storage device, so that a user repairs files of the file server according to the file identifier of the abnormal file obtained through a preset API;

[0031] when it is monitored that the user completes the file repair operation of the file server, a file identifier of a repaired file is determined;

[0032] the API is called to the file identifier of the repaired file in the abnormal file list.

[0033] As an improvement of the above-mentioned scheme, the preset opening threshold is less than the preset downloading threshold, and the preset opening threshold and the preset downloading threshold have a positive correlation relationship; the better the network state for downloading files from the file server, the greater the preset downloading threshold.

[0034] As an improvement of the above-mentioned scheme, the processed file list is stored in a database.

[0035] To achieve the above-mentioned purpose, an embodiment of the present application further provides a file downloading device, which comprises:

[0036] The file total list acquisition module is configured to acquire a specified file total list on a file server in response to a file download instruction, wherein the file total list records file identifiers of at least one file.

[0037] The to-be-downloaded list acquisition module is configured to subtract the file total list from a pre-stored processed file list to obtain a to-be-downloaded total list, wherein the processed file list includes a downloaded file list.

[0038] The file download module is configured to download files from the file server according to the to-be-downloaded total list and update the downloaded file list.

[0039] To achieve the above object, the embodiment of the present application further provides a file download device, comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the file download method as described in any of the above embodiments.

[0040] To achieve the above object, the embodiment of the present application further provides a computer readable storage medium, comprising a stored computer program, wherein the computer program controls a device where the computer readable storage medium is located to execute the file download method as described in any of the above embodiments when the computer program runs.

[0041] Compared with the prior art, the file download method, device, equipment and storage medium disclosed by the embodiment of the present application firstly acquire a specified file total list on a file server in response to a file download instruction, wherein the file total list records file identifiers of at least one file; then subtract the file total list from a pre-stored processed file list to obtain a to-be-downloaded total list, wherein the processed file list includes a downloaded file list; finally download files from the file server according to the to-be-downloaded total list and update the downloaded file list. Therefore, the embodiment of the present application can determine a to-be-downloaded file list by designing a file total list and a processed file list to download files from a file server, without designing an external system, thereby reducing the architecture complexity. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings described in the following only are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0043] Figure 1 is a schematic diagram of a file download method provided by an embodiment of the present application;

[0044] Figure 2 is a schematic diagram of a file download process provided by an embodiment of the present application;

[0045] Figure 3 is a schematic diagram of a large file download control method provided by an embodiment of the present application;

[0046] Figure 4 is a schematic diagram of an architecture of a file download method provided by an embodiment of the present application;

[0047] Figure 5 is a schematic diagram of a structure of a file download device provided by an embodiment of the present application;

[0048] Figure 6 is a schematic diagram of a structure of a file download device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0050] The current file download demand is very common, but there is a premise for the file download, that is, the file to be downloaded this time is determined. Whenever the file download is involved, an external system or component is always associated to get the file list to be downloaded, and the list is sent to the file download module, which increases the complexity of system design.

[0051] Based on this, the embodiments of the present application directly integrate the acquisition work of the file to be downloaded into the internal system, determine the file list to be downloaded by designing a total file list and a processed file list, and download the file from the file server, without the need of designing an external system, thereby reducing the architecture complexity.

[0052] Referring to Figure 1 is a schematic diagram of a file download method provided by an embodiment of the present application. Specifically, the file download method comprises steps S11-S13:

[0053] S11, in response to a file download instruction, acquiring a specified total file list on a file server; wherein the total file list records file identifiers of at least one file;

[0054] S12, subtracting the file total list from a pre-stored processed file list to obtain a total list to be downloaded; wherein the processed file list comprises a downloaded file list;

[0055] S13, downloading files from the file server according to the total list to be downloaded and updating the downloaded file list.

[0056] It is worth mentioning that the method is particularly suitable for multi-batch downloading tasks of files, and is executed by a machine for downloading files. The file downloading instruction can be set to be triggered at a fixed time or input by a user through a keyboard, a mouse, a touchpad or the like. The specific triggering manner is not limited herein.

[0057] Specifically, for the machine, the file list to be downloaded each time needs to be confirmed before executing a file downloading task. The embodiment of the application solves this problem by designing a processed file list and a file total list. The file total list refers to a file identifier set of all files on the file server that meet the downloading requirements at this time. The processed file list comprises a downloaded file list. All file identifiers in the downloaded file list refer to file identifiers of all files that have been downloaded by the machine. The file identifier is metadata of the file, which is used to distinguish between files. The file identifier can be a file name or a file identifier code, which is not limited herein. First, the machine obtains the file total list on the file server, subtracts the processed file list from the file total list to obtain a total list to be downloaded, downloads files from the file server according to the total list to be downloaded, updates the downloaded files according to the downloading situation, and finally records file identifiers of the files downloaded this time in the downloaded file list. In this process, no external system needs to be designed additionally, thereby reducing the architecture complexity.

[0058] In an optional embodiment, the processed file list further comprises a large file list; and the downloading files from the file server according to the total list to be downloaded and updating the downloaded file list comprises:

[0059] obtaining file volumes of files to which the file identifiers in the total list to be downloaded belong;

[0060] recording the file identifiers of the files with the file volumes greater than a preset volume threshold in the large file list, and deleting the file identifiers of the files with the file volumes greater than the preset volume threshold from the total list to be downloaded;

[0061] when the large file downloading permission is closed and the total number of the file identifiers in the total list to be downloaded is greater than or equal to a preset opening threshold, downloading files from the file server according to the total list to be downloaded;

[0062] When the large file download permission is turned off and the total number of file identifiers in the total list to be downloaded is less than a preset threshold, the large file download permission is enabled.

[0063] When the large file download permission is enabled, files are downloaded from the file server according to the total list of files to be downloaded and the large file list.

[0064] The file identifiers of completed file downloads are recorded in the list of downloaded files, and the file identifiers of completed file downloads are deleted from the total list of files to be downloaded and the list of large files.

[0065] Specifically, the processed file list also includes a large file list. The files corresponding to the file identifiers stored in the large file list are all larger than a preset size threshold. The specific value of the preset size threshold is set according to the actual scenario, and its size is positively correlated with network performance. First, the machine obtains a list of all eligible files from the file server. This list is then subtracted from the downloaded file list and the large file list stored in Redis (Redis is a NoSQL database) to obtain the total list to be downloaded. Next, the file sizes corresponding to the file identifiers in the total list to be downloaded are analyzed. Files with sizes larger than the preset size threshold are recorded in the large file list, while other files with sizes larger than the preset size threshold are deleted from the total list to be downloaded. Then, according to the pre-set download strategy, files are downloaded based on the large file list and the total list to be downloaded. See [link to download strategy]. Figure 3 The flowchart illustrating the large file download control method shows the following steps: First, it determines whether large file download is enabled (using a flag to enable or disable large file download). If large file download is enabled, it downloads the files to be downloaded and large files based on the total list of files to be downloaded and the large file list. If large file download is disabled, it checks whether the number of files to be downloaded in the total list is less than half of the threshold (preset size threshold). If so, it enables large file download and downloads the files to be downloaded and large files based on the total list of files to be downloaded and the large file list. If not, it downloads the files to be downloaded based on the total list of files to be downloaded. Then, after the files are downloaded, the local collection (the list of downloaded files and large files on the machine) is updated. Finally, the Redis list is updated based on the local collection, and the local collection is deleted, ending this round of file download task.

[0066] It can be understood that the large file list is an empty list when the file is downloaded for the first time, and the file identifiers in the large file list change as the number of downloads increases. In subsequent file download tasks, the large files and the downloaded files are first screened according to the downloaded file list and the large file list to obtain a total list of files to be downloaded, and then the volume of each file in the total list of files to be downloaded is analyzed. The file identifiers of the files with a volume greater than a preset volume threshold are removed from the total list of files to be downloaded and added to the large file list. This large file screening method transfers the file identifiers of all large files from the files to be downloaded to the large file list while minimizing the cost of large file screening. In addition, it is worth noting that the embodiment of the present application realizes the flexibility of large file processing. It can start or stop large file download through API call, and can also dynamically adjust the large file download strategy according to the current number of files to be downloaded. If the number of files to be downloaded is too small, it will automatically start large file download to fully utilize the network bandwidth and improve resource utilization. If the number of files to be downloaded is too small, it will not automatically start large file download, which can improve the download efficiency of small files.

[0067] In an embodiment, the downloading files from the file server according to the total list of files to be downloaded and updating the downloaded file list comprises:

[0068] splitting the total list of files to be downloaded into at least two sub-lists of files to be downloaded;

[0069] allocating the at least two sub-lists of files to be downloaded to different file download devices respectively, wherein the sub-list of files to be downloaded corresponds to the allocated file download device;

[0070] updating the downloaded file list by downloading files from the file server according to the corresponding sub-list of files to be downloaded through the file download device.

[0071] Specifically, referring to Figure 2To accelerate the completion of file download, a xxl-job scheduling framework (a distributed task scheduling platform) is used to execute file download tasks running on multiple machines (such as machine 1, machine 2 and machine 3), each machine judges the file list that should be downloaded on the file server at this time, and then starts the download work. Among them, the total list to be downloaded is the sum of the file identifiers of all machines to be downloaded; the "index" and "total" values on each machine are obtained through the xxl-job framework, and each machine also stores the to-be-downloaded sub-list corresponding to the machine, and after the download is completed, the file identifiers of the completed file download cached by each machine are stored in the downloaded file list of the external storage Redis. For the file distribution strategy, a simple polling method is used here, that is, the index of the file to be downloaded in the total list to be downloaded is divided by the total number of machines total, and if the remainder is equal to the current machine index index, the file should be distributed to the current machine. Further, it is necessary to judge whether the to-be-downloaded file set of the current machine has reached the maximum file download number configured in the flow limiting, if the maximum number has been reached, the file is discarded, and the subsequent download task is triggered for processing, if the maximum file download number has not been reached, the size of the file is obtained and it is judged whether it reaches the maximum file volume size (preset volume threshold) configured in the flow limiting, if it does not reach, the file name is stored in the to-be-downloaded sub-list on the machine, if it exceeds the configured maximum volume, the file name is stored in the large file set on the machine.

[0072] In an embodiment, the processed file list further comprises a large file list;

[0073] The step of downloading files from the file server according to the corresponding to-be-downloaded sub-list by the file download device and updating the downloaded file list comprises:

[0074] Obtaining the file volume of each file corresponding to the file identifier in the to-be-downloaded sub-list;

[0075] Recording the file identifier of the file in the to-be-downloaded sub-list whose file volume is greater than the corresponding preset volume threshold in the large file list of the corresponding file download device, and deleting the file identifier of the file in the to-be-downloaded sub-list whose file volume is greater than the corresponding preset volume threshold; wherein each file download device is respectively configured with a corresponding preset volume threshold;

[0076] When the large file download permission of the file download device is closed and the total number of file identifiers in the corresponding to-be-downloaded sub-list is greater than or equal to the corresponding preset opening threshold, the file is downloaded from the file server according to the corresponding to-be-downloaded sub-list; wherein each file download device is respectively configured with a corresponding preset opening threshold;

[0077] when the large file downloading permission of the file downloading device is closed and the total number of file identifiers in the corresponding to-be-downloaded sub-list is less than the corresponding preset opening threshold, the large file downloading permission is opened;

[0078] when the large file downloading permission of the file downloading device is opened, files are downloaded from the file server according to the corresponding to-be-downloaded sub-list and the large file list;

[0079] file identifiers for which file downloading is completed are recorded in the downloaded file list, and the file identifiers for which file downloading is completed are deleted from the to-be-downloaded sub-list and the large file list.

[0080] Specifically, the processed file list includes the downloaded file list and the large file list. First, the total file list is subtracted from the downloaded file list and the large file list obtained from Redis to obtain a to-be-downloaded total list, and the to-be-downloaded total list is split into multiple to-be-downloaded sub-lists to be distributed to each machine to obtain a to-be-downloaded sub-list corresponding to each machine. Machine performance and network performance may differ greatly in different actual production scenarios, so flow limiting needs to be considered when distributing these file lists to each machine. For example, the file size threshold allowed on each machine can be increased in an environment with good network performance, and the related configuration can be relatively reduced when the network environment is poor, so as to meet file downloading in different scenarios as much as possible. It can be understood that each machine is configured with a preset size threshold, and the preset size thresholds of different machines can be the same or different, which are set according to actual conditions. After each machine obtains the corresponding to-be-downloaded sub-list, each machine judges the size of the file to which the file identifier recorded in the to-be-downloaded sub-list belongs, and transfers the file identifier of the file with a size greater than the preset size threshold configured on the machine from the to-be-downloaded sub-list to the large file list on the machine. Based on the file downloading strategy, file downloading is performed according to the to-be-downloaded sub-list and the large file list on the machine, and the related lists on the machine are updated. After the current file downloading task is completed, the downloaded list and the large file list in Redis are updated according to the related lists in the machine and the file downloading condition, and the large file list in the updated Redis is the sum of the large file lists on all machines.

[0081] It can be understood that in the file download process, for the processing of large files, a configuration "flag" is used to identify whether to start the download of large files, and the default is to start. The identification can be started or stopped through API interface call. In actual production, we can set the maximum number of files to a proper value and start the download of large files, so as to stabilize the network overhead, and at the same time, not to cause the problem of large delay of subsequent business logic data caused by the fact that large files are not downloaded all the time. Of course, the identification of large file download can also be first stopped, and the maximum number of file downloads is increased, and small files are downloaded in priority. When the number of files to be downloaded is less than the preset opening threshold (for example, half of the maximum number of files), the program will automatically start the download of large files, so as to fully utilize the network bandwidth.

[0082] In an embodiment, before the step of downloading files from the file server according to the total list of files to be downloaded and updating the list of downloaded files, the method further comprises:

[0083] When the total number of file identifiers in the total list of files to be downloaded is greater than a preset download threshold, at least one file identifier in the total list of files to be downloaded is deleted, so that the total number of file identifiers in the total list of files to be downloaded is less than or equal to the preset download threshold.

[0084] It can be understood that the performance of machines and the performance of networks may vary greatly in different scenarios in actual production, and therefore, when the file lists are distributed to each machine, flow limiting needs to be considered. For example, in an environment with good network performance, the number of files to be downloaded by each machine at a time can be increased, that is, the preset download threshold is increased, so as to increase the number of files downloaded in the current file download task. When the network is poor, the preset download threshold is decreased, and the number of files downloaded in the current file download task is decreased.

[0085] In an embodiment, the list of processed files further comprises an abnormal file list, and after the step of downloading files from the file server according to the total list of files to be downloaded, the method further comprises:

[0086] When an abnormal file occurs in the process of downloading files from the file server, the file identifier of the abnormal file is removed from the total list of files to be downloaded, and the file identifier of the abnormal file is recorded in the abnormal file list stored in the external storage device, so that the user repairs the files of the file server according to the file identifier of the abnormal file obtained through the preset API;

[0087] When it is monitored that the user completes the file repair operation of the file server, the file identifier of the repaired file is determined;

[0088] The API is called to the file identifier of the repaired file in the abnormal file list.

[0089] For example, referring to Figure 4 , all file identifiers of qualified files on the file server are acquired to form a total file list, the total file list is subtracted from the downloaded file list, the abnormal file list and the large file list stored in Redis, so as to obtain a total list to be downloaded, the total list to be downloaded is split to obtain a plurality of sub lists to be downloaded and is allocated to a plurality of machines (such as machine 1, machine 2 and machine 3), each machine downloads files from the file server according to the sub list to be downloaded and the large file list on the machine according to a predetermined download strategy, if no exception occurs in the download process, the file identifiers of the downloaded files are stored in the downloaded file list in Redis after the download is completed, and the sub list to be downloaded is emptied; if an exception occurs to a file in the download process of a machine, the file identifier of the abnormal file is stored in the abnormal file list on the machine, so that the abnormal file list on the machine is synchronized to the abnormal file list in Redis and the abnormal file list on the machine is emptied after the current file download task is completed, the file identifiers of normally downloaded files are still stored in the downloaded file list in Redis, and the sub list to be downloaded is emptied; if a file is too large in the download process, the file identifier of the file is stored in the large file list on the machine that downloads the file and is synchronized to the large file list in Redis, the file identifiers of normally downloaded files are stored in the downloaded file list in Redis, and the sub list to be downloaded is emptied.

[0090] Therefore, the file identifiers of files that have exceptions in the download process are uniformly placed in the abnormal file list in Redis, and an API interface of the abnormal file list in Redis is exposed to the outside, so that a user can query the file identifiers in the abnormal file list through the API interface, find the abnormal files from the file server according to the queried file identifiers, and repair the files. If the user repairs the abnormal files, the file identifiers of the repaired files can be deleted from the abnormal file list by calling the API interface of the abnormal file list in Redis, so that the files can be downloaded again when a subsequent scheduled task is triggered.

[0091] In an embodiment, the preset opening threshold is less than the preset download threshold, and the preset opening threshold and the preset download threshold are in a positive correlation relationship; the better the network state for downloading files from the file server, the larger the preset download threshold.

[0092] It can be understood that, in addition to limiting the file size, the file download is also limited in the dimension of the number of downloads, so as to as far as possible meet the actual business scenarios.

[0093] In an embodiment, the processed file list is stored in an external storage device. For example, the processed file list is stored in an external Redis.

[0094] Compared with the prior art, the file downloading method disclosed by the embodiment of the application first acquires a specified file total list on a file server in response to a file downloading instruction; wherein the file total list records file identifiers of at least one file; then, the file total list is subtracted from a pre-stored processed file list to obtain a total list to be downloaded; wherein the processed file list includes a downloaded file list; finally, files are downloaded from the file server according to the total list to be downloaded, and the downloaded file list is updated. As can be seen, the embodiment of the application can determine the total list to be downloaded by designing the file total list and the processed file list, so as to download files from the file server without designing an external system, thereby reducing the architecture complexity.

[0095] Referring to Figure 5 , Figure 5 is a structural schematic diagram of a file downloading device provided by the embodiment of the application, and the file downloading device 20 includes:

[0096] a file total list acquisition module 21, configured to acquire a specified file total list on a file server in response to a file downloading instruction; wherein the file total list records file identifiers of at least one file;

[0097] a total list to be downloaded acquisition module 22, configured to subtract a pre-stored processed file list from the file total list to obtain a total list to be downloaded; wherein the processed file list includes a downloaded file list;

[0098] a file downloading module 23, configured to download files from the file server according to the total list to be downloaded, and update the downloaded file list.

[0099] In an implementation, the processed file list further includes a large file list; and the file downloading module 23 is specifically configured to:

[0100] acquire file volumes of files to which the file identifiers in the total list to be downloaded belong;

[0101] record the file identifiers of files with file volumes greater than a preset volume threshold in the large file list, and delete the file identifiers of files with file volumes greater than the preset volume threshold from the total list to be downloaded;

[0102] when the large file downloading permission is closed and the total number of file identifiers in the total list to be downloaded is greater than or equal to a preset opening threshold, download files from the file server according to the total list to be downloaded;

[0103] when the large file downloading permission is closed and the total number of file identifiers in the total list to be downloaded is less than a preset opening threshold, opening the large file downloading permission;

[0104] when the large file downloading permission is opened, downloading files from the file server according to the total list to be downloaded and the large file list;

[0105] recording file identifiers of files whose downloading is completed in the downloaded file list, and deleting file identifiers of files whose downloading is completed from the total list to be downloaded and the large file list.

[0106] In an embodiment, the file downloading module 23 is specifically configured to:

[0107] splitting the total list to be downloaded into at least two sub lists to be downloaded;

[0108] allocating the at least two sub lists to be downloaded to different file downloading devices respectively; wherein the sub list to be downloaded corresponds to the allocated file downloading device;

[0109] updating the downloaded file list by downloading files from the file server according to the corresponding sub list to be downloaded by the file downloading device.

[0110] In an embodiment, the processed file list further comprises a large file list; and the file downloading module 23 is specifically configured to:

[0111] acquiring file volumes of files to which each file identifier in the sub list to be downloaded belongs;

[0112] recording file identifiers of files in the sub list to be downloaded whose file volumes are greater than a corresponding preset volume threshold in a large file list of the corresponding file downloading device, and deleting file identifiers of files in the sub list to be downloaded whose file volumes are greater than the corresponding preset volume threshold; wherein each file downloading device is respectively configured with a corresponding preset volume threshold;

[0113] when the large file downloading permission of the file downloading device is closed and the total number of file identifiers in the corresponding sub list to be downloaded is greater than or equal to a corresponding preset opening threshold, downloading files from the file server according to the corresponding sub list to be downloaded; wherein each file downloading device is respectively configured with a corresponding preset opening threshold;

[0114] when the large file downloading permission of the file downloading device is closed and the total number of file identifiers in the corresponding sub list to be downloaded is less than the corresponding preset opening threshold, opening the large file downloading permission;

[0115] When the large file downloading permission of the file downloading device is turned on, downloading files from the file server according to the corresponding sub-list and the large file list;

[0116] Recording the file identifier of the file whose downloading is completed in the downloaded file list, and deleting the file identifier of the file whose downloading is completed from the sub-list and the large file list.

[0117] In an embodiment, the file downloading module 23 is further configured to:

[0118] Before the step of downloading files from the file server according to the total list and updating the downloaded file list, when the total number of file identifiers in the total list is greater than a preset downloading threshold, deleting at least one file identifier in the total list so that the total number of file identifiers in the total list is less than or equal to the preset downloading threshold.

[0119] In an embodiment, the processed file list further comprises an abnormal file list, and the step of downloading files from the file server according to the total list further comprises:

[0120] When an abnormal file occurs during the process of downloading files from the file server, removing the file identifier of the abnormal file from the total list, and recording the file identifier of the abnormal file in the abnormal file list stored in the external storage device, so that the user repairs the files of the file server according to the file identifier of the abnormal file obtained through the preset API;

[0121] When it is monitored that the user completes the file repair operation of the file server, determining the file identifier of the repaired file;

[0122] Calling the API to the file identifier of the repaired file in the abnormal file list.

[0123] In an embodiment, the preset turning-on threshold is less than the preset downloading threshold, and the preset turning-on threshold and the preset downloading threshold are in a positive correlation relationship; the better the network state for downloading files from the file server, the greater the preset downloading threshold.

[0124] In an embodiment, the processed file list is stored in a database.

[0125] It is worth noting that the working process of the specific file downloading device can refer to the working process of the file downloading method described in the above embodiments, which will not be repeated here.

[0126] Compared with the prior art, the file downloading device disclosed by the embodiment of the application firstly acquires a total list of files on a file server in response to a file downloading instruction, wherein the total list of files records file identifiers of at least one file; then, the total list of files is subtracted from a pre-stored processed file list to obtain a total list of files to be downloaded, wherein the processed file list includes a downloaded file list; finally, files are downloaded from the file server according to the total list of files to be downloaded, and the downloaded file list is updated. Therefore, the embodiment of the application can determine the total list of files to be downloaded by designing the total list of files and the processed file list, so as to download files from the file server without designing an external system, thereby reducing the complexity of the architecture.

[0127] Referring to Figure 6 , Figure 6 is a structural schematic diagram of a file downloading device provided by the embodiment of the application. The file downloading device 30 includes a processor 31, a memory 32, and a computer program stored in the memory 32 and configured to be executed by the processor 31. When the processor 31 executes the computer program, the steps in the above file downloading method embodiments are implemented, for example, the steps S11-S13 described in the above embodiment. Alternatively, when the processor 31 executes the computer program, the functions of the modules in the above device embodiments are implemented. Figure 1

[0128] For example, the computer program can be divided into one or more modules, which are stored in the memory 32 and executed by the processor 31 to complete the application. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the file downloading device. For example, the computer program can be divided into multiple modules, and the specific functions of the modules are as follows:

[0129] The total list of files acquisition module 21 is configured to acquire a total list of files on a file server in response to a file downloading instruction, wherein the total list of files records file identifiers of at least one file.

[0130] The total list of files to be downloaded acquisition module 22 is configured to subtract a pre-stored processed file list from the total list of files to obtain a total list of files to be downloaded, wherein the processed file list includes a downloaded file list.

[0131] The file downloading module 23 is configured to download files from the file server according to the total list of files to be downloaded, and update the downloaded file list.

[0132] ​The specific working process of each module can refer to the working process of the file downloading apparatus described in the above embodiments, which will not be repeated here.

[0133] The file downloading apparatus can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The file downloading apparatus can include, but is not limited to, a processor 31 and a memory 32. Those skilled in the art can understand that the file downloading apparatus can further include an input / output device, a network access device, a bus, and the like.

[0134] The processor 31 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The processor 31 is the control center of the file downloading apparatus, and connects all parts of the file downloading apparatus through various interfaces and lines.

[0135] The memory 32 can be used to store computer programs and / or modules. The processor 31 realizes various functions of the file downloading apparatus by running or executing the computer programs and / or modules stored in the memory 32, and calling data stored in the memory 32. The memory 32 can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the file downloading apparatus, etc. In addition, the memory 32 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory device.

[0136] If the module integrated with the file downloading device is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by the processor 31, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0137] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.

Claims

1. A file download method characterized by, The method comprises the following steps: in response to a file download instruction, obtaining a total list of files on a file server; wherein the total list of files records file identifiers of at least one file; differencing the total list of files from a pre-stored processed file list to obtain a total list of files to be downloaded; wherein the processed file list comprises a downloaded file list; downloading files from the file server according to the total list of files to be downloaded and updating the downloaded file list; the processed file list further comprises a large file list; the step of downloading files from the file server according to the total list of files to be downloaded and updating the downloaded file list comprises: obtaining file sizes of files to which the file identifiers in the total list of files to be downloaded belong; recording file identifiers of files with file sizes greater than a preset size threshold in the large file list and deleting the file identifiers of the files with file sizes greater than the preset size threshold from the total list of files to be downloaded; downloading files in the total list of files to be downloaded from the file server according to a state of a large file download permission or downloading files in the total list of files to be downloaded and the large file list from the file server after the large file download permission is opened, in combination with a relationship between a total number of file identifiers in the total list of files to be downloaded and a preset opening threshold; recording file identifiers of files for which file downloading is completed in the downloaded file list and deleting the file identifiers of the files for which file downloading is completed from the total list of files to be downloaded and the large file list.

2. The file download method of claim 1, wherein, the step of downloading files in the total list of files to be downloaded from the file server according to a state of a large file download permission or downloading files in the total list of files to be downloaded and the large file list from the file server after the large file download permission is opened, in combination with a relationship between a total number of file identifiers in the total list of files to be downloaded and a preset opening threshold comprises: when the large file download permission is closed and the total number of file identifiers in the total list of files to be downloaded is greater than or equal to the preset opening threshold, downloading files from the file server according to the total list of files to be downloaded; when the large file download permission is closed and the total number of file identifiers in the total list of files to be downloaded is less than the preset opening threshold, opening the large file download permission; when the large file download permission is opened, downloading files from the file server according to the total list of files to be downloaded and the large file list.

3. The file download method of claim 1, wherein, the step of downloading files from the file server according to the total list of files to be downloaded and updating the downloaded file list comprises: splitting the total list of files to be downloaded into at least two sub lists of files to be downloaded; allocating the at least two sub lists of files to be downloaded to different file downloading devices respectively; wherein the sub lists of files to be downloaded correspond to the allocated file downloading devices; downloading files from the file server according to the corresponding sub lists of files to be downloaded by the file downloading devices and updating the downloaded file list.

4. The file download method of claim 3, wherein, the processed file list further comprises a large file list; the step of downloading files from the file server according to the corresponding sub lists of files to be downloaded by the file downloading devices and updating the downloaded file list comprises: obtaining file sizes of files to which the file identifiers in the sub lists of files to be downloaded belong; record file identifiers of files in the to-be-downloaded sub-lists whose file volumes are greater than corresponding preset volume thresholds in a large file list of a corresponding file downloading device, and delete the file identifiers of the files in the to-be-downloaded sub-lists whose file volumes are greater than corresponding preset volume thresholds; wherein each of the file downloading devices is respectively configured with a corresponding preset volume threshold; when the large file downloading permission of the file downloading device is closed and the total number of file identifiers in a corresponding to-be-downloaded sub-list is greater than or equal to a corresponding preset opening threshold, download files from the file server according to the corresponding to-be-downloaded sub-list; wherein each of the file downloading devices is respectively configured with a corresponding preset opening threshold; when the large file downloading permission of the file downloading device is closed and the total number of file identifiers in a corresponding to-be-downloaded sub-list is less than a corresponding preset opening threshold, open the large file downloading permission; when the large file downloading permission of the file downloading device is opened, download files from the file server according to the corresponding to-be-downloaded sub-list and the large file list; record file identifiers of files for which file downloading is completed in the downloaded file list, and delete the file identifiers of the files for which file downloading is completed from the to-be-downloaded sub-list and the large file list.

5. The file download method of claim 1, wherein, The method further comprises the following steps before the step of downloading files from the file server according to the to-be-downloaded total list and updating the downloaded file list: when the total number of file identifiers in the to-be-downloaded total list is greater than a preset downloading threshold, delete at least one file identifier in the to-be-downloaded total list, so that the total number of file identifiers in the to-be-downloaded total list is less than or equal to the preset downloading threshold.

6. The file download method of claim 1, wherein, The processed file list further comprises an abnormal file list, and the method further comprises the following steps: when an abnormal file situation occurs in the process of downloading files from the file server, remove a file identifier of an abnormal file from the to-be-downloaded total list, and record the file identifier of the abnormal file in the abnormal file list stored in an external storage device, so that a user repairs files of the file server according to the file identifier of the abnormal file obtained through a preset API; when it is monitored that the user completes a file repair operation of the file server, determine a file identifier of a repaired file; call the API to the file identifier of the repaired file in the abnormal file list.

7. The file download method of claim 2 or 4 or 5, wherein The preset opening threshold is less than the preset downloading threshold, and the preset opening threshold and the preset downloading threshold are in a positive correlation relationship; The better the network state for downloading files from the file server, the greater the preset downloading threshold.

8. The file download method of claim 1, wherein, The processed file list is stored in a database.

9. A file download apparatus characterized by comprising: The method comprises the following steps: a file total list acquisition module is configured to acquire a specified file total list on a file server in response to a file downloading instruction; wherein the file total list records file identifiers of at least one file; a to-be-downloaded list acquisition module is configured to subtract the file total list from a pre-stored processed file list to obtain a to-be-downloaded total list; wherein the processed file list comprises a downloaded file list. The file downloading module is configured to download files from the file server according to the total list of files to be downloaded, and update the list of downloaded files; The file downloading module is specifically configured to: acquire file sizes of files to which the file identifiers in the total list of files to be downloaded belong; and the processed file list further comprises a large file list; record the file identifiers of files with file sizes greater than a preset size threshold in the large file list, and delete the file identifiers of files with file sizes greater than the preset size threshold from the total list of files to be downloaded; download files in the total list of files to be downloaded from the file server according to a state of a large file downloading permission, in combination with a relationship between a total number of file identifiers in the total list of files to be downloaded and a preset opening threshold, or download files in the total list of files to be downloaded and the large file list from the file server after the large file downloading permission is opened; record the file identifiers of files for which file downloading is completed in the list of downloaded files, and delete the file identifiers of files for which file downloading is completed from the total list of files to be downloaded and the large file list.

10. A file download apparatus characterized by comprising: The computer readable storage medium comprises a stored computer program, wherein the computer readable storage medium controls a device in which the computer readable storage medium is located to perform the file downloading method according to any one of claims 1 to 8 when the computer program is running.

11. A computer readable storage medium, characterized in that, The computer readable storage medium comprises a stored computer program, wherein the computer readable storage medium controls a device in which the computer readable storage medium is located to perform the file downloading method according to any one of claims 1 to 8 when the computer program is running.

Images