Batch file downloading method and device, electronic equipment and storage medium

Abstract

The invention discloses a batch file downloading method and device, electronic equipment and a storage medium, and relates to the field of file transmission, the method comprises the following steps: in response to obtaining a batch downloading request submitted by a target user, reading each to-be-downloaded file according to the batch downloading request; writing the currently downloaded file into a file package, and sending the packaging progress of the batch downloading request to a target user; uploading the downloaded file compression package to an object storage service, and generating a downloading address of the file compression package through the object storage service; and sending the downloading address to the target user, so that the target user obtains the batch files through the downloading address. According to the technical scheme provided by the embodiment of the invention, the quick downloading of batch files is realized, the occupation amount of storage resources of the server is reduced, the phenomena of memory overflow or insufficient disk capacity and the like are avoided, the transparent processing of the packaging progress is realized, and a user is ensured to know the file processing progress of the server in time.

Description

Technical Field

[0001] The present invention relates to the field of file transmission, and in particular to a batch file downloading method, device, electronic equipment and storage medium. Background Art

[0002] With the continuous development of computer technology, B / S architecture (Browser / Server) has become the most widely used software system construction technology.

[0003] Under the B / S architecture, there are a large number of electronic files inside each business system and on the external Internet, and users often have the need to download multiple files in batches. In the prior art, when a user initiates a batch download of multiple files through a browser, the server usually adopts a synchronous processing method, traversing the file request list, reading the files one by one and compressing them, and finally returning the compressed package to the user end.

[0004] However, the existing file downloading method not only takes up a large amount of memory and disk space on the server, which easily leads to memory overflow or insufficient disk capacity, but also has the problem of insufficient transparency, and users cannot know the file processing progress of the server in a timely manner. Summary of the invention

[0005] The present invention provides a batch file downloading method, device, electronic equipment and storage medium to solve the problems of insufficient storage space of a server and fuzzy file processing progress.

[0006] According to one aspect of the present invention, a batch file downloading method is provided, comprising:

[0007] In response to obtaining a batch download request submitted by a target user, reading each file to be downloaded according to the batch download request;

[0008] Writing the currently downloaded file into a file compression package, and sending the packaging progress of the batch download request to the target user;

[0009] Upload the downloaded compressed file package to the object storage service, and generate a download address for the compressed file package through the object storage service;

[0010] The download address is sent to the target user, so that the target user obtains batch files through the download address and deletes the file compression package stored locally.

[0011] The method responds to obtaining a batch download request submitted by a target user and reading each file to be downloaded according to the batch download request, including: responding to obtaining a batch download request submitted by a target user, generating a task code according to the batch download request, storing the task code and the batch download request in a cache, and storing the task code in a message queue; if the task code is obtained from the message queue through a listening process, the batch download request is obtained from the cache based on the task code through the listening process, so that the listening process reads each file to be downloaded according to the batch download request.

[0012] The sending the packaging progress of the batch download request to the target user includes: updating the packaging progress of the batch download request according to the number of download files written into the file compression package and the ratio between the predicted download time and the predicted upload time.

[0013] Before updating the packaging progress of the batch download request according to the number of download files written into the file compression package and the proportional relationship between the predicted download time and the predicted upload time, it includes: obtaining the proportional relationship between the predicted download time and the predicted upload time according to the file format of each file to be downloaded in the batch download request; wherein the file format includes text files, video files and picture files.

[0014] Each file to be downloaded in the batch download request includes at least one first type file and at least one second type file; wherein, the first type file includes a file identifier, and the second type file includes a file address; reading each file to be downloaded according to the batch download request includes: according to the file type of each file to be downloaded, allocating each file to be downloaded to multiple download processes so that each download process maintains a balance of file types, and reading each file to be downloaded through the multiple download processes.

[0015] The first type of file and the second type of file both include a file size identifier; the method of allocating each file to be downloaded to multiple download processes according to the file type of each file to be downloaded so that each download process maintains a balanced file type, including: obtaining a matching weight value according to the file type of each file to be downloaded, and obtaining the download time of each file to be downloaded according to the weight value and the file size identifier of each file to be downloaded; wherein the weight value of the first type of file is less than the weight value of the second type of file; and allocating each file to be downloaded to multiple download processes according to the download time of each file to be downloaded so that each download process maintains a balanced download time.

[0016] According to another aspect of the present invention, there is provided a batch file downloading device, comprising:

[0017] A batch download acquisition module, configured to respond to a batch download request submitted by a target user and read each file to be downloaded according to the batch download request;

[0018] A packaging progress sending module, used to write the currently downloaded file into a file compression package, and send the packaging progress of the batch download request to the target user;

[0019] The object storage execution module is used to upload the downloaded file compression package to the object storage service, and generate a download address of the file compression package through the object storage service;

[0020] The download address sending module is used to send the download address to the target user, so that the target user can obtain batch files through the download address and delete the file compression package stored locally.

[0021] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0022] at least one processor; and

[0023] a memory communicatively connected to the at least one processor; wherein,

[0024] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the batch file downloading method described in any embodiment of the present invention.

[0025] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the batch file downloading method described in any embodiment of the present invention when executed.

[0026] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the batch file downloading method according to any embodiment of the present invention is implemented.

[0027] The technical solution of the embodiment of the present invention responds to the batch download request submitted by the target user, reads each file to be downloaded according to the batch download request; writes the current download file into a file compression package, and sends the packaging progress of the batch download request to the target user; uploads the downloaded file compression package to the object storage service, and generates the download address of the file compression package through the object storage service; sends the download address to the target user, so that the target user can obtain the batch file through the download address, and deletes the locally stored file compression package. In this way, not only the fast download of batch files is achieved, but also the occupation of server storage resources is reduced, memory overflow or insufficient disk capacity is avoided, and the transparent processing of the packaging progress is achieved, ensuring that the user is informed of the file processing progress of the server in a timely manner.

[0028] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 is a flow chart of a batch file downloading method provided according to the first embodiment of the present invention;

[0031] Figure 2 is a flow chart of another batch file downloading method provided according to the second embodiment of the present invention;

[0032] Figure 3 is a flow chart of another batch file downloading method provided according to Embodiment 3 of the present invention;

[0033] Figure 4 is a structural diagram of a batch file downloading device provided according to a fourth embodiment of the present invention;

[0034] Figure 5 It is a structural schematic diagram of an electronic device for implementing the batch file downloading method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] Embodiment 1

[0038] Figure 1 This is a flowchart of a batch file downloading method provided in the first embodiment of the present invention. This embodiment is applicable to the case where the packaging progress is updated to the user and the file compression package is uploaded based on the object storage service. The method can be executed by the batch file downloading device in any embodiment of the present invention. The batch file downloading device can be implemented in the form of hardware and / or software. The batch file downloading device can be configured in a server, for example, a Web (World Wide Web) server, such as Figure 1 As shown, the method includes:

[0039] S101 . In response to obtaining a batch download request submitted by a target user, reading each file to be downloaded according to the batch download request.

[0040] After selecting multiple files to be downloaded, the user clicks a batch download button to combine the multiple files to be downloaded into a batch download request, and submits the request to the server; after the server parses the batch download request, it obtains relevant information of each file therein; wherein, the files in the batch download request may exist in the form of file identifiers, which reflects that the file is located in the system where the current server is located, or in a designated system that is connected to the current server in communication, and the unique matching file can be found in the relevant system through the file identifier.

[0041] In addition, the files in the batch download request may also exist in the form of a file address, which reflects that the file is located at any storage location on the Internet, and the file can be found on the Internet through the file address; in addition, the files in the batch download request may also be displayed in the form of detailed information, and the relevant information may include detailed information such as file size, storage location and file format (for example, text files, video files, image files, etc.). The file may be located in the system where the current server is located, or in a designated system that is connected to the current server in communication, or in any storage location on the Internet.

[0042] After receiving the batch download request, the server can read the relevant information of each file in sequence according to the file sequence to perform the file download operation, or it can start different download processes. For example, according to the different file storage locations, each file to be downloaded is assigned to a different download process, so that the file download in the current system or the specified system can be completed through one download process (that is, internal download), and the file download in other storage locations on the Internet can be completed through another download process (that is, external download).

[0043] S102: Write the currently downloaded file into a file compression package, and send the packaging progress of the batch download request to the target user.

[0044] The server writes the read content into a file compression package (for example, a file compression package in ZIP format) instantly each time it reads a file based on the file identifier or file address. At the same time, each time a file is written, the packaging progress is updated and the updated result is sent to the target user. For example, if at least one file to be downloaded in the batch download request does not include a file size identifier, the packaging progress is updated based on the number of files written into the file compression package, that is, the ratio of the number of files written into the file compression package to the total number of files in the batch download request is used as the packaging progress.

[0045] If each file in the batch download request is configured with a file size identifier, the packaging progress can be updated according to the amount of file data written into the file compression package, that is, the ratio of the amount of data written into the file compression package to the total amount of data in the batch download request is used as the packaging progress. This achieves the transparency of the progress of the file packaging process and ensures that the user is informed of the server's file processing progress in a timely manner. Compared with the traditional technical solution, the user can only obtain the file download progress through the browser and other download tools, but cannot obtain the server's packaging progress. The embodiment of the present invention reflects the specific packaging progress of the server in the file packaging process. In particular, the issuance of the packaging progress can be issued after obtaining the progress query request periodically issued by the user end, or it can be actively sent to the target user after each file is written.

[0046] S103: Upload the downloaded compressed file package to an object storage service, and generate a download address of the compressed file package through the object storage service.

[0047] When the server processes multiple batch download requests at the same time, it will occupy a large amount of memory and disk space, resulting in memory overflow or insufficient disk capacity. Therefore, after the server completes the task of packaging the file compression package, that is, the task of downloading all files in the batch download request, it uploads the file compression package to the object storage service. Among them, the object storage service (Object Storage Service, OSS) is an object-based data storage method, which stores data as objects, each object contains the data itself, metadata and a globally unique identifier.

[0048] In an embodiment of the present invention, the object storage service includes a cloud storage service and a local storage service, for example, a storage server located locally on the current server. After obtaining a compressed file package, the object storage service generates a storage address of the compressed file package within itself (that is, within the object storage service). The storage address is the address information of the compressed file package that can be directly obtained through the object storage service, that is, the download address of the compressed file package.

[0049] S104: Send the download address to the target user, so that the target user can obtain batch files through the download address and delete the file compression package stored locally.

[0050] The server displays the download address to the target user and deletes the locally stored file compression package, thereby reducing the usage of server storage resources and avoiding memory overflow or insufficient disk capacity. After obtaining the download address, the target user can communicate with the object storage service and use the download address to obtain the file compression package stored in the object storage service.

[0051] Optionally, in an embodiment of the present invention, sending the packaging progress of the batch download request to the target user includes: updating the packaging progress of the batch download request based on the number of downloaded files written into the file compression package and the ratio of the predicted download time to the predicted upload time.

[0052] Specifically, after packaging is completed, the file compression package needs to be uploaded to the object storage service. Under the same data volume, depending on the communication bandwidth, the download time of the server to obtain the file compression package is usually longer than the upload time of sending the file compression package to the object storage service. At the same time, after the server downloads the file, it further compresses the file in the process of writing the file compression package, resulting in the data volume of the compressed file package being actually smaller than the total file data volume when it is downloaded, that is, the file data volume downloaded by the server is greater than the data volume of the file compression package it uploads.

[0053] Therefore, under the premise that the download time of the server itself is greater than the upload time, the compression operation during file writing further increases the ratio of the download time to the upload time. Therefore, the proportional relationship between the predicted download time and the predicted upload time can be pre-configured. The configuration value is greater than the inverse of the proportional relationship between the download bandwidth and the upload bandwidth. Under the premise that the file size of each file to be downloaded cannot be obtained, the number of downloaded files that have been written into the file compression package is used as the current writing progress, that is, as the packaging progress during the download process.

[0054] However, while the server is uploading the compressed file package to the object storage service, the target user is still in a waiting state. For the target user, it is also regarded as the entire process of file packaging. The user will not distinguish whether it is in the file packaging process or the upload process at the current moment, and will regard the server's file packaging behavior as incomplete. Therefore, the packaging progress displayed to the user in the implementation of the present invention actually refers to the completion degree of the entire process of downloading and uploading the compressed file package at the current moment.

[0055] Based on this, after taking the ratio of the number of downloaded files written into the file compression package to the total number of files in the file compression package as the packaging progress in the download process, and then according to the proportional relationship between the predicted download time and the predicted upload time, the ratio of the download time to the sum of the download time and the predicted upload time is used as the progress weight. Finally, the product of the packaging progress in the download process and the progress weight is taken as the final packaging progress.

[0056] For example, the ratio of download bandwidth to upload bandwidth is 1:2, so the corresponding download time and upload time of the two can be configured as 2:1, but as described in the above technical solution, due to the compression operation when writing the file, the ratio of download time to upload time is further increased, so the ratio of predicted download time to predicted upload time can be configured as 3:1; assuming that there are a total of 10 files to be downloaded, and 4 files to be downloaded have been written into the file compression package, the packaging progress in the download process is 40% at this time; then, the ratio of download time to the sum of download time and predicted upload time is 3:4, that is, 75%, is used as the progress weight, and the product of 40% and 75%, that is, 30%, is used as the final packaging progress, thereby ensuring that the accurate packaging progress is displayed according to the actual waiting time of the target user, and improving the accuracy of the packaging progress display result.

[0057] Optionally, in an embodiment of the present invention, before updating the packaging progress of the batch download request based on the number of download files written into the file compression package and the proportional relationship between the predicted download time and the predicted upload time, it includes: obtaining the proportional relationship between the predicted download time and the predicted upload time based on the file format of each file to be downloaded in the batch download request; wherein the file format includes text files, video files and picture files.

[0058] Specifically, files of different file formats have different compression degrees when they are written into a file compression package. For video files, their redundancy is relatively high, so when they are written into a file compression package, data compression can be performed to a greater extent. The video files themselves have been compressed using a relatively efficient compression algorithm, so when they are written into a file compression package, the data compression degree is relatively low. Image files have been compressed using a certain compression algorithm, so when they are written into a file compression package, the data compression degree is moderate.

[0059] If the data size of each file to be downloaded can be known, the downloaded data size will be used as the basis for calculation; if the data size of all files cannot be known, the number of files already downloaded will be used as the basis for calculation; obtain the number ratio (or data size ratio) of each file format in all downloaded files; the higher the ratio of text files, the greater the difference between the downloaded data size and the uploaded data size, that is, the greater the numerical value of the proportional relationship between the predicted download time and the predicted upload time; and the higher the ratio of video files, the smaller the difference between the downloaded data size and the uploaded data size, that is, the smaller the numerical value of the proportional relationship between the predicted download time and the predicted upload time.

[0060] Based on this, the ratio of the predicted download time to the predicted upload time can be obtained according to the numerical ranges of the text file ratio, video file ratio and image file ratio, according to preset calculation rules or preset mapping tables. This realizes the calculation and acquisition of the ratio of the predicted download time to the predicted upload time based on the file format, improves the accuracy of the packaging progress display results, and enhances the user experience.

[0061] The technical solution of the embodiment of the present invention responds to the batch download request submitted by the target user, reads each file to be downloaded according to the batch download request; writes the current download file into a file compression package, and sends the packaging progress of the batch download request to the target user; uploads the downloaded file compression package to the object storage service, and generates the download address of the file compression package through the object storage service; sends the download address to the target user, so that the target user can obtain the batch file through the download address, and deletes the locally stored file compression package. In this way, not only the fast download of batch files is achieved, but also the occupation of server storage resources is reduced, memory overflow or insufficient disk capacity is avoided, and the transparent processing of the packaging progress is achieved, ensuring that the user is informed of the file processing progress of the server in a timely manner.

[0062] Embodiment 2

[0063] Figure 2 Flow chart of a batch file downloading method provided by Embodiment 2 of the present invention. The relationship between this embodiment and the above-mentioned embodiment is that each batch download request is managed by a message queue, such as Figure 2 As shown, the method includes:

[0064] S201. In response to obtaining a batch download request submitted by a target user, generating a task code according to the batch download request, storing the task code and the batch download request in a cache, and storing the task code in a message queue.

[0065] When the server receives a large number of batch file download requests, it is often unable to flexibly control the number of concurrent tasks, resulting in excessive resource utilization of the server. Therefore, a message queue mechanism is introduced. After receiving the batch download request, a one-to-one matching task code is first generated. Among them, the task codes can be sorted in ascending or descending order based on the acquisition order, or based on the basic information of each file in the batch download request and the receiving time, the task code of the batch download request is generated through a digest algorithm such as the MD5 digest algorithm and the SHA-1 digest algorithm.

[0066] The server stores the task code and the batch download request in a cache (e.g., Redis), and pushes the task code to a message queue (e.g., a RabbitMQ message queue or a Kafka message queue); wherein the message queue may be configured locally on the server, or may be configured in other local devices that are connected to the current server for communication; in addition, the server may also send the task code to the target user, and store the packaging progress in the cache, so as to send an updated result of the packaging progress to the target user according to the mapping relationship between the task code and the packaging progress and the task code sent by the target user.

[0067] S202: If the task code is obtained from the message queue through the listening process, the batch download request is obtained from the cache based on the task code through the listening process, so that the listening process reads each file to be downloaded according to the batch download request.

[0068] The server starts one or more listening processes to listen to the message queue according to its own available resources (for example, available memory resources and disk resources), and obtains the current task code to be executed according to the queue order or priority; after obtaining the task code to be executed, the listening process reads the batch file request matching the task code from the cache, and then reads the relevant information of each file in turn according to the file sequence, so as to perform the file download operation.

[0069] S203: Write the currently downloaded file into a file compression package, and send the packaging progress of the batch download request to the target user.

[0070] S204: Upload the downloaded compressed file package to an object storage service, and generate a download address for the compressed file package through the object storage service.

[0071] S205: Send the download address to the target user, so that the target user obtains batch files through the download address, and deletes the file compression package stored locally.

[0072] The technical solution of the embodiment of the present invention, in response to obtaining a batch download request submitted by a target user, generates a task code according to the batch download request, stores the task code and the batch download request in a cache, and stores the task code in a message queue; if the task code is obtained from the message queue through a listening process, the batch download request is obtained from the cache based on the task code through the listening process, so that the listening process reads each file to be downloaded according to the batch download request. In this way, asynchronous processing of batch download requests is realized through the message queue, and the hardware resources of the server are reasonably utilized. At the same time, it avoids the situation that the batch download request response timeout is caused by the packaging process taking too long, and the target user cannot be informed of the file processing progress in time.

[0073] Embodiment 3

[0074] Figure 3 Flow chart of a batch file downloading method provided in Embodiment 3 of the present invention. The relationship between this embodiment and the above embodiments is that each file to be downloaded is assigned to a different download process based on the file type, such as Figure 3 As shown, the method includes:

[0075] S301. In response to obtaining a batch download request submitted by a target user, each file to be downloaded is allocated to multiple download processes according to the file type of each file to be downloaded, so that each download process maintains a balance of file types, and each file to be downloaded is read through the multiple download processes; wherein each file to be downloaded in the batch download request includes at least one first type file and at least one second type file; the first type file includes a file identifier, and the second type file includes a file address.

[0076] A batch download request may contain both first-type files and second-type files. The first-type files are displayed in the form of file identifiers and do not include file addresses, while the second-type files exist in the form of file addresses and do not include file identifiers. Files with file identifiers are usually stored in local disks or network disks (for example, NAS network disks) and have faster download speeds. Files with file addresses are usually files accessed based on the HTTP protocol and have slower download speeds.

[0077] When partial download tasks of the same batch download request are executed separately by multiple download processes, the download time is actually determined by the last download process that completes the download task. This requires ensuring that the download time of each download process is as close as possible to increase the file download speed. Compared with the traditional allocation method, different types of files or files from different sources are allocated to different download processes. In an embodiment of the present invention, each type of file is evenly allocated to different download processes so that the number of each type of file in each download process is equal or close (in fact, the difference in number is less than or equal to one).

[0078] For example, when two download processes are configured based on the remaining available resources of the server, if the batch download request includes six first-type files and six second-type files, three first-type files and three second-type files are assigned to one of the download processes, and the remaining three first-type files and the remaining three second-type files are assigned to the other download process, so as to ensure the balance of file types in different download processes. Thus, files of one type are evenly distributed to multiple download processes, ensuring that the number of files of the same type is the same between each download process, or the difference in number is within one, thereby improving the efficiency of obtaining file compression packages.

[0079] Optionally, in an embodiment of the present invention, both the first type of file and the second type of file include a file size identifier; and the allocating each file to be downloaded to multiple download processes according to the file type of each file to be downloaded so that each download process maintains a file type balance, including: obtaining a matching weight value according to the file type of each file to be downloaded, and obtaining the download time of each file to be downloaded according to the weight value and the file size identifier of each file to be downloaded; wherein the weight value of the first type of file is smaller than the weight value of the second type of file; and allocating each file to be downloaded to multiple download processes according to the download time of each file to be downloaded so that each download process maintains a balanced download time.

[0080] Specifically, if each file to be downloaded in the batch download request includes a file size identifier, then the data volume of each file can be directly determined by the file size identifier. As described in the above technical solution, the download speed of the first type of file is faster, and the download speed of the second type of file is slower. At this time, a matching weight value can be obtained according to the file type, wherein the weight value of the first type of file is less than the weight value of the second type of file; the product of the file data volume of each file to be downloaded and the weight value is used as the download time of each file to be downloaded; according to the download time of each file to be downloaded, each file to be downloaded is allocated to multiple download processes to ensure that the total download time of each download process is the same or relatively close, so as to keep the download time balanced, thereby further improving the efficiency of obtaining the file compression package.

[0081] S302: Write the currently downloaded file into a file compression package, and send the packaging progress of the batch download request to the target user.

[0082] S303: Upload the downloaded compressed file package to an object storage service, and generate a download address for the compressed file package through the object storage service.

[0083] S304: Send the download address to the target user, so that the target user obtains batch files through the download address, and deletes the file compression package stored locally.

[0084] The technical solution of the embodiment of the present invention, in response to obtaining a batch download request submitted by a target user, distributes each file to be downloaded to multiple download processes according to the file type of each file to be downloaded, so that the file types of each download process are balanced, and each file to be downloaded is read through multiple download processes, thereby further improving the efficiency of obtaining file compression packages.

[0085] Embodiment 4

[0086] Figure 4This is a structural block diagram of a batch file downloading device provided by Embodiment 4 of the present invention, which specifically includes:

[0087] The batch download acquisition module 401 is used to respond to the batch download request submitted by the target user and read each file to be downloaded according to the batch download request;

[0088] The packaging progress sending module 402 is used to write the currently downloaded file into a file compression package and send the packaging progress of the batch download request to the target user;

[0089] The object storage execution module 403 is used to upload the downloaded file compression package to the object storage service, and generate a download address of the file compression package through the object storage service;

[0090] The download address sending module 404 is used to send the download address to the target user, so that the target user can obtain batch files through the download address and delete the file compression package stored locally.

[0091] The technical solution of the embodiment of the present invention responds to the batch download request submitted by the target user, reads each file to be downloaded according to the batch download request; writes the current download file into a file compression package, and sends the packaging progress of the batch download request to the target user; uploads the downloaded file compression package to the object storage service, and generates the download address of the file compression package through the object storage service; sends the download address to the target user, so that the target user can obtain the batch file through the download address, and deletes the locally stored file compression package. In this way, not only the fast download of batch files is achieved, but also the occupation of server storage resources is reduced, memory overflow or insufficient disk capacity is avoided, and the transparent processing of the packaging progress is achieved, ensuring that the user is informed of the file processing progress of the server in a timely manner.

[0092] Optionally, the batch download acquisition module 401 is specifically used to respond to a batch download request submitted by a target user, generate a task code according to the batch download request, store the task code and the batch download request in a cache, and store the task code in a message queue; if the task code is obtained from the message queue through a listening process, the batch download request is obtained from the cache based on the task code through the listening process, so that the listening process reads each file to be downloaded according to the batch download request.

[0093] Optionally, the packaging progress sending module 402 is specifically used to update the packaging progress of the batch download request according to the number of download files written into the file compression package and the ratio between the predicted download time and the predicted upload time.

[0094] Optionally, the packaging progress sending module 402 is further used to obtain the proportional relationship between the predicted download time and the predicted upload time according to the file format of each file to be downloaded in the batch download request; wherein the file format includes text files, video files and picture files.

[0095] Optionally, each file to be downloaded in the batch download request includes at least one first type file and at least one second type file; wherein the first type file includes a file identifier, and the second type file includes a file address.

[0096] The batch download acquisition module 401 is further configured to distribute each file to be downloaded to multiple download processes according to the file type of each file to be downloaded, so that each download process maintains a balance of file types, and read each file to be downloaded through the multiple download processes.

[0097] Optionally, both the first type file and the second type file include a file size identifier.

[0098] The batch download acquisition module 401 is specifically used to obtain matching weight values ​​according to the file types of each file to be downloaded, and to obtain the download time of each file to be downloaded according to the weight value and file size identifier of each file to be downloaded; wherein the weight value of the first type of file is less than the weight value of the second type of file; and according to the download time of each file to be downloaded, each file to be downloaded is allocated to multiple download processes so that the download time of each download process is balanced.

[0099] The above device can execute the batch file downloading method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the batch file downloading method provided by any embodiment of the present invention.

[0100] Embodiment 5

[0101] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, electronic devices, blade electronic devices, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0102] like Figure 5As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0103] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0104] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a batch file download method.

[0105] In some embodiments, the batch file download method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on a heterogeneous hardware accelerator via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the batch file download method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform the batch file download method by any other appropriate means (e.g., by means of firmware).

[0106] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or electronic device.

[0108] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0109] To provide interaction with a user, the systems and techniques described herein may be implemented on a heterogeneous hardware accelerator having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) through which a user can provide input to the heterogeneous hardware accelerator. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0110] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data electronic device), or a computing system that includes middleware components (e.g., an application electronic device), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0111] The computing system may include a client and an electronic device. The client and the electronic device are generally remote from each other and usually interact through a communication network. The relationship between the client and the electronic device is generated by computer programs running on corresponding computers and having a client-electronic device relationship with each other. The electronic device may be a cloud electronic device, also known as a cloud computing electronic device or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0113] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A batch file downloading method, characterized in that: include: In response to obtaining a batch download request submitted by a target user, reading each file to be downloaded according to the batch download request; Writing the currently downloaded file into a file compression package, and sending the packaging progress of the batch download request to the target user; Upload the downloaded compressed file package to the object storage service, and generate a download address for the compressed file package through the object storage service; The download address is sent to the target user, so that the target user obtains batch files through the download address and deletes the file compression package stored locally.

2. The method according to claim 1, characterized in that The step of responding to the batch download request submitted by the target user and reading each file to be downloaded according to the batch download request includes: In response to obtaining a batch download request submitted by a target user, generating a task code according to the batch download request, storing the task code and the batch download request in a cache, and storing the task code in a message queue; If the task code is obtained from the message queue through the listening process, the batch download request is obtained from the cache based on the task code through the listening process, so that the listening process reads each file to be downloaded according to the batch download request.

3. The method according to claim 1, characterized in that The step of sending the packaging progress of the batch download request to the target user includes: The packaging progress of the batch download request is updated according to the number of download files written into the file compression package and the proportional relationship between the predicted download time and the predicted upload time.

4. The method according to claim 3, characterized in that Before updating the packaging progress of the batch download request according to the number of download files written into the file compression package and the ratio of the predicted download time to the predicted upload time, the method includes: According to the file formats of the files to be downloaded in the batch download request, a proportional relationship between the predicted download time and the predicted upload time is obtained; wherein the file formats include text files, video files and picture files.

5. The method according to claim 1, characterized in that Each file to be downloaded in the batch download request includes at least one first type file and at least one second type file; wherein the first type file includes a file identifier, and the second type file includes a file address; The step of reading each file to be downloaded according to the batch download request includes: According to the file type of each file to be downloaded, each file to be downloaded is distributed to multiple download processes, so that each download process maintains a balance of file types, and each file to be downloaded is read through the multiple download processes.

6. The method according to claim 5, characterized in that The first type of file and the second type of file both include a file size identifier; The method of allocating each file to be downloaded to multiple download processes according to the file type of each file to be downloaded, so that each download process maintains a balance of file types, includes: Obtaining matching weight values ​​according to the file types of the files to be downloaded, and respectively obtaining downloading times of the files to be downloaded according to the weight values ​​and file size identifiers of the files to be downloaded; wherein the weight values ​​of the first type of files are smaller than the weight values ​​of the second type of files; According to the downloading time of each file to be downloaded, each file to be downloaded is distributed to multiple downloading processes, so that the downloading time of each downloading process is balanced.

7. A batch file downloading device, characterized in that: include: A batch download acquisition module, configured to respond to a batch download request submitted by a target user and read each file to be downloaded according to the batch download request; A packaging progress sending module, used to write the currently downloaded file into a file compression package, and send the packaging progress of the batch download request to the target user; The object storage execution module is used to upload the downloaded file compression package to the object storage service, and generate a download address of the file compression package through the object storage service; The download address sending module is used to send the download address to the target user, so that the target user can obtain batch files through the download address and delete the file compression package stored locally.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the batch file downloading method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the batch file downloading method according to any one of claims 1 to 6 when executed.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the batch file downloading method according to any one of claims 1 to 6 is implemented.

Images