Distributed file synchronization and downloading method and system, electronic device and storage medium

Abstract

The application discloses a distributed file synchronization and downloading method and system, electronic equipment and a storage medium, and belongs to the technical field of data synchronization. The technical problem to be solved by the application is how to avoid the workload of supplement collection caused by incomplete file collection, improve the timeliness and accuracy of data synchronization, and adopt the technical scheme. The method is to balance the manufacturer servers of each download client by using a message queue and a central node, and to establish a file index system to distribute the management of the download information of each manufacturer server. Each download client adopts a multi-thread or multi-process mode to download different files in parallel, and supports the functions of breakpoint continuation and error retry, so that the network optimization system realizes the complete automation of file collection when connecting with the northbound network management of the manufacturer, and improves the completeness and timeliness of downloading.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data synchronization, in particular to a distributed file synchronization and download method and system, electronic equipment and storage medium. BACKGROUND

[0002] With the explosive growth of communication network users, the event information of each user also increases exponentially, resulting in a large increase in the number and size of network management northbound files, and further resulting in a large increase in file storage servers. The traditional timing download program has been insufficient to support business needs, and the following problems exist:

[0003] ① The specified file is not generated: the specified file is not downloaded, resulting in incomplete collection of files, and the corresponding files need to be manually supplemented;

[0004] ② The download program runs for a long time: resulting in untimely file download and inability to output relevant indicators in a timely manner;

[0005] ③ Multiple manufacturer servers: resulting in high concurrency of download processes;

[0006] ④ Uneven deployment of download processes on manufacturer servers: resulting in uneven load of download processes, with some processes busy and some processes idle.

[0007] Therefore, how to avoid the workload of file collection incompleteness and improve the timeliness and accuracy of data synchronization is a technical problem to be solved. SUMMARY

[0008] The technical task of the present application is to provide a distributed file synchronization and download method, system, electronic equipment and storage medium to solve the problem of how to avoid the workload of file collection incompleteness and improve the timeliness and accuracy of data synchronization.

[0009] The technical task of the present application is achieved in the following manner: a distributed file synchronization and download method, which utilizes message queues and central node load balancing of manufacturer servers of each download client, and establishes a file index system to distribute management of download information of each manufacturer server. Each download client uses a multi-thread or multi-process parallel download method to download different files, and supports breakpoint resume and error retry functions, so that the network optimization system realizes complete automation of file collection when interfacing with the manufacturer northbound network management collection, and improves the completeness and timeliness of download.

[0010] As a preferred embodiment, the method is as follows:

[0011] Distributed file list management;

[0012] Intelligent scheduling and load balancing;

[0013] Parallel and block download;

[0014] Data verification.

[0015] More preferably, the distributed file list management is as follows:

[0016] A file index system is established at the central control node to record the basic information of the downloaded files on each vendor server; wherein the basic information of the downloaded files on each vendor server includes the file name, path, size and collection time.

[0017] More preferably, the intelligent scheduling and load balancing is as follows:

[0018] Intelligent task allocation: according to the current state of each download client, the central control node intelligently allocates each vendor server to the download client using a message queue to achieve load balancing; wherein the current state of the download client includes the degree of idleness, network condition and downloaded file list;

[0019] File download and update: each download client scans the files on the vendor server and compares with the file index system, downloads the undownloaded files, and reports the update to the central node.

[0020] More preferably, the breakpoint resume is as follows:

[0021] If the download is interrupted, continue downloading from the breakpoint to avoid repeated downloading and save bandwidth resources.

[0022] More preferably, error retry refers to automatically scheduling a retry for a failed download file until it is successful or reaches the upper limit of retries.

[0023] A distributed file synchronization and download system, which is used to implement the above-mentioned distributed file synchronization and download method; the system comprises:

[0024] A distributed file list management module for establishing a file index system at the central control node to record the basic information of the downloaded files on each vendor server; wherein the basic information of the downloaded files on each vendor server includes the file name, path, size and collection time;

[0025] An intelligent scheduling and load balancing module for intelligent task allocation and file download update;

[0026] A parallel and block download module for parallel downloading of different files by the download client using multi-thread or multi-process mode to improve download speed, and also for supporting breakpoint resume;

[0027] A data verification module for error retry; wherein error retry refers to automatically scheduling a retry for a failed download file until it is successful or reaches the upper limit of retries.

[0028] As preferred, the intelligent scheduling and load balancing module comprises:

[0029] An intelligent task allocation sub-module is configured to allocate the servers of each manufacturer to the download clients intelligently by the central control node using a message queue according to the current state of each download client, so as to realize load balancing, wherein the current state of each download client includes the idle degree, network condition and downloaded file list.

[0030] A file downloading and updating sub-module is configured to scan the files on the servers of each manufacturer by each download client, compare with the file index system, download the files not downloaded and report the update to the central node.

[0031] An electronic device comprises a memory and at least one processor.

[0032] The memory stores a computer program.

[0033] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the distributed file synchronization and downloading method as described above.

[0034] A computer readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the distributed file synchronization and downloading method as described above.

[0035] The distributed file synchronization and downloading method, system, electronic device and storage medium of the present application have the following advantages:

[0036] (1) The present application realizes complete automation of file collection when the network optimization system is connected to the northbound network management of the manufacturer, effectively improves the completeness and timeliness of downloading, effectively copes with the challenge of cross-server and large file quantity, and solves the problem of original manual cost;

[0037] (2) The present application is also applicable to the scenario of efficiently downloading a large number of files scattered on multiple servers, continuously generated and generated at different times, can intelligently manage and optimize the download of files, and realizes fast and full download of files;

[0038] (3) The present application realizes the intelligentization of file download through automatic means, reduces the workload of supplementary collection caused by incomplete file collection each time, and further reduces manual intervention, improves the timeliness and accuracy of data collection;

[0039] (4) The method of the present application can correct the situation of file not downloaded, increase data verification, automatically supplement the files failed to download, and effectively improve the completeness of downloading;

[0040] (V) The method of the present application can intelligently distribute each manufacturer server to the download client through intelligent scheduling, thereby reducing the problems of uneven load and high concurrency of the download process.

[0041] (VI) The download client of the present application uses a multi-thread or multi-process parallel download mode to download different files, so that the timeliness of file download is also greatly improved.

[0042] (VII) The traditional method requires professional personnel to perform real-time inspection on each download process every day and supplement collection; and the present application 4 effectively copes with cross-server and realizes efficient and reliable collection, so that only professional personnel need to perform periodic inspection, thereby greatly reducing the labor cost.

[0043] (VIII) The traditional method often causes data collection abnormality due to incomplete or untimely file download, thereby affecting the data quality; and the present application supports the breakpoint resume and error retry functions, thereby improving the completeness of file download and rarely causing data abnormality.

[0044] (IX) The traditional method often causes uneven load due to the collection of specific manufacturer servers by each download client; and the central node of the present application intelligently distributes each manufacturer server to the download client through the message queue, thereby adopting intelligent scheduling, realizing cross-server and balancing each download client.

[0045] The present application has the characteristics of reasonable design, simple structure, easy processing, small size, convenient use, one object for multiple uses and the like, and thus has good popularization and use value. BRIEF DESCRIPTION OF DRAWINGS

[0046] The present application will be further described below in combination with the drawings.

[0047] FIG. 1 is a schematic diagram of a distributed file synchronization and download method; Figure 1 FIG. 2 is an interface screenshot of a manufacturer server intelligently distributed to a download client;

[0048] Figure 2 FIG. 3 is an interface screenshot of a file automatically supplemented;

[0049] FIG. 4 is a schematic diagram of a distributed file synchronization and download system; Figure 3 DETAILED DESCRIPTION

[0050] The distributed file synchronization and download method, system, electronic device and storage medium of the present application will be described in detail below in combination with the drawings and specific embodiments.

[0051] Embodiment 1:

[0052] As shown in FIG. 1, the present application provides a distributed file synchronization and download method. Figure 1 ​​As shown, the embodiment provides a distributed file synchronization and download method, which balances the download client manufacturer servers by using a message queue and a central node, and establishes a file index system to distribute the download information of each manufacturer server. Each download client uses a multi-thread or multi-process mode to download different files in parallel, and supports breakpoint resume and error retry functions, so that the network optimization system realizes complete automation of file collection when interfacing with the manufacturer northbound network management collection, and improves the completeness and timeliness of the download. The details are as follows:

[0053] S1, distributed file list management;

[0054] S2, intelligent scheduling and load balancing;

[0055] S3, parallel and block download;

[0056] S4, data verification.

[0057] The distributed file list management in step S1 of the embodiment is as follows:

[0058] A file index system is established in the central control node to record the basic information of the downloaded files on each manufacturer server; wherein the basic information of the downloaded files on each manufacturer server includes the file name, path, size and collection time.

[0059] The intelligent scheduling and load balancing in step S2 of the embodiment is as follows:

[0060] S201, intelligent task allocation: as shown in the attached Figure 2 According to the current state of each download client, the central control node uses a message queue to intelligently allocate each manufacturer server to a download client to achieve load balancing; wherein the current state of the download client includes the idle degree, network condition and downloaded file list;

[0061] S202, file download and update: each download client scans the files on the manufacturer server and compares with the file index system, downloads the undownloaded files, and reports to the central node for update.

[0062] The breakpoint resume in the embodiment is as follows:

[0063] If the download is interrupted, continue downloading from the breakpoint to avoid repeated download and save bandwidth resources.

[0064] As shown in the attached Figure 3 Error retry in the embodiment refers to automatically scheduling retry for the downloaded files until success or reaching the upper limit of retry.

[0065] Embodiment 2:

[0066] The embodiment provides a distributed file synchronization and download system, which is used for implementing the distributed file synchronization and download method in the embodiment 1; the system comprises:

[0067] A distributed file list management module is used for establishing a file index system in the central control node, and recording basic information of the downloaded files on each vendor server; wherein the basic information of the downloaded files on each vendor server comprises a file name, a path, a size and a collection time;

[0068] An intelligent scheduling and load balancing module is used for intelligent task allocation and file download updating.

[0069] A parallel and block download module is used for parallel downloading of different files in a multi-thread or multi-process mode on the download client, so as to improve the download speed, and is also used for supporting breakpoint resume.

[0070] A data checking module is used for error retry; wherein the error retry refers to automatically arranging retry for the failed file until success or reaching the upper limit of the retry.

[0071] The intelligent scheduling and load balancing module in the embodiment comprises:

[0072] An intelligent task allocation sub-module is used for intelligently allocating each vendor server to the download client by the central control node by using a message queue according to the current state of each download client, so as to realize load balancing; wherein the current state of the download client comprises an idle degree, a network condition and a downloaded file list.

[0073] A file download and updating sub-module is used for scanning the files on the vendor server by each download client, comparing with the file index system, downloading the undownloaded files, and reporting the updating to the central node.

[0074] Embodiment 3:

[0075] The embodiment further provides an electronic device, comprising a memory and a processor.

[0076] The memory stores computer execution instructions.

[0077] The processor executes the computer execution instructions stored in the memory, so that the processor executes the distributed file synchronization and download method in any embodiment of the application.

[0078] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable logic devices (PLDs), discrete gates or transistor logic, discrete hardware components, and the like. The processor can be a microprocessor or the processor can be any conventional processor.

[0079] The memory can be used to store computer programs and / or modules, and the processor can realize various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store operating systems, application programs required by at least one function, and the like; and the data storage area can store data created according to the use of the terminal, and the like. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state memory devices.

[0080] Embodiment 4:

[0081] The embodiment also provides a computer readable storage medium, in which a plurality of instructions are stored, the instructions are loaded by a processor to make the processor execute the distributed file synchronization and download method in any embodiment of the application. Specifically, a system or device provided with a storage medium can be provided, the storage medium stores software program codes for realizing the functions of any one of the above embodiments, and the computer (or CPU or MPU) of the system or device reads and executes the program codes stored in the storage medium.

[0082] In this case, the program codes read from the storage medium can realize the functions of any one of the above embodiments, and thus the program codes and the storage medium storing the program codes constitute a part of the application.

[0083] The storage medium for providing the program codes includes floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program codes can be downloaded from a server computer through a communication network.

[0084] In addition, it should be clear that not only the program codes read by the computer can be executed, but also part or all of the actual operations can be completed by the operating system and the like operating on the computer based on the instructions of the program codes, so as to realize the functions of any one of the above embodiments.

[0085] Further, it is understood that the programs read out from the storage medium can be written to the memories of an expansion board inserted into the computer or an expansion unit connected to the computer, and then the CPU or the like mounted on the expansion board or the expansion unit is caused to perform part or all of the actual operations based on the instructions of the programs, thereby realizing the functions of any of the above-described embodiments.

[0086] Finally, it should be noted that the above-described embodiments are merely used to illustrate the technical solutions of the present application, rather than limit the present application; although the present application has been described in detail with reference to the above-described embodiments, those ordinarily skilled in the art should understand that they can still modify the technical solutions recorded in the above-described embodiments, or equivalently replace some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A distributed file synchronization and download method, characterized in that, This method utilizes message queues and a central node to load balance the vendor servers of each download client, and establishes a file indexing system to distribute and manage the download information of each vendor server. Each download client uses multi-threading or multi-processing to download different files in parallel, and supports breakpoint resumption and error retry functions. This enables the network optimization system to achieve fully automated file collection when connecting to the vendor's northbound network management system, improving the completeness and timeliness of downloads; details are as follows: Distributed file inventory management; Intelligent scheduling and load balancing; Parallel and chunked downloading; Data validation; The intelligent scheduling and load balancing are specifically as follows: Intelligent task allocation: Based on the current status of each download client, the central control node uses a message queue to intelligently allocate servers from various vendors to the download clients, thereby achieving load balancing; the current status of the download client includes its idle level, network status, and list of downloaded files; File download and update: Each download client scans the files on the vendor's server and compares them with the file indexing system, downloads any files that have not been downloaded, and reports the updates to the central node.

2. The distributed file synchronization and download method according to claim 1, characterized in that, The distributed file manifest management is as follows: A file indexing system is established at the central control node to record the basic information of the files downloaded to each vendor's server. The basic information of the files downloaded to each vendor's server includes the file name, path, size, and collection time.

3. The distributed file synchronization and download method according to claim 1 or 2, characterized in that, The details of resuming interrupted downloads are as follows: If the download is interrupted, it will resume from the point of interruption to avoid repeated downloads.

4. The distributed file synchronization and download method according to claim 3, characterized in that, Error retry refers to automatically retries for files that fail to download, until success is achieved or the retry limit is reached.

5. A distributed file synchronization and download system, characterized in that, The system includes: The distributed file inventory management module is used to establish a file index system at the central control node to record basic information about the files downloaded on each vendor's server. The basic information about the files downloaded on each vendor's server includes the file name, path, size, and collection time. The intelligent scheduling and load balancing module is used for intelligent task allocation and file download and update; The parallel and chunked download module is used to download different files in parallel using multi-threading or multi-processing in the download client, thereby improving download speed. It also supports resuming interrupted downloads. The data verification module is used for error retries; error retries refer to automatically retries for files that fail to download until they are successfully downloaded or the retry limit is reached. The intelligent scheduling and load balancing module includes: The intelligent task allocation submodule is used by the central control node to intelligently allocate servers from various vendors to download clients based on the current status of each download client, thereby achieving load balancing. The current status of the download client includes its idle level, network status, and list of downloaded files. The file download and update submodule is used by each download client to scan files on the vendor's server and compare them with the file indexing system, download any files that have not been downloaded, and report updates to the central node.

6. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the distributed file synchronization and download method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the distributed file synchronization and download method as described in any one of claims 1 to 4.

Images