Server warehouse source synchronization method and system

Abstract

The invention discloses a server warehouse source synchronization method and system, and the method comprises the following steps: S1, taking an external network address of a warehouse source as a parameter, transmitting the parameter to a difference comparison module, and obtaining a packet list needing to be deleted and a packet list needing to be downloaded through the difference comparison module; s2, taking an external network address of a warehouse source as a parameter, transmitting the parameter to a warehouse metadata module, and obtaining a verification table by the warehouse metadata module; the verification table stores a first verification value of each software package in an external network warehouse; s3, the packet list needing to be deleted, the packet list needing to be downloaded and the check list serve as parameters and are transmitted to a synchronous execution module; the synchronous execution module deletes the corresponding software package in the local warehouse based on the package list needing to be deleted; the synchronous execution module downloads the corresponding software package to a local warehouse based on the package list needing to be downloaded; and obtaining a second verification value based on the downloaded software package, and comparing the second verification value with the first verification value.

Description

Technical Field

[0001] The present invention relates to the technical field of software synchronization, and specifically to a method and system for synchronizing server repository sources. Background Art

[0002] The repository source is one of the sources for installing software on the server operating system. Server operating system manufacturers generally place the repository source on the external network to facilitate operating system users to install the software inside. However, there is also a part of the operating system placed in the internal network environment. To install software, it is necessary to synchronize the manufacturer's repository source, put it into the internal network environment, and build an internal network repository source to install software. Therefore, it is necessary to synchronize the repository source.

[0003] In the current technical solutions, it is impossible to support multi-threading and repository metadata at the same time. Not only the bandwidth cannot be fully utilized, the synchronization time is long, but also the accuracy of the downloaded software package data cannot be verified using the repository metadata, and the synchronization consistency is poor. Therefore, how to quickly and accurately synchronize the repository sources of the internal network and the external network is a problem to be solved. Summary of the Invention

[0004] To solve the above problems, the purpose of the present invention is to provide a method and system for synchronizing server repository sources. By comparing the differences between the software packages in the local repository and the external network repository, multiple threads are started to delete or download the software packages, and the downloaded software packages are verified.

[0005] A method for synchronizing server repository sources of the present invention includes the following steps: Step S1: Pass the external network address of the repository source as a parameter to the difference comparison module. The difference comparison module obtains the list of packages to be deleted and the list of packages to be downloaded in the local repository based on the external network address of the repository source; Step S2: Pass the external network address of the repository source as a parameter to the repository metadata module. The repository metadata module obtains the verification table based on the external network address of the repository source; the verification table stores the first verification values of each software package in the external network repository; Step S3: Pass the list of packages to be deleted and the list of packages to be downloaded in Step S1 and the verification table in Step S2 as parameters to the synchronization execution module; the synchronization execution module deletes the corresponding software packages in the local repository based on the list of packages to be deleted; the synchronization execution module downloads the corresponding software packages to the local repository based on the list of packages to be downloaded; calculates the second verification value based on the downloaded software packages, and compares the second verification value of the downloaded software packages with the first verification values in the verification table corresponding to the downloaded software packages.

[0006] Further, the Step S1 includes the following steps: Step S11: Use the external network address of the repository source as a parameter to pass to the difference comparison module. The difference comparison module accesses the Packages page under the external network address, and uses the lxml library to parse the Packages page to obtain the first package list of the external network repository corresponding to the external network address of the repository source; Step S12: Obtain the path of the corresponding local repository through the external network address of the repository source, and obtain the second package list of the local repository under the path of the local repository; Step S13: The difference comparison module compares the names of the packages in the first package list with the names of the packages in the second package list; if the name of a package does not exist in the first package list but exists in the second package list, the name of the package is saved to the list of packages to be deleted; if the name of a package exists in the first package list but does not exist in the second package list, the name of the package is saved to the list of packages to be downloaded.

[0007] Further, the step S2 includes the following steps: Step S21: Use the external network address of the repository source as a parameter to pass to the repository metadata module. The repository metadata module accesses the repodata page under the external network address of the repository source, parses the href attribute value in the html of the repodata page, and filters to obtain the basic information file; Step S22: Read the basic information file using the bz2 library or the gzip library according to the suffix of the basic information file, and use the xml library for parsing; Step S23: Based on the href attribute value and the checksum value under the location in the basic information file, obtain the first checksum value of each package in the external network repository, and save it in the checksum table.

[0008] Further, the step S3 includes the following steps: Step S31: Pass the list of packages to be deleted and the list of packages to be downloaded in step S1 and the checksum table in step S2 as parameters to the executor of the synchronization execution module; Step S32: The executor of the synchronization execution module starts multiple deletion threads under the local repository according to the list of packages to be deleted, and passes the names of the packages in the list of packages to be deleted to the deletion threads respectively; the deletion threads delete the corresponding packages from the path of the local repository; Step S33: The executor of the synchronization execution module starts multiple download threads according to the list of packages to be downloaded and the checksum table, and passes the names of the packages in the list of packages to be downloaded and the corresponding first checksum values to the download threads respectively; Step S34: The download thread downloads the corresponding software package from the external network repository to the local repository, and calculates the second check value for the downloaded software package using the sha256 algorithm; compares and verifies the second check value with the first check value corresponding to the downloaded software package. Step S35: Based on the result of the comparison and verification, return a download result.

[0009] Further, in step S35, if the comparison and verification are consistent, mark the download result as successful. If the comparison and verification are inconsistent, the current download fails, delete the downloaded software package, and judge the number of download failures; if the number of download failures does not exceed a specified number, download again, if the number of download failures exceeds the specified number, do not download again, and directly mark the download result as an error.

[0010] The present invention also provides a server repository source synchronization system, which is used to implement the synchronization method. The system includes a difference comparison module, a repository metadata module, and a synchronization execution module. The difference comparison module takes the external network address of the repository source as a parameter to obtain the first software package list of the external network repository and the path of the local repository, and obtains the second software package list of the local repository based on the path of the local repository; compares the first software package list and the second software package list to obtain a list of packages to be deleted and a list of packages to be downloaded. The repository metadata module takes the external network address of the repository source as a parameter to obtain a basic information file, and obtains the first check value corresponding to each software package of the external network repository based on the basic information file. The synchronization execution module deletes the software packages in the local repository based on the list of packages to be deleted, and downloads the corresponding software packages from the external network repository to the local repository based on the list of packages to be downloaded; calculates the second check value of the downloaded software package, and compares the second check value with the first check value to obtain a download result.

[0011] Advantages of the present invention: In the process of synchronously downloading software packages in the present invention, multiple threads are used, which can make full use of bandwidth resources and reduce the total download time. In the thread, a verification mechanism is added. It ensures the consistency between the downloaded software packages and the software packages in the external network repository. At the same time, the software package check values are calculated and compared in parallel, maximizing the use of device resources and reducing the total time required for the verification process, so as to achieve fast and accurate synchronization of the repository source. Description of the Drawings

[0012] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0013] Figure 1 Schematic diagram of the system of the present invention; Figure 2 Flow chart of the synchronization method of the present invention; Figure 3 Flow chart of the download thread of the present invention. Specific embodiments

[0014] The following combines the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar generalizations without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0015] The object of the present invention is to provide a server repository source synchronization method and system, which compares the differences between the software packages in the local repository and the external network repository, starts multiple threads to delete and download the software packages, and verifies the downloaded software packages.

[0016] A server repository source synchronization method of the present invention, as Figure 2 shown, includes the following steps: Step S1: Pass the external network address of the repository source as a parameter to the difference comparison module (which executes the difference comparison program in Figure 2 ), and the difference comparison module obtains the list of packages to be deleted and the list of packages to be downloaded in the local repository based on the external network address of the repository source. The external network address of the repository source is the external network repository address.

[0017] Step S2: Pass the external network address of the repository source as a parameter to the repository metadata module (which executes the repository metadata program in Figure 2 ), and the repository metadata module obtains the verification table of the software packages in the external network repository based on the external network address of the repository source; the verification table stores the names and first verification values of each software package in the external network repository.

[0018] Step S3: Pass the list of packages to be deleted and the list of packages to be downloaded in step S1 and the verification table in step S2 as parameters to the synchronization execution module (which executes Figure 2The synchronous execution program in); The synchronous execution module deletes the corresponding software packages in the local repository based on the list of packages to be deleted; The synchronous execution module downloads the corresponding software packages from the external network repository to the local repository based on the list of packages to be downloaded; Calculate the second check value based on the downloaded software packages, and compare the second check value of the downloaded software packages with the first check value in the check table corresponding to the downloaded software packages.

[0019] The specific implementation steps of step S1 are as follows: Use the external network address of the repository source as a parameter to pass to the difference comparison module. The difference comparison module accesses the Packages page under the external network address, uses the lxml library to parse all href attribute values in the html under the Packages page, and obtains the first software package list (step S11) containing the names of all software packages in the external network repository.

[0020] Obtain the path of the corresponding local repository through the external network address of the repository source (assuming the external network address of the repository source is "xxx.com / path / to / file / ", then the path of the local repository is: "base directory / path / to / file / "). Under the Packages folder in the path of the local repository, obtain the names of all software packages in the local repository, so as to obtain the second software package list (step S12).

[0021] The difference comparison module performs a difference comparison on the names of the software packages in the first software package list and the names of the software packages in the second software package list; If the name of the software package does not exist in the first software package list but exists in the second software package list, the name of the software package is saved to the list of packages to be deleted delete_list; If the name of the software package exists in the first software package list but does not exist in the second software package list, the name of the software package is saved to the list of packages to be downloaded download_list (step S13).

[0022] The specific implementation steps of step S2 are as follows: Use the external network address of the repository source as a parameter to pass to the repository metadata module. The repository metadata module accesses the repodata page under the external network address of the repository source, parses the href attribute values in its html page, filters the file names containing the primary.xml field, and downloads the file corresponding to the file name. The downloaded file is the basic information file in the repository metadata (step S21).

[0023] Select the corresponding bz2 library or gzip library to read the basic information file according to the suffix of the basic information file. Parse the read basic information file using the xml library (step S22).

[0024] Extract the href attribute value and the checksum value under location in the basic information file, obtain the names of all software packages in the external network repository and their corresponding first checksum values, and save them to the checksum table packs_sha256_dict. Each software package name in the external network repository corresponds to a first checksum value (step S23).

[0025] The specific implementation steps of step S3 are as follows: Pass the package list delete_list to be deleted and the package list download_list to be downloaded in step S1, and the checksum table packs_sha256_dict in step S2 as parameters to the executor of the synchronous execution module (step S31).

[0026] According to the package list delete_list to be deleted, the executor starts multiple deletion threads under the local repository and passes the parameters in the package list delete_list to be deleted: the name of the software package. Each deletion thread will obtain the name of a software package to be deleted, and then the deletion thread will delete the corresponding software package from the local repository path according to the name of the software package in the package list delete_list to be deleted (step S32).

[0027] According to the package list download_list to be downloaded and the checksum table packs_sha256_dict, the executor starts multiple download threads and passes the parameters in the package list download_list to be downloaded (the name of the software package) and the first checksum value of the software package in the external network repository queried in the checksum table packs_sha256_dict to the download threads (step S33).

[0028] As Figure 3 shown, each download thread downloads the specified software package from the external network repository to the local repository. After downloading, the second checksum value is calculated using the sha256 algorithm, and the second checksum value is compared with the first checksum value in the external network repository metadata, and a download result is returned (step S34).

[0029] If the two values are the same, the download result is marked as successful. If the two values are different, it proves that the downloaded data is incomplete and this download fails. Delete the software package from the local repository and download it again. When the number of download failures exceeds a specified number, the download result is marked as an error. Finally, the download thread returns the download result (step S35).

[0030] The present invention also discloses a server repository source synchronization system, including a difference comparison module, a repository metadata module, and a synchronous execution module, as Figure 1 shown.

[0031] The difference comparison module takes the external network address of the repository source as a parameter to obtain the first software package list of the external network repository and the path of the local repository. Based on the path of the local repository, it obtains the second software package list of the local repository; based on the comparison of the first software package list and the second software package list (corresponding to Figure 1 "comparing the differences between software packages in the external network repository and the local repository"), it obtains the package list delete_list to be deleted and the package list download_list to be downloaded.

[0032] The repository metadata module takes the external network address of the repository source as a parameter to find the basic information file in the corresponding repository metadata, and based on the basic information file, it obtains the first check value corresponding to each software package in the external network repository (corresponding to Figure 1 "analyzing the metadata to obtain the check data of the repository packages and providing the packs_sha256_dict for verification).

[0033] The synchronous execution module deletes the software packages in the local repository based on the package list delete_list to be deleted, and downloads the corresponding software packages from the external network repository to the local repository based on the package list download_list to be downloaded; calculates the second check value of the downloaded software packages, and compares the second check value with the first check value to obtain the download result.

[0034] The repository source synchronization system of the present invention has the following functions: (1) Support for multi-threading: It can make full use of bandwidth resources and shorten the download time of the repository.

[0035] (2) Support for repository metadata: It can correctly process the metadata of the repository to ensure the accuracy and consistency of the repository source.

[0036] (3) Incremental synchronization: Only download the newly added or updated software packages in the external network repository source to reduce network overhead.

[0037] (4) Support for deleting redundant files (software packages): It can delete the files (software packages) in the local repository that are more than those in the external network repository to achieve synchronization consistency.

[0038] Generally speaking, in the process of software package synchronization, the present invention uses multi-threaded downloading, and in the multi-threading, a verification mechanism for repository metadata is added. The verification data comes from the external network repository metadata, realizing the simultaneous downloading and verification of multiple software packages, enabling the system to have both synchronization speed and synchronization accuracy, making full use of bandwidth and machine resources, and achieving fast and accurate synchronization of the repository source.

[0039] Abbreviations: Warehouse source: The server system software package warehouse source, which is a warehouse storing the server operating system installation packages.

[0040] Intranet: Refers to the network environment being a local area network environment.

[0041] Extranet: Refers to the network environment being an Internet environment.

[0042] The above are only the preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the concept of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art in this technical field, several improvements and refinements made without departing from the principle of the present invention should also be regarded as within the protection scope of the present invention.

Claims

1. A server warehouse source synchronization method, characterized in that: The steps include: Step S1: The external network address of the warehouse source is passed as a parameter to the difference comparison module, and the difference comparison module obtains a list of packages to be deleted and a list of packages to be downloaded based on the external network address of the warehouse source; Step S2: The external network address of the warehouse source is passed as a parameter to the warehouse metadata module, and the warehouse metadata module obtains a verification table based on the external network address of the warehouse source; The verification table stores the first verification value of each software package in the external network warehouse; Step S3: passing the list of packages to be deleted and the list of packages to be downloaded in step S1 and the checklist in step S2 as parameters to the synchronization execution module; The synchronous execution module deletes the corresponding software packages in the local repository based on the list of packages to be deleted; The synchronous execution module downloads the corresponding software package to the local warehouse based on the package list to be downloaded; calculates a second check value based on the downloaded software package, and compares the second check value of the downloaded software package with the first check value in the check table corresponding to the downloaded software package.

2. A server warehouse source synchronization method according to claim 1, characterized in that: The step S1 comprises the following steps: Step S11: The external network address of the warehouse source is passed as a parameter to the difference comparison module, and the difference comparison module accesses the Packages page under the external network address, and uses the lxml library to parse the Packages page to obtain the first software package list of the external network warehouse corresponding to the external network address of the warehouse source; Step S12: obtaining the path of the corresponding local warehouse through the external network address of the warehouse source, and obtaining the second software package list of the local warehouse under the path of the local warehouse; Step S13: The difference comparison module compares the names of the software packages in the first software package list with the names of the software packages in the second software package list; if the name of the software package does not exist in the first software package list but exists in the second software package list, the name of the software package is saved to the list of packages to be deleted; if the name of the software package exists in the first software package list but does not exist in the second software package list, the name of the software package is saved to the list of packages to be downloaded.

3. A server warehouse source synchronization method according to claim 1, characterized in that: The step S2 comprises the following steps: Step S21: The external network address of the warehouse source is passed as a parameter to the warehouse metadata module, and the warehouse metadata module accesses the repodata page under the external network address of the warehouse source, parses the href attribute value in the HTML of the repodata page, and screens to obtain the basic information file; Step S22: read the basic information file using the bz2 library or the gzip library according to the suffix of the basic information file, and parse it using the xml library; Step S23: Based on the href attribute value under location in the basic information file and the checksum value, a first check value of each software package in the external network warehouse is obtained and stored in a check table.

4. A server warehouse source synchronization method according to claim 1, characterized in that: The step S3 comprises the following steps: Step S31: passing the package list to be deleted and the package list to be downloaded in step S1 and the check table in step S2 as parameters to the executor of the synchronous execution module; Step S32: the executor of the synchronous execution module starts multiple deletion threads in the local repository according to the package list to be deleted, and transmits the names of the software packages in the package list to be deleted to the deletion threads respectively; The deletion thread deletes the corresponding software package from the path of the local repository; Step S33: the executor of the synchronous execution module starts multiple download threads according to the package list and checklist to be downloaded, and transmits the names of the software packages in the package list to be downloaded and the corresponding first check values ​​to the download threads respectively; Step S34: the download thread downloads the corresponding software package from the external network warehouse to the local warehouse, and calculates the downloaded software package using the sha256 algorithm to obtain a second check value; Compare and verify the second verification value with the first verification value corresponding to the downloaded software package; Step S35: Based on the comparison and verification result, a download result is returned.

5. A server warehouse source synchronization method according to claim 4, characterized in that: In step S35, if the comparison and verification are consistent, the download result is marked as downloaded successfully; If the comparison and verification are inconsistent, the download fails, the downloaded software package is deleted, and the number of download failures is determined; If the number of download failures does not exceed a specified number, the download is repeated. If the number of download failures exceeds a specified number, the download is not repeated and the download result is directly marked as an error.

6. A server warehouse source synchronization system, the system is used to implement the synchronization method according to claim 1, characterized in that: The system includes a difference comparison module, a warehouse metadata module and a synchronous execution module; The difference comparison module uses the external network address of the warehouse source as a parameter to obtain a first software package list of the external network warehouse and a path of the local warehouse, and obtains a second software package list of the local warehouse based on the path of the local warehouse; Obtaining a list of packages to be deleted and a list of packages to be downloaded based on a comparison between the first software package list and the second software package list; The warehouse metadata module obtains a basic information file using the external network address of the warehouse source as a parameter, and obtains a first check value corresponding to each software package of the external network warehouse based on the basic information file; The synchronous execution module deletes the software packages in the local warehouse based on the list of packages to be deleted, and downloads the corresponding software packages from the external network warehouse to the local warehouse based on the list of packages to be downloaded; A second check value of the downloaded software package is calculated, and the second check value is compared with the first check value to obtain a download result.

Images