Continuous integration method and system for automatically monitoring compilation dependency change of processing packet

Abstract

The invention discloses a continuous integration method and system for automatically monitoring and processing package compiling dependency change. The continuous integration method comprises the steps that koji is tracked through koschei, and dependency relationship change of a software source is analyzed; performing simulation installation on the software packages subjected to the dependency change in the software source in sequence, and creating a real compiling task for the software packages which fail in simulation installation; preprocessing the real compiling task and then submitting the real compiling task to the koji; and polling and synchronizing the execution state of the real compiling task in the koji by using the polling service of the koschei, and if the failure times of the real compiling task reach a threshold value, creating a notification task for reminding manual intervention processing. The invention aims to solve the problem that the koschei cannot be continuously integrated aiming at a platform based on the koji and the koschei management software package, so that the problem that the software package in a software source is unavailable is automatically and continuously solved.

Description

Technical Field

[0001] The present invention relates to the technical field of software source management, and in particular to a continuous integration method and system for automatically monitoring and processing package compilation dependency changes. Background Art

[0002] There are usually dependencies between packages in the software sources in the Linux system. For example, when updating package A in the software source, the dependency of package B that depends on package A may change. Usually, the maintainer of the package may not be aware of this change in dependency, resulting in the failure to update package B in time, which may cause package B to be unable to be installed or used, and further affect the availability of the entire software source. Due to the transmission of dependencies, the update of a single package may have an impact on the entire software source. For example, if package A depends on package B, and package B depends on package C, when package C is updated, packages A and B may not be installed. Since packages A and B may be dependencies of other packages, this transmission of dependencies may cause many packages in the software source to be unusable, and they will not be discovered until the next large-scale rebuild, resulting in delayed delivery.

[0003] Currently, there are two open source tools that can be used to compile and monitor rpm packages: koji and koschei. Koji is a powerful package compilation management tool that uses mock as the underlying environment to batch compile software packages. Koschei is a continuous integration service for rpm packages, mainly used to automatically and continuously monitor changes in rpm package dependencies within a specified software source. Koji can be used to compile and manage software packages, while koschei can be used to monitor rpm package dependencies. Although koschei can detect changes in package dependencies, it cannot confirm whether the package can be installed within the source. When a software package cannot be installed, it needs to be recompiled to resolve dependency issues. However, koschei cannot automatically generate version numbers for software packages, nor can it update the source code repository for software packages, let alone submit real compilations to koji. Summary of the invention

[0004] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a continuous integration method and system for automatically monitoring and processing package compilation dependency changes are provided. The present invention aims to solve the problem that koschei cannot be continuously integrated for a platform based on koji and koschei to manage software packages, thereby automatically and continuously solving the problem of unavailability of software packages in software sources.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: A continuous integration method for automatically monitoring and processing package compilation dependency changes includes the following steps: Track koji and resolve dependency changes of software sources through koschei; All software packages with changed dependencies are analyzed and installed in the software source in turn; Create real compilation tasks for software packages that cannot be simulated and installed in the software source, preprocess the real compilation tasks to generate software package information, update the code repository, and then submit it to Koji; The execution status of the real compilation task in koji is synchronized through koschei polling. If the number of failures of the real compilation task reaches the threshold, a notification task is created to remind manual intervention.

[0006] Optionally, tracking koji and resolving dependency changes of software sources through koschei includes: polling and synchronizing koji's software sources through koschei's repository resolution service repo_resolver, resolving new software sources and corresponding software packages, and analyzing dependencies of all software packages, thereby obtaining all software packages with changed dependencies.

[0007] Optionally, the step of respectively analyzing all software packages with changed dependencies and installing them in software source simulation includes traversing all software packages with changed dependencies, and processing the traversed software packages includes: Determine whether the software package has dependency changes. If not, end and return to continue traversal. Determine whether the software package can be installed in software source simulation. If so, end the process and return to continue traversal. Otherwise, store the software package in the task scheduling database as a software package that cannot be installed in software source simulation to create a real compilation task for the software package that cannot be installed in software source simulation.

[0008] Optionally, when determining whether the software package can be installed in software source simulation, if the software package cannot be installed in software source simulation, the step includes: Check whether the software package has any unfinished scheduling tasks for scheduling the creation of real compilation tasks. If no unfinished scheduling tasks exist, jump to the next step; otherwise, end and return to continue traversal; Determine whether manual review is required. If manual review is required, create a manual review task to remind manual intervention, end the task and return to continue traversal; otherwise, jump to the next step; Determine whether the number of times the actual compilation task of the software package has been submitted to Koji has exceeded the preset threshold. If it exceeds the preset threshold, create a manual review task to remind manual intervention, end and return to continue traversal; otherwise jump to the next step; Determine whether koschei has submitted a real compilation task of the software package to koji within the recent preset time. If so, create a manual review task to remind manual intervention, end and return to continue traversal; otherwise, store the software package in the task scheduling database as a software package that cannot be installed in software source simulation to create a real compilation task for the software package that cannot be installed in software source simulation.

[0009] Optionally, the scheduling of creating a real compilation task for the software package that cannot be simulated and installed in the software source, preprocessing the real compilation task to generate information of the software package and updating the code repository before submitting it to Koji includes: Read the task scheduling database to obtain the real compilation task data that needs to be submitted for real compilation; Determine whether there is an unsubmitted real compile task in the task scheduling database. If so, obtain the unsubmitted real compile task and jump to the next step. If not, exit and wait for the next scheduling. Determine whether the software package of the unsubmitted real compilation task is a software package that can be installed in software source simulation. If it is a software package that can be installed in software source simulation, update the unsubmitted real compilation task to completed and exit; otherwise, jump to the next step; Determine whether there is an unsent task order for reminding manual intervention in the task scheduling database. If there is an unsent task order for reminding manual intervention, create a message report of the task order and alarm notification, and exit to wait for the next scheduling; otherwise, jump to the next step; Determine whether there are any unresolved task orders in the task scheduling database. If there are any unresolved task orders, exit and wait for the next scheduling; otherwise, jump to the next step; Get all identifiers nvr and file sizes of the software package on Koji, where the identifier nvr includes the software name, version and release number; Clone the git source code of the package based on all the identifiers nvr and file sizes of the package on Koji; Generate the software package identifier nvr in a simulated test environment based on the git source code of the software package, update the git source code of the software package after deduplication, and perform environment cleanup; Update the data status of the real compilation task in the task scheduling database, and send a real compilation request to Koji to trigger Koji to perform real compilation.

[0010] Optionally, when determining whether there are any unresolved task orders in the task scheduling database, before jumping to the next step, it also includes determining whether the current time is the preset busy execution time. If it is not the preset busy execution time, jump to the next step; if it is the preset busy execution time, exit to wait for the next scheduling.

[0011] Optionally, when synchronizing the execution status of the real compilation task in koji through koschei polling, it also includes updating the compilation status in the compilation database, the identifier nvr of the software package, and updating the task information in the scheduling task database, and the task information includes the execution status of the real compilation task of koji and the sending status information of the task order used to remind manual intervention.

[0012] In addition, the present invention also provides a continuous integration system for automatically monitoring and processing package compilation dependency changes, including a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the continuous integration method for automatically monitoring and processing package compilation dependency changes.

[0013] In addition, the present invention also provides a computer-readable storage medium, in which a computer program or instruction is stored. The computer program or instruction is programmed or configured to execute the continuous integration method of automatically monitoring and processing package compilation dependency changes through a processor.

[0014] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the continuous integration method for automatically monitoring and processing package compilation dependency changes through a processor.

[0015] Compared with the prior art, the present invention mainly has the following advantages: The present invention includes tracking koji and parsing the dependency changes of software sources through koschei; simulating the installation of software packages with changed dependencies in the software source in turn, and creating real compilation tasks for software packages that fail the simulated installation; submitting the real compilation tasks to koji after preprocessing the real compilation tasks; using koschei's polling service to poll and synchronize the execution status of the real compilation tasks in koji, and if the number of failures of the real compilation tasks reaches a threshold, creating a notification task to remind manual intervention. The present invention solves the problem that koschei cannot be continuously integrated based on the platform of koji and koschei management software packages, thereby automatically and continuously solving the problem of unavailable software packages in the software source, and the present invention supports continuous and automatic detection of unavailable software packages in the software source, which can realize timely discovery of problems. The present invention supports continuous and automatic problem processing and submission of real compilation, which can realize timely repair of software sources. The present invention supports multiple submission methods of real compilation, which can adapt to diverse software source processing requirements. The present invention supports manual intervention of failure alarms, and ensures that the software source is always reliable in the form of a closed-loop task flow processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.

[0017] Figure 2 Detailed flowchart of the method of the embodiment of the present invention. DETAILED DESCRIPTION

[0018] like Figure 1 As shown, the continuous integration method for automatically monitoring and processing package compilation dependency changes in this embodiment includes: S1, tracks koji and parses the dependency changes of software sources through koschei; S2, simulate and install all software packages with changed dependencies in the software source one by one; S3, creates real compilation tasks for software packages that cannot be simulated and installed in the software source, preprocesses the real compilation tasks to generate software package information, updates the code repository, and then submits it to Koji; S4, synchronize the execution status of the real compilation task in koji through koschei polling. If the number of failures of the real compilation task reaches a threshold, a notification task is created to remind manual intervention.

[0019] See also Figure 2 In step S1 of this embodiment, tracking koji and resolving the dependency changes of software sources through koschei includes: polling and synchronizing koji's software sources through koschei's warehouse resolution service repo_resolver, resolving new software sources and corresponding software packages, and analyzing the dependencies of all software packages, thereby obtaining all software packages with changed dependencies. When koji generates a new repo, koschei's warehouse resolution service repo_resolver will synchronize koji repo information and resolve the dependency changes of software packages in the new repo.

[0020] See also Figure 2 In step S2 of this embodiment, all software packages with changed dependencies are parsed out one by one in the simulation installation of the software source, including traversing all software packages with changed dependencies, and processing the traversed software packages includes: S2.1, determine whether the software package has a dependency change, if not, end and return to continue traversal; S2.2, determine whether the software package can be installed in software source simulation. If so, end the process and return to continue traversal. Otherwise, store the software package in the task scheduling database as a software package that cannot be installed in software source simulation so as to create a real compilation task for the software package that cannot be installed in software source simulation.

[0021] See also Figure 2 In step S2.2 of this embodiment, when determining whether the software package can be installed in software source simulation, if it cannot be installed in software source simulation, the following steps are included: S2.2.1, check whether the software package has any unfinished scheduling task for scheduling the creation of real compilation tasks. If no unfinished scheduling task exists, jump to the next step; otherwise, end and return to continue traversal; S2.2.2, determine whether manual review is required. If manual review is required, create a manual review task to remind manual intervention, end and return to continue traversal; otherwise, jump to the next step; S2.2.3, determine whether the number of times the actual compilation task of the software package is submitted to Koji has exceeded the preset threshold. If it exceeds the preset threshold, create a manual review task to remind manual intervention, end and return to continue traversal; otherwise jump to the next step; S2.2.4, determine whether koschei has submitted a real compilation task of the software package to koji within the recent preset time (which can be determined according to actual needs, for example, 24 hours in this embodiment); if so, create a manual review task to remind manual intervention, end and return to continue traversal; otherwise, store the software package in the task scheduling database as a software package that cannot be installed in software source simulation to create a real compilation task for the software package that cannot be installed in software source simulation.

[0022] See also Figure 2 In step S3 of this embodiment, a real compilation task is created for scheduling the software package that cannot be simulated and installed in the software source, and the real compilation task is preprocessed to generate information of the software package and update the code repository before submitting it to Koji, including: S3.1, read the task scheduling database to obtain the real compilation task data that needs to be submitted for real compilation; S3.2, determine whether there is an unsubmitted real compile task in the task scheduling database, if so, obtain the unsubmitted real compile task and jump to the next step, if not, exit and wait for the next scheduling; S3.3, determine whether the software package of the unsubmitted real compilation task is a software package that can be installed in software source simulation. If it is a software package that can be installed in software source simulation, update the unsubmitted real compilation task to completed and exit; otherwise, jump to the next step; S3.4, determine whether there is an unsent task order for reminding manual intervention in the task scheduling database. If there is an unsent task order for reminding manual intervention, create a message report of the task order and alarm notification, and exit to wait for the next scheduling; otherwise jump to the next step; in this embodiment, the content of the message report of the task order and alarm notification mainly includes: 1. Tasks that failed to submit real compilation 3 times; 2. Tasks that failed to schedule and poll execution; 3. Packages that have been submitted for real compilation within 24 hours; 4. Check whether the number of translation tasks of packages that cannot be installed in newrepo exceeds the threshold; 5. Tasks that require manual review; S3.5, determine whether there are unresolved task orders in the task scheduling database. If there are unresolved task orders (indicating that manual review is required before continuing), exit and wait for the next scheduling; otherwise, jump to the next step; S3.6, obtaining all identifiers nvr and file sizes of the software package on Koji, wherein the identifier nvr includes the software name, version and release number; S3.7, clone the git source code of the package according to all the identifiers nvr and file sizes of the package on Koji; S3.8, generating the software package identifier nvr in a simulated test environment according to the git source code of the software package, updating the git source code of the software package after deduplication, and performing environment cleanup; S3.9, updates the data status of the real compilation task in the task scheduling database, and sends a real compilation request to Koji to trigger Koji to perform real compilation.

[0023] See also Figure 2 In step S3.5 of this embodiment, before jumping to the next step, it also includes judging whether the current time is the preset busy execution time. If it is not the preset busy execution time, it jumps to the next step; if it is the preset busy execution time, it exits to wait for the next scheduling. The busy execution time can be set according to actual needs. In this embodiment, 10 pm to 8 am is not the preset busy execution time, and the rest of the time is the busy execution time.

[0024] See also Figure 2 In step S4 of this embodiment, when the execution status of the real compilation task in koji is synchronized by polling through koschei (specifically, koschei's polling service polling), it also includes updating the compilation status in the compilation database, the identifier nvr of the software package, and updating the task information in the scheduling task database, wherein the task information includes the execution status of the real compilation task of koji and the sending status information of the task sheet for reminding manual intervention and the manual processing status information of the task sheet.

[0025] In summary, for software packages with changed dependencies, the method of this embodiment will simulate the installation in the latest software source one by one. When the installation fails, it can not only confirm that the current software package needs to submit real compilation, but also obtain and record the detailed conflict information of the software package to facilitate manual processing. Koschei only supports submitting draft compilation to koji. Draft compilation cannot repair problems found in the software source. The method of this embodiment adds a functional module for submitting real compilation to koji in koschei. After successful compilation, the software source will be updated, which can truly realize the repair of the software source. In order to cope with the compilation requirements of different koji tags, the method of this embodiment implements two real compilation methods, manual review and automatic compilation, and different modes are selected by different configuration file parameters. The manual review mode means that after confirming that the software package has a dependency conflict and needs to submit a real compilation, the dependency conflict is written into a new task list, and the administrator is notified by email. After the administrator reviews and processes the task list, when the task list is obtained through polling as passed, the real compilation task will be automatically submitted during non-busy hours. The manual review mode is rigorous and reliable. The administrator can pay attention to the problems of the software source and the detailed conflicts of the software package in a timely manner, which is essential for the maintenance of important software sources. Automatic submission of real compilation means that there is no need for a manual review process. When a software package dependency conflict is found, the real compilation is directly submitted during non-busy hours. This method can automatically and sustainably solve the software package dependency problem. For tasks that fail to submit real compilation, the method of this embodiment will repeatedly submit the real compilation task a specified number of times at a certain time interval according to the configuration parameters to eliminate the problem of real compilation failure caused by factors such as insufficient resources and offline services in a short period of time. For tasks whose number of retry failures reaches a threshold, a task order will be created to record the software package conflict information and failure information, and the administrator will be notified by email, and manual intervention will be carried out to ensure the reliability of the software source by ensuring the task closed loop.

[0026] In addition, this embodiment also provides a continuous integration system for automatically monitoring and processing package compilation dependency changes, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the continuous integration method for automatically monitoring and processing package compilation dependency changes.

[0027] In addition, this embodiment also provides a computer-readable storage medium, in which a computer program or instruction is stored. The computer program or instruction is programmed or configured to execute the continuous integration method of automatically monitoring and processing package compilation dependency changes through a processor.

[0028] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the continuous integration method for automatically monitoring and processing package compilation dependency changes through a processor.

[0029] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0030] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A continuous integration method for automatically monitoring and processing package compilation dependency changes, characterized in that: The steps include: Track koji and resolve dependency changes of software sources through koschei; All software packages with changed dependencies are analyzed and installed in the software source in turn; Create real compilation tasks for software packages that cannot be simulated and installed in the software source, preprocess the real compilation tasks to generate software package information, update the code repository, and then submit it to Koji; The execution status of the real compilation task in koji is synchronized through koschei polling. If the number of failures of the real compilation task reaches the threshold, a notification task is created to remind manual intervention.

2. The continuous integration method for automatically monitoring and processing package compilation dependency changes according to claim 1, characterized in that: Tracking koji and resolving the dependency changes of software sources through koschei includes: polling and synchronizing koji's software sources through koschei's warehouse resolution service repo_resolver, resolving new software sources and corresponding software packages, and analyzing the dependencies of all software packages, thereby obtaining all software packages with changed dependencies.

3. The continuous integration method for automatically monitoring and processing package compilation dependency changes according to claim 1, characterized in that: The software packages whose dependencies have been parsed out are traversed one by one when the software source is simulated and installed respectively, and the processing of the traversed software packages includes: Determine whether the software package has dependency changes. If not, end and return to continue traversal. Determine whether the software package can be installed in software source simulation. If so, end the process and return to continue traversal. Otherwise, store the software package in the task scheduling database as a software package that cannot be installed in software source simulation to create a real compilation task for the software package that cannot be installed in software source simulation.

4. The continuous integration method for automatically monitoring and processing package compilation dependency changes according to claim 3 is characterized in that: The determining whether the software package can be installed in software source simulation, if it cannot be installed in software source simulation, includes: Check whether the software package has any unfinished scheduling tasks for scheduling the creation of real compilation tasks. If no unfinished scheduling tasks exist, jump to the next step; otherwise, end and return to continue traversal; Determine whether manual review is required. If manual review is required, create a manual review task to remind manual intervention, end the task and return to continue traversal; otherwise, jump to the next step; Determine whether the number of times the actual compilation task of the software package has been submitted to Koji has exceeded the preset threshold. If it exceeds the preset threshold, create a manual review task to remind manual intervention, end and return to continue traversal; otherwise jump to the next step; Determine whether koschei has submitted a real compilation task of the software package to koji within the recent preset time. If so, create a manual review task to remind manual intervention, end and return to continue traversal; otherwise, store the software package in the task scheduling database as a software package that cannot be installed in software source simulation to create a real compilation task for the software package that cannot be installed in software source simulation.

5. The continuous integration method for automatically monitoring and processing package compilation dependency changes according to claim 1, characterized in that: The scheduling of creating a real compilation task for a software package that cannot be simulated and installed in the software source, preprocessing the real compilation task to generate software package information, updating the code repository, and then submitting it to Koji includes: Read the task scheduling database to obtain the real compilation task data that needs to be submitted for real compilation; Determine whether there is an unsubmitted real compile task in the task scheduling database. If so, obtain the unsubmitted real compile task and jump to the next step. If not, exit and wait for the next scheduling. Determine whether the software package of the unsubmitted real compilation task is a software package that can be installed in software source simulation. If it is a software package that can be installed in software source simulation, update the unsubmitted real compilation task to completed and exit; otherwise, jump to the next step; Determine whether there is an unsent task order for reminding manual intervention in the task scheduling database. If there is an unsent task order for reminding manual intervention, create a message report of the task order and alarm notification, and exit to wait for the next scheduling; otherwise, jump to the next step; Determine whether there are any unresolved task orders in the task scheduling database. If there are any unresolved task orders, exit and wait for the next scheduling; otherwise, jump to the next step; Get all identifiers nvr and file sizes of the software package on Koji, where the identifier nvr includes the software name, version and release number; Clone the git source code of the package based on all the identifiers nvr and file sizes of the package on Koji; Generate the software package identifier nvr in a simulated test environment based on the git source code of the software package, update the git source code of the software package after deduplication, and perform environment cleanup; Update the data status of the real compilation task in the task scheduling database, and send a real compilation request to Koji to trigger Koji to perform real compilation.

6. The continuous integration method for automatically monitoring and processing package compilation dependency changes according to claim 5, characterized in that: When determining whether there is an unresolved task order in the task scheduling database, before jumping to the next step, it also includes determining whether the current time is the preset busy execution time. If it is not the preset busy execution time, jump to the next step; if it is the preset busy execution time, exit to wait for the next scheduling.

7. The continuous integration method for automatically monitoring and processing package compilation dependency changes according to claim 5, characterized in that: When synchronizing the execution status of the real compilation task in koji through koschei polling, it also includes updating the compilation status in the compilation database, the identifier nvr of the software package, and updating the task information in the scheduling task database, and the task information includes the execution status of the real compilation task of koji and the sending status information of the task sheet used to remind manual intervention.

8. A continuous integration system for automatically monitoring and processing package compilation dependency changes, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the continuous integration method for automatically monitoring and processing package compilation dependency changes as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the continuous integration method for automatically monitoring and processing package compilation dependency changes as described in any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the continuous integration method for automatically monitoring and processing package compilation dependency changes as described in any one of claims 1 to 7 through a processor.

Images