Continuous integration and deployment method and device, computer equipment and readable storage medium
By creating a pipeline data model and updating the status based on the timing task, the problem of lack of general pipeline devices in the prior art is solved, and efficient management and maintenance of continuous integration and deployment is achieved.
Patent Information
- Application Number
- CN202411756144.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-02-28
AI Technical Summary
The prior art lacks general pipeline devices and cannot effectively realize continuous integration and deployment.
By defining and creating pipeline data models, including pipelines, pipeline stages and task entities, insert records based on preset orchestration order, and querying and updating task status through timed tasks, ensuring the real-time and accuracy of pipeline data models.
It realizes effective management and maintenance of pipeline data models, improves processing efficiency, and ensures the real-time and accuracy of data in task recording, pipeline stage recording and pipeline recording.
Smart Images
Figure CN119536753A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of cloud computing technology, and in particular to a continuous integration and deployment method, apparatus, computer equipment, and computer-readable storage medium. Background Art
[0002] DevOps (a combination of Development and Operations, i.e., integrated development and operation) emphasizes the communication and cooperation between software developers and operation and maintenance personnel, and uses automated processes to make software building, testing, and release faster, more frequent, and more reliable. The practice of the DevOps concept is based on an automated tool chain, the core of which is a continuous integration and deployment device, i.e., an assembly line. However, there is currently a lack of universal assembly line devices. Summary of the invention
[0003] Based on this, it is necessary to provide a simple and universal continuous integration and deployment method, apparatus, computer device, and computer-readable storage medium to address the above technical problems.
[0004] In a first aspect, the present application provides a continuous integration and deployment method, the method comprising:
[0005] Defining and creating a pipeline data model; entities in the pipeline data model include pipelines, pipeline stages, and tasks, wherein the pipeline includes a plurality of pipeline stages numbered in sequence, and the pipeline stage includes at least one task;
[0006] According to the running data of the pipeline data model, define and generate entity records corresponding to each of the entities;
[0007] In the initial stage of the pipeline data model running, based on a preset arrangement order, the pipeline record, the pipeline stage record, and the task record belonging to the first pipeline stage are respectively inserted into the pipeline data model;
[0008] Based on the first scheduled task, query whether there is a task record to be executed, and if it is determined that there is the task record to be executed, submit the task corresponding to the task record to be executed to the executor, and update the task record to be executed to be executed;
[0009] Periodically obtain the actual execution status of the task corresponding to the task record in execution, and update the task record in execution, the target pipeline stage record and the target pipeline record in sequence according to the actual execution status of the task; wherein the target pipeline stage record is a pipeline stage record in an incomplete state; and the target pipeline record is a pipeline record in an incomplete state.
[0010] In one embodiment, inserting the pipeline record, the pipeline stage record, and the task record belonging to the first pipeline stage into the pipeline data model based on the preset arrangement order includes:
[0011] Get pipeline information;
[0012] Inserting the pipeline record into the pipeline data model according to the pipeline information;
[0013] Get all pipeline stage information;
[0014] Traversing each of the pipeline stages in turn, and inserting the corresponding pipeline stage record into the pipeline data model according to the pipeline stage information;
[0015] When the traversed pipeline stage is the first pipeline stage, task information corresponding to the first pipeline stage is obtained, and the task record is inserted into the pipeline data model according to the task information.
[0016] In one embodiment, the periodic acquisition of the actual execution status of the task corresponding to the task record being executed, and sequentially updating the task record being executed, the target pipeline stage record, and the target pipeline record according to the actual execution status of the task, include:
[0017] Based on the second scheduled task, query whether there is a task record being executed;
[0018] When it is determined that there is the task record being executed, traverse each of the task records being executed in sequence;
[0019] For each of the task records being executed, calling the executor to obtain the actual execution status of the task, and updating the task record according to the actual execution status of the task;
[0020] According to the task record, updating the target pipeline stage record;
[0021] The target pipeline record is updated according to the pipeline stage record.
[0022] In one embodiment, updating the target pipeline stage record according to the task record includes:
[0023] Based on the third scheduled task, query whether there is a record of the target pipeline stage;
[0024] When it is determined that the target pipeline stage record exists, traverse each of the target pipeline stage records in sequence;
[0025] For each of the target pipeline stage records, obtain the task record corresponding to the target pipeline stage record. When all the task records are in a completed state, update the target pipeline stage record to a completed state; when there is a task record in an uncompleted state, the target pipeline stage record remains unchanged.
[0026] In one of the embodiments, the completion status includes execution failure and execution success;
[0027] The updating of the target pipeline record according to the pipeline stage record includes:
[0028] Based on the fourth scheduled task, check whether there is a target pipeline record;
[0029] When it is determined that the target pipeline record exists, traverse each of the target pipeline records in sequence;
[0030] For each of the target pipeline records, obtain the pipeline stage record corresponding to the target pipeline record. When all the pipeline stage records are in the completed state, or when there is a pipeline stage record that fails to execute, update the target pipeline record to the completed state; when there is a pipeline stage record that is in the non-completed state, the record state of the target pipeline record remains unchanged.
[0031] In one embodiment, after inserting the pipeline record, the pipeline stage record, and the task record belonging to the first pipeline stage into the pipeline data model based on a preset arrangement order, the method further includes:
[0032] Based on the fifth scheduled task, query whether there is a task to be inserted that meets the preset conditions;
[0033] When it is determined that the task record to be inserted exists, the task record to be inserted is generated according to the task to be inserted, and the task record to be inserted is added to the pipeline data model in sequence.
[0034] In a second aspect, the present application further provides a continuous integration and deployment device, the device comprising:
[0035] A model creation module, used to define and create a pipeline data model; the entities in the pipeline data model include pipelines, pipeline stages and tasks, wherein the pipeline includes a plurality of pipeline stages numbered in sequence, and the pipeline stage includes at least one task;
[0036] A record generation module, configured to define and generate entity records corresponding to each of the entities according to the operation data of the pipeline data model; and in the initial stage of the operation of the pipeline data model, based on a preset arrangement order, insert the pipeline record, the pipeline stage record, and the task record belonging to the first pipeline stage into the pipeline data model respectively;
[0037] A task execution module, used for querying whether there is a task record to be executed based on the first scheduled task, and if it is determined that there is the task record to be executed, submitting the task corresponding to the task record to be executed to the executor, and updating the task record to be executed to be executed;
[0038] A record maintenance module is used to periodically obtain the actual execution status of the task corresponding to the task record in execution, and update the task record in execution, the target pipeline stage record and the target pipeline record in sequence according to the actual execution status of the task; wherein the target pipeline stage record is a pipeline stage record in an incomplete state; and the target pipeline record is a pipeline record in an incomplete state.
[0039] In one embodiment, the continuous integration and deployment apparatus further includes:
[0040] The task insertion module is used to query whether there is a task to be inserted that meets the preset conditions based on the fifth scheduled task; when it is determined that there is the task to be inserted, generate a task record to be inserted according to the task to be inserted, and add the task record to be inserted to the pipeline data model in sequence.
[0041] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the continuous inheritance and deployment method provided in any of the above embodiments is implemented.
[0042] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the continuous inheritance and deployment method provided in any of the above embodiments is implemented.
[0043] The above-mentioned continuous integration and deployment method, device, computer equipment, and computer-readable storage medium define and create a three-layer pipeline data model consisting of pipelines, pipeline stages, and tasks, wherein each pipeline includes multiple serial pipeline stages, and each pipeline stage includes multiple parallel tasks. The pipeline data model can process these tasks simultaneously, thereby improving processing efficiency, and the data processing process is divided into pipelines, pipeline stages, and tasks. This layered design facilitates system maintenance and management. In the initial stage of the operation of the pipeline data model, based on a preset arrangement order, the pipeline records, pipeline stage records, and task records belonging to the first pipeline stage are respectively inserted into the pipeline data model. During the entire process of the operation of the pipeline data model, based on the first scheduled task, it is queried whether there are tasks to be executed. Task records, when it is determined that there are task records to be executed, the task corresponding to the task record to be executed is submitted to the executor, and the task record to be executed is updated to being executed, and the actual execution status of the task corresponding to the task record in execution is also obtained periodically, and the executing task record, target pipeline stage record and target pipeline record are updated in sequence according to the actual execution status of the task. It can be understood that the running data of the task, pipeline stage and pipeline are recorded respectively through the task record, pipeline stage record and pipeline record, and the task record, pipeline stage record and pipeline record are updated by periodically obtaining the actual execution status of the task, so as to ensure the real-time and accuracy of the data in the task record, pipeline stage record and pipeline record, and timely discover and handle abnormal problems in the operation of the pipeline data model. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0045] Figure 1 A flowchart of a continuous inheritance and deployment method in one embodiment;
[0046] Figure 2 It is an example of a pipeline data model in implementation;
[0047] Figure 3 This is a schematic diagram of a process after the first scheduled task is triggered in an embodiment;
[0048] Figure 4 A deployment framework diagram of a pipeline data model instance in an embodiment;
[0049] Figure 5 A flow chart of inserting pipeline records, pipeline stage records, and task records belonging to the first pipeline stage into a pipeline data model based on a preset arrangement order in one embodiment;
[0050] Figure 6 A schematic diagram of a process for periodically obtaining the actual execution status of a task corresponding to a task record being executed, and sequentially updating the task record being executed, the target pipeline stage record, and the target pipeline record according to the actual execution status of the task in an embodiment;
[0051] Figure 7 This is a schematic diagram of a process after the second scheduled task is triggered in one embodiment;
[0052] Figure 8 This is a schematic diagram of a process after the third scheduled task is triggered in an embodiment;
[0053] Fig. 9 This is a schematic diagram of a process after the fourth scheduled task is triggered in an embodiment;
[0054] Fig.10 This is a schematic diagram of a process after the fifth scheduled task is triggered in one embodiment;
[0055] Fig.11 A structural block diagram of a continuous integration and deployment device in an embodiment. DETAILED DESCRIPTION
[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0057] The continuous integration and deployment method of the present application can be applied to the control scheduler of the pipeline, and the control scheduler can be a physical server, a cloud server, a virtual machine, a container or a function service, etc.
[0058] In an exemplary embodiment, Figure 1 As shown, a continuous integration and deployment method is provided, including the following steps 102 to 110.
[0059] Step 102, define and create a pipeline data model.
[0060] The entities in the pipeline data model include pipelines, pipeline stages and tasks. Among them, the pipeline includes multiple pipeline stages numbered in sequence, and the pipeline stage includes at least one task. Pipelines, pipeline stages and tasks are all custom data structure types. You can first define one or more pipelines, and define the basic information of the pipeline, such as the pipeline name, the pipeline creator, the pipeline ID, etc. Under each pipeline, multiple serial pipeline stages can be defined, including defining the stage name, stage ID, and the pipeline ID to which each pipeline stage belongs. Multiple pipeline stages under one pipeline are numbered in sequence. Under each pipeline stage, multiple parallel tasks can be defined, and the configuration information of each task needs to be defined, including the task name, task ID, the name of the parameters required for task execution, and the pipeline stage ID to which the task belongs. After defining the pipeline data model, you can call the control scheduler to create an instance of the pipeline data model.
[0061] For example, Figure 2 is a simple pipeline data model example, in which the pipeline includes a serial compilation stage, an image building stage, and a container deployment stage. The editing stage includes a compilation task, the image building stage includes n image building tasks that can be performed simultaneously, and the container deployment stage includes m container deployment tasks that can be performed simultaneously (m and n are both positive integers). Among them, serial means that the next pipeline stage can only be entered after the previous pipeline stage is completed. Figure 2 For example, the image building phase can only be entered after the compilation phase is completed. Similarly, the container deployment phase can only be entered after the image building phase is completed.
[0062] Step 104: define and generate entity records corresponding to each entity according to the operation data of the pipeline data model.
[0063] The entity record is used to record the operation data of the corresponding entity in the pipeline data model and the generated intermediate result files, etc. The control scheduler can generate entity records corresponding to each entity according to the operation data of the pipeline data model, including pipeline records, pipeline stage records and task records.
[0064] Step 106, in the initial stage of the pipeline data model operation, based on the preset arrangement order, the pipeline record, the pipeline stage record, and the task record belonging to the first pipeline stage are respectively inserted into the pipeline data model.
[0065] The preset scheduling order is related to the execution order of each entity in the pipeline data model and can be set in advance by the user. When the pipeline data model just starts running, the control scheduler can insert pipeline records, pipeline stage records, and task records belonging to the first pipeline stage into the pipeline data model based on the preset scheduling order. Since each pipeline stage is serial, in the initial stage of the pipeline data model operation, when executing the tasks under the first pipeline stage, the tasks under the remaining pipeline stages are not executed, and naturally no corresponding task records are generated. Therefore, at this time, only the task records belonging to the first pipeline stage need to be inserted.
[0066] Step 108, based on the first scheduled task, query whether there is a task record to be executed. If it is determined that there is a task record to be executed, submit the task corresponding to the task record to be executed to the executor, and update the task record to be executed to be executed.
[0067] The timing period of the first scheduled task can be set by the user based on empirical rules. To be executed and being executed refer to the values of the fields in the task record that represent the task execution status.
[0068] During the entire process of running the pipeline data model, the control scheduler needs to periodically query whether there is a task record to be executed based on the first scheduled task. When it is determined that there is a task record to be executed, the control scheduler submits the task corresponding to the task record to be executed to the executor, and the executor executes the corresponding task. Then, the control scheduler updates the task record to be executed to being executed.
[0069] Specifically, Figure 3 As shown, after the first scheduled task starts, the control scheduler queries whether there are task records to be executed. If there are no task records to be executed, the first scheduled task is terminated. If there are task records to be executed, the control scheduler will record the number of task records to be executed, and set a traversal number taskindex_1, with an initial value of 1, and traverse all task records to be executed in sequence. When taskindex_1 is not greater than the number of task records to be executed, the control scheduler will encapsulate the parameter information required for task execution according to the currently traversed task records to be executed, and submit this parameter information to the executor. If the task is submitted successfully, the control scheduler will update the task execution status in the task record from pending to executing, and increase the value of taskindex_1 by 1; if the task submission fails, the control scheduler will update the task execution status in the task record from pending to execution failure, and increase the value of taskindex_1 by 1. When taskindex_1 is greater than the number of task records to be executed, the first scheduled task is terminated.
[0070] After receiving the task submitted by the control scheduler, the executor can execute the corresponding task by calling commonly used open source components, such as continuous integration software Jenkins, product management software Nexus, Git code hosting software Gogs, object storage software Minio, and image management software Harbor. Figure 2 For example, please refer to Figure 4 When executing the compilation task, Jenkins pulls the code from Gogs, obtains the compilation package from Nexus, compiles it, and stores the products of the compilation task, such as running data and intermediate result files, on Minio. After executing the image building task, Jenkins stores the products of the image building task in the system Harbor. When executing the container deployment task, Jenkins pulls the products of the image building from the system Harbor, pushes it to the Harbor of the target deployment environment, and then calls the interface of the container console to complete the container deployment. The container console sends instructions to the Kubernetes cluster, and the Kubernetes cluster pulls the image from the Harbor of the deployment environment and starts the service.
[0071] Step 110, periodically obtain the actual execution status of the task corresponding to the task record in execution, and update the task record in execution, the target pipeline stage record and the target pipeline record in sequence according to the actual execution status of the task.
[0072] Among them, the target pipeline stage record is a pipeline stage record in an unfinished state; the target pipeline record is a pipeline record in an unfinished state. The unfinished state includes pending execution and executing. The actual execution state of the task refers to the task execution state recorded in the executor. It can be understood that during the entire process of executing the pipeline data model, the control scheduler also needs to call the executor to periodically obtain the actual execution state of the task corresponding to the task record in execution, so as to timely update the task record in execution, the target pipeline stage record and the target pipeline record.
[0073] In an embodiment of the present application, a three-layer pipeline data model consisting of pipelines, pipeline stages and tasks is defined and created, wherein each pipeline includes multiple serial pipeline stages, and each pipeline stage includes multiple parallel tasks. The pipeline data model can process these tasks simultaneously, thereby improving processing efficiency. In addition, the data processing process is divided into pipelines, pipeline stages and tasks. This layered design facilitates system maintenance and management. In the initial stage of the operation of the pipeline data model, based on a preset arrangement order, the pipeline records, pipeline stage records, and task records belonging to the first pipeline stage are respectively inserted into the pipeline data model. During the entire process of the operation of the pipeline data model, based on the first scheduled task, it is queried whether there are task records to be executed. After determining that there are task records to be executed, the task records are checked. In the case of a task record to be executed, the task corresponding to the task record to be executed is submitted to the executor, and the task record to be executed is updated to being executed. In addition, the actual execution status of the task corresponding to the task record in execution is obtained at regular intervals, and the executing task record, target pipeline stage record and target pipeline record are updated in sequence according to the actual execution status of the task. It can be understood that the running data of the task, pipeline stage and pipeline are recorded respectively by the task record, pipeline stage record and pipeline record, and the task record, pipeline stage record and pipeline record are updated by regularly obtaining the actual execution status of the task, so as to ensure the real-time and accuracy of the data in the task record, pipeline stage record and pipeline record, and timely discover and handle abnormal problems in the operation of the pipeline data model.
[0074] In one embodiment, based on a preset arrangement order, pipeline records, pipeline stage records, and task records belonging to the first pipeline stage are respectively inserted into the pipeline data model, including: obtaining pipeline information, and inserting pipeline records into the pipeline data model according to the pipeline information; obtaining all pipeline stage information, traversing each pipeline stage in turn, and inserting corresponding pipeline stage records into the pipeline data model according to the pipeline stage information; and when the traversed pipeline stage is the first pipeline stage, obtaining task information corresponding to the first pipeline stage, and inserting the task record into the pipeline data model according to the task information.
[0075] The pipeline information includes information such as the pipeline name, pipeline ID, and pipeline stage IDs included in the pipeline. The pipeline record includes pipeline status, such as completion status (including successful execution and failed execution), executing status, pending execution status, etc. The pipeline stage information includes the stage name, stage ID, number of tasks belonging to the stage, and ID of tasks belonging to the stage for each pipeline stage. The pipeline stage record includes pipeline stage status, such as failed execution, successful execution, executing, and pending execution. Task information includes information such as task name, task ID, and configuration parameters required for task execution. Task records include task execution status, such as failed execution, successful execution, executing, and pending execution.
[0076] Specifically, Figure 5 As shown, the control scheduler can first obtain the pipeline information, generate the corresponding pipeline record according to the pipeline information, and insert the pipeline record into the pipeline data model. After inserting the pipeline record, the control scheduler can obtain all the pipeline stage information belonging to the pipeline, first obtain the number of pipeline stages according to the pipeline stage information, and define the traversal number stageindex_1 of the pipeline stage when the number of pipeline stages is greater than 0, with an initial value of 1. The control scheduler will repeat the following steps until the value of stageindex_1 is greater than the number of pipeline stages:
[0077] (1) Generate corresponding pipeline stage records according to pipeline stage information, and insert the pipeline stage records into the pipeline data model;
[0078] (2) Determine whether the current pipeline stage is the first pipeline stage according to the traversal number stageindex_1;
[0079] (3) If it is determined that the pipeline stage traversed is not the first pipeline stage, directly execute step (5);
[0080] (4) When it is determined that the traversed pipeline stage is the first pipeline stage, the task information corresponding to the first pipeline stage is obtained, and the corresponding task record is generated according to the task information and inserted into the pipeline data model;
[0081] (5) Increase the value of stageindex_1 by 1.
[0082] In one embodiment, Figure 6 As shown, the actual execution status of the task corresponding to the task record in execution is obtained periodically, and the task record in execution, the target pipeline stage record and the target pipeline record are updated in sequence according to the actual execution status of the task, including steps 602-610.
[0083] Step 602: Based on the second scheduled task, query whether there is a task record being executed.
[0084] The timing period of the second scheduled task can be set according to empirical rules. The second scheduled task is used to trigger the process of querying the task records in execution. The control scheduler can query each task record regularly based on the second scheduled task to determine whether there is a task record in which the value of the field representing the task execution status is in execution.
[0085] Step 604: When it is determined that there is a task record being executed, traverse each task record being executed in sequence.
[0086] When the control scheduler determines that there is a task record being executed, it will traverse each task record being executed in sequence.
[0087] Step 606: for each task record being executed, call the executor to obtain the actual execution status of the task, and update the task record according to the actual execution status of the task.
[0088] The control scheduler can obtain the actual execution status of the task by calling the executor, and update the field representing the task execution status in the task record according to the actual execution status of the task.
[0089] Specifically, Figure 7 As shown, after the second scheduled task is triggered, the control scheduler queries whether there is a task record in execution. If not, the second scheduled task is terminated directly. If it exists, the number of task records in execution is obtained, and the task record traversal number taskindex_2 is defined, and the initial value is set to 1. When the value of taskindex_2 is not greater than the number of task records in execution, the executor is called to obtain the actual execution status of the task. If the actual execution status of the task is in execution, the value of taskindex_2 is increased by 1, and the next task record in execution is traversed. If the actual execution status of the task is execution failure, the field representing the task execution status in the corresponding task record is updated from execution to execution failure, and the value of taskindex_2 is increased by 1, and the next task record in execution is traversed. If the actual execution status of the task is successful execution, the field representing the task execution status in the corresponding task record is updated from execution to execution failure, the result file generated after the task is successfully executed is recorded in the task record, and the value of taskindex_2 is increased by 1, and the next task record in execution is traversed.
[0090] Step 608, update the target pipeline stage record according to the task record.
[0091] The status of the pipeline stage record is related to the task execution status of multiple tasks corresponding to the pipeline stage. Therefore, based on the multiple task records corresponding to the pipeline stage, the task execution status of multiple tasks can be obtained, thereby updating the pipeline stage record.
[0092] Specifically, the control scheduler may first query whether there is a target pipeline stage record based on the third scheduled task. The third scheduled task is used to trigger the process of querying whether there is a target pipeline stage record, and the timing period of the third scheduled task can be set according to empirical rules. The completion status includes execution failure and execution success, and the non-completion status includes executing and pending execution. When the control scheduler determines that there is a target pipeline stage record, it traverses each target pipeline stage record in turn. For each target pipeline stage record, the control scheduler obtains the task record corresponding to the target pipeline stage record. When all task records are in the completion status, the target pipeline stage record is updated to the completion status; when there is a task record in the non-completion status, the target pipeline stage record remains unchanged.
[0093] For example, Figure 8 As shown, when the control scheduler determines that there is a target pipeline stage record, it obtains the number of target pipeline stage records, defines the pipeline stage traversal number stageindex_2, and sets the initial value to 1. When the value of stageindex_2 is not greater than the number of target pipeline stage records, all task records to which the target pipeline stage record currently traversed belongs are obtained, and when all task records are in the completed state, the target pipeline stage record is updated to the completed state; when there is a task record in the non-completed state, the target pipeline stage record remains unchanged. Then, the control scheduler adds 1 to the value of stageindex_2 and traverses the next target pipeline stage record.
[0094] Step 610, update the target pipeline record according to the pipeline stage record.
[0095] The pipeline includes multiple pipeline stages, and the pipeline state is related to the states of the multiple pipeline stages it includes. Therefore, the state of the pipeline stage can be obtained based on the pipeline stage record, so as to update the pipeline record.
[0096] Specifically, the control scheduler may first query whether there is a target pipeline record based on the fourth timing task. The fourth timing task is used to trigger the process of querying whether there is a target pipeline record, and the timing period of the fourth timing task may also be set based on the empirical rule. In this embodiment, in order to reduce resource waste, the timing period of the third timing task may be greater than the timing period of the second timing task, and less than the timing period of the fourth timing task. When it is determined that there is a target pipeline record, the control scheduler traverses each target pipeline record in turn. For each target pipeline record, the pipeline stage record corresponding to the target pipeline record is obtained. When all pipeline stage records are in the completed state, or when there is a pipeline stage record that fails to execute, the target pipeline record is updated to the completed state; when there is a pipeline stage record that is in the non-completed state, the record state of the target pipeline record remains unchanged. It can be understood that since the multiple pipeline stages contained in the pipeline are executed serially, when a certain pipeline stage fails to execute, the subsequent pipeline stages will not be executed. At this time, it can be considered that the pipeline execution fails.
[0097] For example, Fig. 9 As shown, after the fourth scheduled task is triggered, the control scheduler queries whether there is a target pipeline record. If there is no target pipeline record, the fourth scheduled task is terminated directly. If there is a target pipeline record, the number of target pipeline records is obtained, and the pipeline traversal number index is defined, and the initial value is set to 1. When the value of index is not greater than the number of target pipeline records, all pipeline stage records corresponding to the currently traversed target pipeline record are obtained. When all pipeline stage records are in the completed state, the target pipeline record is updated to be executed successfully; when there is a pipeline stage record that fails to execute, the target pipeline record is updated to be executed failed; when there is a pipeline stage record that is in a non-completed state, the record status of the target pipeline record remains unchanged. Then, the control scheduler adds 1 to the value of index and traverses the next target pipeline record.
[0098] In an embodiment of the present application, the idea of hierarchical maintenance of task records, pipeline stage records and pipeline records is adopted. Through the second scheduled task, the task records are updated regularly, through the third scheduled task, the pipeline stage records are updated according to the task records, and through the fourth scheduled task, the pipeline records are updated according to the pipeline stage records. This realizes the scheduled maintenance of task records, pipeline stage records and pipeline records, and ensures the real-time and accuracy of the data in each record.
[0099] In one embodiment, after the pipeline records, pipeline stage records, and task records belonging to the first pipeline stage are respectively inserted into the pipeline data model based on a preset arrangement order, the continuous inheritance and deployment method of the present application also includes: based on the fifth scheduled task, querying whether there are tasks to be inserted that meet preset conditions; when it is determined that there are task records to be inserted, generating task records to be inserted according to the tasks to be inserted, and adding the task records to be inserted to the pipeline data model in sequence.
[0100] Meeting the preset conditions means that the pipeline stage to which the task to be inserted belongs is recorded as being executed, and the previous pipeline stage is recorded as being completed. The fifth scheduled task is used to trigger a process of querying whether there is a task to be inserted that meets the preset conditions. The task cycle of the fifth scheduled task can also be set based on empirical rules. In the pipeline system, a task table is also defined, which can be used to store configuration information of the task to be inserted. The control scheduler can query in the task table whether there is a task to be inserted that meets the preset conditions.
[0101] Specifically, Fig.10 As shown, after the fifth scheduled task is triggered, the control scheduler queries the task table to see if there is a task to be inserted that meets the preset conditions. If not, the fifth scheduled task is terminated directly. If it exists, the number of tasks to be inserted is obtained, and the task traversal number taskindex_3 is defined, and the initial value is set to 1. When the value of taskindex_3 is not greater than the number of tasks to be inserted, the corresponding task record is generated according to the configuration information of the task to be inserted, the task record is inserted into the pipeline data model, and the value of taskindex_3 is increased by 1 to traverse the next task to be inserted. When the value of taskindex_3 is greater than the number of tasks to be inserted, the fifth scheduled task is terminated.
[0102] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0103] Based on the same inventive concept, the embodiment of the present application also provides a continuous integration and deployment device for implementing the continuous integration and deployment method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more continuous integration and deployment device embodiments provided below can refer to the limitations of the continuous integration and deployment method above, and will not be repeated here.
[0104] In an exemplary embodiment, Fig.11 As shown, a continuous integration and deployment device is provided, including: a model creation module 1102, a record generation module 1104, a task execution module 1106 and a record maintenance module 1108, wherein:
[0105] The model creation module 1102 is used to define and create a pipeline data model; the entities in the pipeline data model include pipelines, pipeline stages and tasks, wherein the pipeline includes a plurality of pipeline stages numbered in sequence, and the pipeline stage includes at least one task;
[0106] The record generation module 1104 is used to define and generate entity records corresponding to each entity according to the operation data of the pipeline data model; and in the initial stage of the pipeline data model operation, based on the preset arrangement order, respectively insert the pipeline record, the pipeline stage record, and the task record belonging to the first pipeline stage into the pipeline data model;
[0107] The task execution module 1106 is used to query whether there is a task record to be executed based on the first scheduled task, and if it is determined that there is a task record to be executed, submit the task corresponding to the task record to be executed to the executor, and update the task record to be executed to be executed;
[0108] The record maintenance module 1108 is used to periodically obtain the actual execution status of the task corresponding to the task record in execution, and update the task record in execution, the target pipeline stage record and the target pipeline record in sequence according to the actual execution status of the task; among which, the target pipeline stage record is a pipeline stage record in an incomplete state; and the target pipeline record is a pipeline record in an incomplete state.
[0109] In one embodiment, the record generation module is also used to obtain pipeline information; insert pipeline records into the pipeline data model based on the pipeline information; obtain all pipeline stage information; traverse each pipeline stage in turn, and insert corresponding pipeline stage records into the pipeline data model based on the pipeline stage information; when the traversed pipeline stage is the first pipeline stage, obtain task information corresponding to the first pipeline stage, and insert task records into the pipeline data model based on the task information.
[0110] In one embodiment, the record maintenance module is also used to query whether there is an executing task record based on the second scheduled task; when it is determined that there is an executing task record, traverse each executing task record in turn; for each executing task record, call the executor to obtain the actual execution status of the task, and update the task record according to the actual execution status of the task; according to the task record, update the target pipeline stage record; according to the pipeline stage record, update the target pipeline record.
[0111] In one embodiment, the record maintenance module is also used to query whether there is a target pipeline stage record based on a third scheduled task; when it is determined that there is a target pipeline stage record, each target pipeline stage record is traversed in turn; for each target pipeline stage record, the task record corresponding to the target pipeline stage record is obtained, and when all task records are in a completed state, the target pipeline stage record is updated to a completed state; when there is a task record in an uncompleted state, the target pipeline stage record remains unchanged.
[0112] In one embodiment, the record maintenance module is also used to query whether there is a target pipeline record based on the fourth scheduled task; when it is determined that there is a target pipeline record, traverse each target pipeline record in turn; for each target pipeline record, obtain the pipeline stage record corresponding to the target pipeline record, and when all pipeline stage records are in a completed state, or there is a pipeline stage record that fails to execute, update the target pipeline record to a completed state; when there is a pipeline stage record in a non-completed state, the record status of the target pipeline record remains unchanged.
[0113] In one embodiment, the continuous inheritance and deployment device also includes a task insertion module, which is used to query whether there is a task to be inserted that meets the preset conditions based on the fifth scheduled task; when it is determined that there is a task to be inserted, a task record to be inserted is generated according to the task to be inserted, and the task record to be inserted is added to the pipeline data model in sequence.
[0114] Each module in the above-mentioned continuous inheritance and deployment device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.
[0115] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the continuous inheritance and deployment method provided in any of the above embodiments is implemented.
[0116] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the continuous inheritance and deployment method provided by any of the above embodiments is implemented.
[0117] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the continuous inheritance and deployment method provided in any of the above embodiments.
[0118] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0119] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0120] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A continuous integration and deployment method, characterized in that: The method comprises: Defining and creating a pipeline data model; entities in the pipeline data model include pipelines, pipeline stages, and tasks, wherein the pipeline includes a plurality of pipeline stages numbered in sequence, and the pipeline stage includes at least one task; According to the running data of the pipeline data model, define and generate entity records corresponding to each of the entities; In the initial stage of the pipeline data model running, based on a preset arrangement order, the pipeline record, the pipeline stage record, and the task record belonging to the first pipeline stage are respectively inserted into the pipeline data model; Based on the first scheduled task, query whether there is a task record to be executed, and if it is determined that there is the task record to be executed, submit the task corresponding to the task record to be executed to the executor, and update the task record to be executed to be executed; Periodically obtain the actual execution status of the task corresponding to the task record in execution, and update the task record in execution, the target pipeline stage record and the target pipeline record in sequence according to the actual execution status of the task; wherein the target pipeline stage record is a pipeline stage record in an incomplete state; and the target pipeline record is a pipeline record in an incomplete state.
2. The method according to claim 1, characterized in that The method of inserting the pipeline record, the pipeline stage record, and the task record belonging to the first pipeline stage into the pipeline data model based on the preset arrangement order includes: Get pipeline information; Inserting the pipeline record into the pipeline data model according to the pipeline information; Get all pipeline stage information; Traversing each of the pipeline stages in turn, and inserting the corresponding pipeline stage record into the pipeline data model according to the pipeline stage information; When the traversed pipeline stage is the first pipeline stage, task information corresponding to the first pipeline stage is obtained, and the task record is inserted into the pipeline data model according to the task information.
3. The method according to claim 1, characterized in that The step of periodically acquiring the actual execution status of the task corresponding to the task record being executed, and sequentially updating the task record being executed, the target pipeline stage record, and the target pipeline record according to the actual execution status of the task, includes: Based on the second scheduled task, query whether there is a task record being executed; When it is determined that there is the task record being executed, traverse each of the task records being executed in sequence; For each of the task records being executed, calling the executor to obtain the actual execution status of the task, and updating the task record according to the actual execution status of the task; According to the task record, updating the target pipeline stage record; The target pipeline record is updated according to the pipeline stage record.
4. The method according to claim 3, characterized in that The updating of the target pipeline stage record according to the task record includes: Based on the third scheduled task, query whether there is a record of the target pipeline stage; When it is determined that the target pipeline stage record exists, traverse each of the target pipeline stage records in sequence; For each of the target pipeline stage records, obtain the task record corresponding to the target pipeline stage record. When all the task records are in a completed state, update the target pipeline stage record to a completed state; when there is a task record in an uncompleted state, the target pipeline stage record remains unchanged.
5. The method according to claim 3, characterized in that: The completion status includes execution failure and execution success; The updating of the target pipeline record according to the pipeline stage record includes: Based on the fourth scheduled task, check whether there is a target pipeline record; When it is determined that the target pipeline record exists, traverse each of the target pipeline records in sequence; For each of the target pipeline records, obtain the pipeline stage record corresponding to the target pipeline record. When all the pipeline stage records are in the completed state, or when there is a pipeline stage record that fails to execute, update the target pipeline record to the completed state; when there is a pipeline stage record that is in the non-completed state, the record state of the target pipeline record remains unchanged.
6. The method according to claim 1, characterized in that After inserting the pipeline record, the pipeline stage record, and the task record belonging to the first pipeline stage into the pipeline data model based on the preset arrangement order, the method further includes: Based on the fifth scheduled task, query whether there is a task to be inserted that meets the preset conditions; When it is determined that the task to be inserted exists, a task record to be inserted is generated according to the task to be inserted, and the task record to be inserted is added to the pipeline data model in sequence.
7. A continuous integration and deployment device, characterized in that: The device comprises: A model creation module, used to define and create a pipeline data model; the entities in the pipeline data model include pipelines, pipeline stages and tasks, wherein the pipeline includes a plurality of pipeline stages numbered in sequence, and the pipeline stage includes at least one task; A record generation module, configured to define and generate entity records corresponding to each entity according to the operation data of the pipeline data model; and in the initial stage of the operation of the pipeline data model, based on a preset arrangement order, insert the pipeline record, the pipeline stage record, and the task record belonging to the first pipeline stage into the pipeline data model respectively; A task execution module, used for querying whether there is a task record to be executed based on the first scheduled task, and if it is determined that there is the task record to be executed, submitting the task corresponding to the task record to be executed to the executor, and updating the task record to be executed to be executed; A record maintenance module is used to periodically obtain the actual execution status of the task corresponding to the task record in execution, and update the task record in execution, the target pipeline stage record and the target pipeline record in sequence according to the actual execution status of the task; wherein the target pipeline stage record is a pipeline stage record in an incomplete state; and the target pipeline record is a pipeline record in an incomplete state.
8. The continuous integration and deployment device according to claim 7, characterized in that: The device also includes: The task insertion module is used to query whether there is a task to be inserted that meets the preset conditions based on the fifth scheduled task; when it is determined that there is the task to be inserted, generate a task record to be inserted according to the task to be inserted, and add the task record to be inserted to the pipeline data model in sequence.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.