DevSecOps pipeline execution control method, device, electronic device and storage medium

Abstract

The present application discloses a DevSecOps pipeline execution control method, apparatus, electronic device, and storage medium. The method includes: obtaining tasks and classifying them by stage to construct a pipeline execution control data structure, wherein the pipeline execution control data structure includes pipeline information and a stage array, wherein the stage array includes different stage data, wherein the stage data includes stage information and a task array, wherein the task array includes different task data; a task scheduler starts a first thread to call a task scheduling function; the task scheduling function extracts the stage array and starts a first loop body, which traverses the stage array, extracts the task array from the stage data, and starts a second loop body, which traverses the task array and sends the task data to a first message queue, wherein the task data is sent to a task executor, and the task executor sends execution process data to the task scheduler. The present application facilitates pipeline management and control, improving efficiency.

Description

Technical Field

[0001] The present application relates to the field of pipeline control technology, and in particular to a Devsecops pipeline execution control method, device, electronic device, and storage medium. Background Art

[0002] The DevSecOps pipeline refers to the automation of software development processes, including source code acquisition, building, security testing, and deployment, according to DevSecOps specifications. Compared to the DevOps pipeline, the DevSecOps pipeline adds automation to the security testing phase to improve process security monitoring.

[0003] However, the number of pipeline tasks is usually huge, and the pipeline often consists of different stages. If the pipeline is executed directly according to the process, it is difficult to ensure that the pipeline can accurately execute the corresponding pipeline tasks at different stages. This is not only difficult to manage, but also inefficient. Summary of the Invention

[0004] The purpose of this application is to provide a Devsecops pipeline execution control method, device, electronic device and storage medium to address the deficiencies of some of the above-mentioned technical problems, which can better classify tasks at different stages and execute tasks at different stages in an orderly manner, thereby facilitating the management and control of the pipeline and improving efficiency.

[0005] To achieve the above objectives, the present application provides a DevSecOps pipeline execution control method, which includes:

[0006] The task scheduler obtains the pipeline execution tasks and classifies the pipeline execution tasks according to the stages;

[0007] The task scheduler constructs a pipeline execution control data structure, the pipeline execution control data structure including pipeline information and a stage array, the stage array including different stage data, each stage data including stage information and a task array, the task array including different task data;

[0008] The task scheduler starts a first thread to call a task scheduling function;

[0009] The task scheduling function starts the task executor; the task scheduling function extracts the stage array in the pipeline execution control data structure and starts the first loop body, the first loop body traverses the stage array, the first loop body extracts the task array in the stage data and starts the second loop body, the second loop body traverses the task array and sends the extracted task data to the first message queue of the task scheduler; the task scheduler uses the first message queue to send the task data to the task executor, the task executor sends execution process data to the task scheduler during task execution, and the task scheduler uses the second message queue to receive the execution process data.

[0010] Optionally, the pipeline execution control data structure further includes a pipeline execution ID;

[0011] The method further comprises:

[0012] The task scheduling function extracts the pipeline execution ID as a task resource identifier before starting the task executor;

[0013] The task scheduling function transfers the task resource identifier to the task executor when starting the task executor;

[0014] The task scheduler declares the first message queue according to the task resource identifier;

[0015] The task executor monitors the first message queue according to the task resource identifier.

[0016] Optionally, the task scheduler declares the second message queue according to the task resource identifier.

[0017] Optionally, the method further includes:

[0018] The task scheduler starts a second thread to monitor data in the second message queue;

[0019] According to the signal monitored by the second thread indicating that the execution of each task data in the task array is completed, the first loop body extracts the task array in the next stage data and starts the second loop body.

[0020] Optionally, the method further includes:

[0021] The “according to the signal of completion of execution of each task data in the task array monitored by the second thread, the first loop body extracts the task array in the next stage data and starts the second loop body” includes:

[0022] After the second thread monitors the signal that the task is completed, the task ID in the completed task data is stored in a global array variable;

[0023] When it is determined that the task ID of each task data of the task array exists in the global array variable, the first loop body extracts the task array in the next stage data and starts the second loop body.

[0024] Optionally, the task scheduler uses the second thread to send the received execution process log of the task executor to the Web end.

[0025] Optionally, the method further includes:

[0026] After the pipeline execution control data structure is constructed, the task scheduler starts a third thread to monitor the operation commands executed by the Web user on the pipeline;

[0027] The task scheduler obtains a task ID according to the operation command and sends the task ID and operation type to the first message queue;

[0028] The task executor performs an operation on the corresponding task according to the received task ID and the operation type.

[0029] In order to achieve the above objectives, the present application also provides a Devsecops pipeline execution control device, which includes:

[0030] An acquisition module, wherein the acquisition module is used by the task scheduler to acquire pipeline execution tasks and classify the pipeline execution tasks according to stages;

[0031] A construction module, wherein the construction module is used by the task scheduler to construct a pipeline execution control data structure, wherein the pipeline execution control data structure includes pipeline information and a stage array, wherein the stage array includes different stage data, and each stage data includes stage information and a task array, wherein the task array includes different task data;

[0032] A startup module, wherein the startup module is used by the task scheduler to start the first thread to call the task scheduling function;

[0033] A processing module, the processing module is used for the task scheduling function to start the task executor; the task scheduling function extracts the stage array in the pipeline execution control data structure and starts a first loop body, the first loop body traverses the stage array, the first loop body extracts the task array in the stage data and starts a second loop body, the second loop body traverses the task array and sends the extracted task data to the first message queue of the task scheduler; the task scheduler uses the first message queue to send the task data to the task executor, the task executor sends execution process data to the task scheduler during task execution, and the task scheduler uses the second message queue to receive the execution process data.

[0034] In order to achieve the above objectives, the present application further provides an electronic device, comprising:

[0035] processor;

[0036] a memory storing executable instructions for the processor;

[0037] The processor is configured to execute the Devsecops pipeline execution control method as described above by executing the executable instructions.

[0038] In order to achieve the above-mentioned objectives, the present application also provides a computer-readable storage medium having a computer program stored thereon, which includes the Devsecops pipeline execution control method as described above implemented when the computer program is executed by a processor.

[0039] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the DevSecOps pipeline execution control method described above.

[0040] This application first obtains pipeline execution tasks through the task scheduler, classifies pipeline execution tasks according to stages and constructs a pipeline execution control data structure. The pipeline execution control data structure includes pipeline information and a stage array. The stage array includes task data under different stages. During operation, the task scheduler starts the first thread to call the task scheduling function. The task scheduling function starts the task executor. The task scheduling function extracts the stage array in the pipeline execution control data structure and starts the first loop body. The first loop body traverses the stage array. The first loop body extracts the task array in the stage data and starts the second loop body. The second loop body traverses the task array and sends the extracted task data to the first message queue of the task scheduler. The task scheduler uses the first message queue to send task data to the task executor. The task executor sends execution process data to the task scheduler during task execution. The task scheduler uses the second message queue to receive execution process data. The present application can classify pipeline execution tasks according to stages and construct a pipeline execution control data structure, and can better distinguish the tasks that need to be executed in different stages. At runtime, the loop body is used to extract the task data corresponding to each stage in the stage array in the pipeline execution control data structure in turn, and the extracted task data is sent to the first message queue for the task executor to execute the corresponding task, so that the task executor can execute the corresponding tasks in different stages in turn, facilitate the management and control of tasks in different stages, and improve efficiency. Therefore, not only can the tasks in different stages be better classified, but also the tasks in different stages can be executed in an orderly manner, which is convenient for the management and control of the pipeline and improve efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of the Devsecops pipeline execution control method according to an embodiment of the present application.

[0042] Figure 2 This is a schematic block diagram of a Devsecops pipeline execution control device according to an embodiment of the present application.

[0043] Figure 3 This is a schematic block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to explain in detail the technical content, structural features, implementation principles, and achieved objectives and effects of this application, the following is a detailed description in conjunction with the implementation methods and the accompanying drawings.

[0045] See also Figure 1 , the embodiment of the present application discloses a Devsecops pipeline execution control method, which includes:

[0046] 101. The task scheduler obtains the pipeline execution tasks and classifies the pipeline execution tasks according to the stages.

[0047] Generally speaking, stages include source code pull, build, security testing, quality testing, deployment, and artifact archiving. Of course, other stages are also possible and are not limited here. By classifying pipeline execution tasks according to the form of stages, different pipeline execution tasks can be better divided.

[0048] 102. The task scheduler constructs a pipeline execution control data structure. The pipeline execution control data structure includes pipeline information and a stage array. The stage array includes different stage data. Each stage data includes stage information and a task array. The task array includes different task data.

[0049] Specifically, the pipeline execution tasks are classified according to the stages, and the classified pipeline execution tasks are reassembled to construct a pipeline execution control data structure, such as constructing a pipeline execution control data structure based on the json data format.

[0050] Specifically, the pipeline information includes a pipeline execution ID and a pipeline name, wherein the pipeline execution ID refers to an identification number corresponding to the pipeline.

[0051] Specifically, different stage data can be related data under stages such as source code pulling, building, security testing, quality testing, deployment, and product archiving. Each stage data has corresponding stage information and a task array. The stage information includes the stage ID (the stage ID refers to the identification number corresponding to the stage) and the stage name. The task array usually includes different task data, each of which has a corresponding task ID, task name, etc. Of course, task data can also include task execution parameters, task failure execution strategy, etc., which are not limited here.

[0052] 103. The task scheduler starts the first thread to call the task scheduling function.

[0053] Specifically, the task scheduler passes the pipeline execution control data to the task scheduling function.

[0054] 104. The task scheduling function starts the task executor. The task scheduling function extracts the stage array in the pipeline execution control data structure and starts the first loop body. The first loop body traverses the stage array. The first loop body extracts the task array in the stage data and starts the second loop body. The second loop body traverses the task array and sends the extracted task data to the first message queue of the task scheduler. The task scheduler uses the first message queue to send the task data to the task executor. The task executor sends the execution process data to the task scheduler during the task execution process. The task scheduler uses the second message queue to receive the execution process data.

[0055] In some embodiments, the above method further comprises:

[0056] The task scheduler starts a second thread to monitor data in the second message queue;

[0057] According to the signal of completion of execution of each task data in the task array monitored by the second thread, the first loop body extracts the task array in the next stage data and starts the second loop body.

[0058] By monitoring the data in the second message queue, and only when the signal of the completion of each task data in the monitored task array is received, the first loop body is triggered to extract the task array in the next stage data and start the second loop body. This not only makes it easy to confirm the execution status of each task data in the task array in the current stage, but also ensures that each task data in the task array in the current stage can be successfully executed, avoiding the phenomenon that some related tasks are not executed or are ignored.

[0059] The following example illustrates this, assuming that the stage data includes relevant data for stages such as source code pull, build, security testing, quality testing, deployment, and artifact archiving. The first loop traverses the stage array. If the first loop first extracts the task array from the stage data pulled from the source code and starts the second loop, the second loop traverses the task array from the stage data pulled from the source code and extracts the task data from the task array and sends it to the first message queue of the task scheduler. The task scheduler uses the first message queue to send the task data to the task executor. During task execution, the task executor sends execution progress data to the task scheduler. When the second thread monitors the completion signal for each task data in the task array (the completion signal for each task data in the task array from the stage data pulled from the source code), the first loop extracts the task array from the next stage data (the build stage) and starts the second loop. At this point, the second loop traverses the task array from the stage data of the build stage and extracts the task data from the task array and sends it to the first message queue of the task scheduler for the task executor to execute the corresponding task. (At the same time, the second thread continues to monitor the completion signal for each task data in the task array to determine whether to proceed to the next stage data extraction). It should be noted that the above specific forms are only provided as examples for ease of understanding and are not intended to be limiting.

[0060] Furthermore, the above method also includes:

[0061] The above-mentioned “according to the signal of completion of execution of each task data in the task array monitored by the second thread, the first loop body extracts the task array in the next stage data and starts the second loop body” includes:

[0062] After the second thread monitors the signal that the task is completed, it stores the task ID in the completed task data in the global array variable;

[0063] When it is determined that the task ID of each task data of the task array exists in the global array variable, the first loop body extracts the task array in the next stage data and starts the second loop body.

[0064] After the second thread listens to the signal that the task is completed, the task ID in the completed task data is stored in the global array variable. Only when it is determined that the task ID of each task data in the task array exists in the global array variable, the first loop body is triggered to extract the task array in the next stage data and the second loop body is started. This not only makes it easy to quickly and clearly confirm the execution completion signal of each task data in the task array in the current stage, but also facilitates management and improves efficiency.

[0065] Continuing to use the above specific form for example, when the second thread monitors the signal that each task data in the task array has been executed (the signal that each task data in the task array in the source code pull phase has been executed), the task ID in the completed task data is stored in the global array variable. By determining that the task ID of each task data in the task array (each task ID in the source code pull phase) exists in the global array variable, the first loop body extracts the task array in the next stage data (construction phase) and starts the second loop body. It should be noted that the above specific form is only for the sake of ease of understanding and is not limited here.

[0066] In some embodiments, the pipeline execution control data structure further includes a pipeline execution ID;

[0067] The above method further includes:

[0068] Before starting the task executor, the task scheduling function extracts the pipeline execution ID as the task resource identifier;

[0069] The task scheduling function passes the task resource identifier to the task executor when starting the task executor;

[0070] The task scheduler declares a first message queue according to the task resource identifier;

[0071] The task executor monitors the first message queue according to the task resource identifier.

[0072] By using the pipeline execution ID as a task resource identifier to declare the first message queue and monitor the first message queue, it is convenient to accurately communicate with the first message queue, facilitate accurate data transmission, and improve transmission reliability.

[0073] Specifically, based on the task resource identifier, custom characters are added to declare and monitor the first message queue. For example, the first message queue is declared using the task resource identifier + main as the queue name. After the task executor is started, it obtains the task resource identifier + main to monitor the first message queue. Of course, other specific forms are also possible and are not limited here.

[0074] Furthermore, the task scheduler declares a second message queue according to the task resource identifier.

[0075] Declaring the second message queue through the task resource identifier facilitates accurate communication with the second message queue and better distinguishes the first message queue from the second message queue, which is conducive to accurate data transmission and improves transmission reliability.

[0076] Specifically, based on the task resource identifier, a custom character is added to declare the second message queue, such as the task resource identifier + callback as the queue name to declare the second message queue, which is easy to distinguish from the first message queue. Of course, other specific forms are also possible and are not limited here.

[0077] In some implementations, the task scheduler uses the second thread to send the received execution process log of the task executor to the Web end.

[0078] By using the second thread to send the received task executor's execution process log to the Web end, users can view the task execution process log in real time, which is beneficial for user monitoring and improves practicality.

[0079] In some embodiments, the above method further comprises:

[0080] After the pipeline execution control data structure is built, the task scheduler starts the third thread to listen to the operation commands executed by the Web user on the pipeline;

[0081] The task scheduler obtains the task ID according to the operation command and sends the task ID and the operation type to the first message queue;

[0082] The task executor performs operations on the corresponding task based on the received task ID and operation type.

[0083] By starting a third thread to listen to the operation commands executed by Web users on the pipeline and obtaining the corresponding task ID and operation type, the task executor can perform relevant operations based on the received task ID and operation type, which is beneficial for users to perform task-related operations, facilitates management and control, and improves practicality.

[0084] Specifically, when the web side receives an operation command such as pause, cancel, continue, or terminate from the user for a task, the task scheduler parses the corresponding operation command to obtain the task ID, and sends the task ID and operation type to the first message queue. The task executor receives the task ID and operation type and performs related operations to execute the operation triggered by the user on the web side, such as pause, cancel, continue, or terminate the corresponding task.

[0085] This application first obtains pipeline execution tasks through the task scheduler, classifies pipeline execution tasks according to stages and constructs a pipeline execution control data structure. The pipeline execution control data structure includes pipeline information and a stage array. The stage array includes task data under different stages. During operation, the task scheduler starts the first thread to call the task scheduling function. The task scheduling function starts the task executor. The task scheduling function extracts the stage array in the pipeline execution control data structure and starts the first loop body. The first loop body traverses the stage array. The first loop body extracts the task array in the stage data and starts the second loop body. The second loop body traverses the task array and sends the extracted task data to the first message queue of the task scheduler. The task scheduler uses the first message queue to send task data to the task executor. The task executor sends execution process data to the task scheduler during task execution. The task scheduler uses the second message queue to receive execution process data. The present application can classify pipeline execution tasks according to stages and construct a pipeline execution control data structure, and can better distinguish the tasks that need to be executed in different stages. At runtime, the loop body is used to extract the task data corresponding to each stage in the stage array in the pipeline execution control data structure in turn, and the extracted task data is sent to the first message queue for the task executor to execute the corresponding task, so that the task executor can execute the corresponding tasks in different stages in turn, facilitate the management and control of tasks in different stages, and improve efficiency. Therefore, not only can the tasks in different stages be better classified, but also the tasks in different stages can be executed in an orderly manner, which is convenient for the management and control of the pipeline and improve efficiency.

[0086] See also Figure 2 , the embodiment of the present application further discloses a Devsecops pipeline execution control device, which includes:

[0087] The acquisition module 10 is used by the task scheduler to obtain the pipeline execution tasks and classify the pipeline execution tasks according to the stages;

[0088] Construction module 11, construction module 11 is used for the task scheduler to construct a pipeline execution control data structure, the pipeline execution control data structure includes pipeline information and a stage array, the stage array includes different stage data, each stage data includes stage information and a task array, and the task array includes different task data;

[0089] The starting module 12 is used by the task scheduler to start the first thread and call the task scheduling function;

[0090] Processing module 13, processing module 13 is used for the task scheduling function to start the task executor; the task scheduling function extracts the stage array in the pipeline execution control data structure and starts the first loop body, the first loop body traverses the stage array, the first loop body extracts the task array in the stage data and starts the second loop body, the second loop body traverses the task array and sends the extracted task data to the first message queue of the task scheduler; the task scheduler uses the first message queue to send task data to the task executor, the task executor sends execution process data to the task scheduler during the task execution process, and the task scheduler uses the second message queue to receive the execution process data.

[0091] For a detailed description of the Devsecops pipeline execution control device, please refer to the above-mentioned Devsecops pipeline execution control method, which will not be repeated here.

[0092] See also Figure 3 , the embodiment of the present application further discloses an electronic device, which includes:

[0093] Processor 21;

[0094] a memory 20 storing executable instructions of a processor 21;

[0095] The processor 21 is configured to execute the above-mentioned Devsecops pipeline execution control method by executing executable instructions.

[0096] An embodiment of the present application further discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned Devsecops pipeline execution control method is implemented.

[0097] The present application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned DevSecOps pipeline execution control method.

[0098] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented through hardware associated with computer program instructions. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0099] The above disclosure is only a preferred example of the present application and cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are all within the scope covered by the present application.

Claims

1. A Devsecops pipeline execution control method, characterized in that: include: The task scheduler obtains the pipeline execution tasks and classifies the pipeline execution tasks according to the stages; The task scheduler constructs a pipeline execution control data structure, wherein the pipeline execution control data structure includes a pipeline execution ID and a stage array, wherein the stage array includes different stage data, each stage data includes stage information and a task array, wherein the task array includes different task data; The task scheduler starts a first thread to call a task scheduling function; The task scheduling function starts a task executor; the task scheduling function extracts a stage array from the pipeline execution control data structure and starts a first loop body, the first loop body traverses the stage array, the first loop body extracts a task array from the stage data and starts a second loop body, the second loop body traverses the task array and sends the extracted task data to a first message queue of the task scheduler; The task scheduler sends the task data to the task executor using the first message queue, the task executor sends execution process data to the task scheduler during the task execution process, and the task scheduler receives the execution process data using the second message queue; The method further comprises: The task scheduling function extracts the pipeline execution ID as a task resource identifier before starting the task executor; The task scheduling function transfers the task resource identifier to the task executor when starting the task executor; The task scheduler declares the first message queue according to the task resource identifier; The task executor monitors the first message queue according to the task resource identifier.

2. The Devsecops pipeline execution control method according to claim 1, wherein: The task scheduler declares the second message queue according to the task resource identifier.

3. The Devsecops pipeline execution control method according to claim 1, wherein: Also includes: The task scheduler starts a second thread to monitor data in the second message queue; According to the signal monitored by the second thread indicating that the execution of each task data in the task array is completed, the first loop body extracts the task array in the next stage data and starts the second loop body again.

4. The Devsecops pipeline execution control method according to claim 3, wherein: Also includes: The "according to the signal of the second thread monitoring the completion of execution of each task data in the task array, the first loop body extracts the task array in the next stage data and starts the second loop body" includes: After the second thread monitors the signal that the task is completed, the task ID in the completed task data is stored in a global array variable; When it is determined that the task ID of each task data of the task array exists in the global array variable, the first loop body extracts the task array in the next stage data and starts the second loop body.

5. The Devsecops pipeline execution control method according to claim 3, wherein: The task scheduler uses the second thread to send the received execution process log of the task executor to the Web end.

6. The Devsecops pipeline execution control method according to claim 1, wherein: Also includes: After the pipeline execution control data structure is constructed, the task scheduler starts a third thread to monitor the operation commands executed by the Web user on the pipeline; The task scheduler obtains a task ID according to the operation command and sends the task ID and operation type to the first message queue; The task executor performs an operation on the corresponding task according to the received task ID and the operation type.

7. A Devsecops pipeline execution control device, characterized in that: include: An acquisition module, wherein the acquisition module is used by the task scheduler to acquire pipeline execution tasks and classify the pipeline execution tasks according to stages; A construction module, wherein the construction module is used by the task scheduler to construct a pipeline execution control data structure, wherein the pipeline execution control data structure includes a pipeline execution ID and a stage array, wherein the stage array includes different stage data, each stage data includes stage information and a task array, wherein the task array includes different task data; A startup module, wherein the startup module is used by the task scheduler to start the first thread to call the task scheduling function; a processing module, the processing module being used for the task scheduling function to start a task executor; the task scheduling function extracting a stage array from the pipeline execution control data structure and starting a first loop body, the first loop body traversing the stage array, the first loop body extracting a task array from the stage data and starting a second loop body, the second loop body traversing the task array and sending the extracted task data to a first message queue of the task scheduler; The task scheduler sends the task data to the task executor using the first message queue, the task executor sends execution process data to the task scheduler during the task execution process, and the task scheduler receives the execution process data using the second message queue; The task scheduling function extracts the pipeline execution ID as a task resource identifier before starting the task executor; The task scheduling function transfers the task resource identifier to the task executor when starting the task executor; The task scheduler declares the first message queue according to the task resource identifier; The task executor monitors the first message queue according to the task resource identifier.

8. An electronic device, characterized in that: include: processor; a memory storing executable instructions for the processor; The processor is configured to execute the Devsecops pipeline execution control method according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the Devsecops pipeline execution control method according to any one of claims 1 to 6 is implemented.

Images