Task scheduling method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a task scheduling method and device, electronic equipment, a storage medium and a program product. The method comprises the steps of determining a task label of a continuous integration task in response to a task processing request of the continuous integration task, determining a selection mode of a node cluster for processing the continuous integration task based on the task label and a preset association relationship between candidate task labels and candidate construction nodes, and determining a node cluster for processing the continuous integration task based on the selection mode. And selecting a first node cluster adaptive to the continuous integration task from the plurality of node clusters, and scheduling the continuous integration task to the first node cluster. The accuracy of task scheduling can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a task scheduling method, device, electronic device, storage medium and program product. Background Art

[0002] Continuous Integration (CI) is an important software development practice. In the process of continuous integration, code can be submitted to the code base multiple times, and a series of build and test processes will be automatically run after each submission. The purpose of continuous integration is to quickly discover and fix problems that may occur during the integration process, so as to keep the code base healthy and stable. In related technologies, continuous integration can be achieved through clusters, but the accuracy of scheduling continuous integration tasks in related technologies is low. Summary of the invention

[0003] The embodiments of the present application provide a task scheduling method, device, electronic device, storage medium and program product, which can improve the accuracy of task scheduling.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a task scheduling method, the method comprising:

[0006] In response to a task processing request of a continuous integration task, determining a task tag of the continuous integration task;

[0007] Based on the task label and the association between the preset candidate task labels and the candidate build nodes, determine a method for selecting a node cluster for processing the continuous integration task;

[0008] Based on the selection method, selecting a first node cluster adapted to the continuous integration task from multiple node clusters;

[0009] The continuous integration task is scheduled to the first node cluster.

[0010] The present application provides a task scheduling device, the device comprising:

[0011] A response module, configured to respond to a task processing request of a continuous integration task and determine a task tag of the continuous integration task;

[0012] A selection module, used to determine a selection method for a node cluster for processing the continuous integration task based on the task label and a pre-set association relationship between candidate task labels and candidate build nodes;

[0013] The selection module is further used to select a first node cluster adapted to the continuous integration task from multiple node clusters based on the selection method;

[0014] A scheduling module is used to schedule the continuous integration task to the first node cluster.

[0015] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0016] A memory for storing computer executable instructions;

[0017] The processor is used to implement the task scheduling method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0018] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the task scheduling method provided in the embodiment of the present application when executed by a processor.

[0019] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the task scheduling method provided in the embodiment of the present application is implemented.

[0020] The embodiments of the present application have the following beneficial effects:

[0021] The task scheduling method provided by the embodiment of the present application can determine the selection method of the node cluster based on the task label of the continuous integration task, and the association relationship between the pre-set candidate task label and the candidate construction node, and then can select the first node cluster adapted to the continuous integration task from multiple node clusters based on the selection method, so that the continuous integration task can be scheduled to the first node cluster. The present application involves multiple node clusters. In the process of performing task scheduling, the first node cluster can be selected from multiple node clusters. Compared with the solution of scheduling tasks with a single node cluster, the fault tolerance rate of the present application is higher. In addition, the present application can determine the screening method based on the task label of the continuous integration task, and then can select the first node cluster based on the screening method, which can improve the accuracy of task scheduling and can effectively utilize multiple node clusters. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a schematic diagram of the structure of the task scheduling system provided in the embodiment of the present application. Figure 1 ;

[0023] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0024] Figure 3This is a flowchart of the task scheduling method provided in the embodiment of the present application. Figure 1 ;

[0025] Figure 4 This is a flowchart of the task scheduling method provided in the embodiment of the present application. Figure 2 ;

[0026] Figure 5 This is a flowchart of the task scheduling method provided in the embodiment of the present application. Figure 3 ;

[0027] Figure 6 This is a flowchart of the task scheduling method provided in the embodiment of the present application. Figure 4 ;

[0028] Figure 7 This is a schematic diagram of the structure of the task scheduling system provided in the embodiment of the present application. Figure 2 ;

[0029] Figure 8 This is a flowchart of the task scheduling method provided in the embodiment of the present application. Figure 5 ;

[0030] Fig. 9 This is a flowchart of the task scheduling method provided in the embodiment of the present application. Figure 6 ;

[0031] Fig.10 It is a schematic diagram of the structure of the server provided in the embodiment of the present application;

[0032] Fig.11 This is a flowchart of the task scheduling method provided in the embodiment of the present application. Figure 7 . DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0034] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0035] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0036] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0037] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0038] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0039] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0040] 1) Node cluster: In this application, multiple node clusters for processing continuous integration tasks are multiple Kubernetes clusters (i.e., K8S clusters), each of which includes multiple worker nodes. Kubernetes is an open source container orchestration system for automatically deploying, scaling, and managing containerized applications. Continuous integration tasks are deployed to the cluster and implemented through scheduling functions. The system is responsible for complex tasks such as specific deployment scheduling, load balancing, service discovery, automatic scaling, and fault tolerance.

[0041] 2) Continuous integration tasks, that is, tasks involved in continuous integration. Continuous integration tasks can include multiple subtasks, including code pulling subtasks, dependency pulling subtasks, compilation and construction subtasks, testing subtasks, code quality checking subtasks, image creation subtasks, and product upload subtasks.

[0042] The code pull subtask is used to pull code from a preset code storage location. The dependency pull subtask is used to pull dependencies from a preset dependency storage location. The compile and build subtask is used to compile the code through a compiler or build tool, and the code can be compiled into a file or program.

[0043] The testing subtask includes unit testing, integration testing and other subtasks, which aim to ensure that code changes do not cause changes to the original functions. The code quality check subtask is used to check the code style, static code analysis and other checks to ensure the quality of the code. The image creation subtask is used to build container images. The product upload subtask is to upload the generated products to the product warehouse, where the generated products can include container images.

[0044] Embodiments of the present application provide a task scheduling method, device, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of task scheduling.

[0045] See also Figure 1 , Figure 1 This is a schematic diagram of the structure of the task scheduling system provided in the embodiment of the present application. Figure 1 , Figure 1 The task scheduling system 100 shown is for supporting a task scheduling application. The task scheduling system 100 includes a scheduling server 200 (ie, a server), a plurality of node clusters for processing continuous integration tasks, and a terminal 500 .

[0046] exist Figure 1 Only node cluster 400-1 and node cluster 400-2 are schematically shown, and the number of node clusters can also be 5, 10, or 20, which is not specifically limited here. Node cluster 400-1, node cluster 400-2, and terminal 500 are all connected to scheduling server 200 via network 300, and network 300 can be a wide area network or a local area network, or a combination of the two.

[0047] In some embodiments, when a user needs to process a continuous integration task, a task processing request may be generated in response to a task processing instruction triggered by the user based on a display page of the terminal 500 , and the terminal 500 may send the task processing request to the scheduling server 200 .

[0048] The scheduling server 200 determines the task label of the continuous integration task in response to the task processing request of the continuous integration task, and determines the selection method of the node cluster for processing the continuous integration task based on the task label and the association relationship between the pre-set candidate task labels and the candidate build nodes. Based on the selection method, the first node cluster adapted to the continuous integration task can be selected from the node cluster 400-1 and the node cluster 400-2.

[0049] The node cluster 400 - 1 and the node cluster 400 - 2 are both used to process the continuous integration task. In the case where the node cluster 400 - 1 is the first node cluster, the continuous integration task can be scheduled to the node cluster 400 - 1 .

[0050] When node cluster 400-2 is the first node cluster, the continuous integration task can be scheduled to node cluster 400-2. After scheduling the continuous integration task to node cluster 400-1 or node cluster 400-2, scheduling server 200 can send task scheduling completion information to terminal 500, so as to facilitate users to know the task scheduling completion information.

[0051] The electronic device provided in the embodiments of the present application is described below. The electronic device may be a terminal, a server, or a combination of the two. The electronic device provided in the embodiments of the present application may be implemented as various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart TV, and a vehicle-mounted terminal.

[0052] The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0053] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Figure 2 The electronic device shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the electronic device are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0054] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0055] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0056] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0057] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0058] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0059] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0060] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0061] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0062] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0063] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The task scheduling device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a response module 4551, a selection module 4552 and a scheduling module 4553. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0064] In other embodiments, the task scheduling device provided in the embodiments of the present application can be implemented in hardware. As an example, the task scheduling device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the task scheduling method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0065] Combine the following Figure 3 The task scheduling method provided by the embodiment of the present application is described. Figure 3 , Figure 3 This is a flowchart of the task scheduling method provided in the embodiment of the present application. Figure 1 As mentioned above, the electronic device that implements the task scheduling method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Below, taking the server implementation as an example, the task scheduling method provided by the embodiment of the present application is explained.

[0066] In step 101 , in response to a task processing request of a continuous integration task, a task tag of the continuous integration task is determined.

[0067] When a user needs to process a continuous integration task, a task tag of the continuous integration task is determined in response to a task processing request of the continuous integration task, wherein the continuous integration task is a task involved in the continuous integration process.

[0068] A continuous integration task can include multiple subtasks, including code pulling subtasks, dependency pulling subtasks, compilation and construction subtasks, testing subtasks, code quality check subtasks, image creation subtasks, and product upload subtasks.

[0069] The number of subtasks included in a continuous integration task can be set according to actual usage requirements. For example, continuous integration task 1 can include a code pull subtask, a dependency pull subtask, a compile and build subtask, and an artifact upload subtask. In other words, a continuous integration task can include 4 subtasks.

[0070] In some embodiments, the user can pre-set initial task tags for different continuous integration tasks based on the task type. For example, the continuous integration task can be a continuous integration task based on machine learning, that is, the task type is machine learning, and the user can set the initial task tag of the continuous integration task to a graphics processing unit (GPU).

[0071] In some embodiments, the task processing request of the continuous integration task carries an initial task tag preset by the user, and in response to the task processing request of the continuous integration task, the initial task tag may be determined as the task tag of the continuous integration task.

[0072] In order to improve the accuracy of the task label, the initial task label can also be compared with the pre-set standard label. If the information represented by the initial task label belongs to the standard label, the standard label is determined as the task label of the continuous integration task. If the information represented by the initial task label does not belong to any standard label, the initial label is deleted.

[0073] In some embodiments, an association relationship between candidate task types and candidate task tags can also be pre-set, and then in response to a task processing request for a continuous integration task, the task tag associated with the task type of the continuous integration task can be determined from the association relationship between the candidate task types and the candidate task tags according to the task type of the continuous integration task, thereby determining the task tag of the continuous integration task.

[0074] For ease of understanding, the task label is explained below. The task label may include a first task label and a second task label, wherein the first task label is a label that can characterize the cached build node corresponding to the continuous integration task. The build node is the basic unit for executing the continuous integration task, and the build node can be any working node in the Kubernetes cluster. The first task label can be the node name of the build node, the node address of the build node, etc. The first task label can also be the task identifier of the continuous integration task. The first task label can also be "yes", "exists", etc.

[0075] The second task tag is a tag that characterizes the requirement information of the continuous integration task. Specifically, the second task tag can be a tag that characterizes one or more of the resource requirement information, area requirement information, node type requirement information, isolation requirement information, and environment requirement information of the continuous integration task, which is described in detail below.

[0076] Resource requirement information is information required to execute continuous integration tasks, such as whether a GPU is needed. Another example is whether a GPU needs to be exclusively used. Another example is the number of cores of a central processing unit (CPU). Another example is the size of memory, etc.

[0077] Accordingly, when a GPU is needed, the second task tag may be GPU, and when a GPU is not needed, the second task tag may not be set. When an exclusive GPU is needed, the second task tag may be exclusive GPU, and when an exclusive GPU is not needed, the second task tag may not be set.

[0078] When there is a demand for the number of CPU cores, the second task tag can be CPUX, where X represents the number of CPU cores required. Of course, the demand for the CPU can also be processing speed, multi-threading capability, whether virtualization technology is supported, etc. The second task tag can be set according to actual usage requirements.

[0079] When there is a demand for memory size, the second task tag can be memory Y, where Y represents the required memory size. Of course, the memory requirement can also be performance requirement, memory bandwidth, memory type, etc. The second task tag can be set according to actual usage requirements.

[0080] The regional requirement information is the regional information required for executing the continuous integration task, and the second label information can be the identifier corresponding to the region. The node type requirement information is the information of the node type required for executing the continuous integration task, and the node type can be linux (first node type), macos (second node type), windows (third node type), etc. The node type requirement information can be the identifier for different node types, and the second task label can be the node type, or it can be the identifier for different node types.

[0081] The isolation requirement information includes business isolation requirement information and user isolation requirement information. The business isolation requirement information is the business isolation information required to execute the continuous integration task. For example, the continuous integration task corresponds to different applications, and the business isolation requirement information is the information corresponding to the application. The second task tag can be an identifier of the application.

[0082] The user isolation requirement information is the user (tenant) isolation information required to perform the continuous integration task. The second task tag can be the user's identifier. The environment requirement information is the environment information required to perform the continuous integration task. The environment information can be a development environment or a test environment. The second task tag can correspond to development or testing. Through step 101, the task tag of the continuous integration task can be determined.

[0083] In step 102, based on the task label and the association relationship between the preset candidate task labels and the candidate build nodes, a method for selecting a node cluster for processing the continuous integration task is determined.

[0084] In some embodiments, step 102 may be implemented in the following manner: after determining the task tag, a method for selecting a node cluster for processing the continuous integration task may be determined based on the task tag and the association between the preset candidate task tags and the candidate build nodes.

[0085] See also Figure 4 , Figure 4 This is a flowchart of the task scheduling method provided in the embodiment of the present application. Figure 2 , the following combination Figure 4 The steps shown are Figure 3 Step 102 is shown for explanation.

[0086] In step 1021, the construction node associated with the task tag is searched from the preset association relationship between the candidate task tags and the candidate construction nodes to obtain a search result.

[0087] In some embodiments, step 1021 is implemented in the following manner: the construction node associated with the first task tag can be searched from the pre-stored association relationship between the candidate task tags and the candidate construction nodes to obtain the search result.

[0088] In some embodiments, when the first task tag is the node name of the construction node and / or the node address of the construction node, the association between the candidate task tag and the candidate construction node is the association between the information of the candidate construction node and the candidate construction node.

[0089] In some embodiments, the construction node corresponding to the first task tag can be found from the information of the candidate construction node and the association relationship of the candidate construction node. The information of the candidate construction node includes the node name, node address and status of the candidate construction node.

[0090] In some embodiments, available candidate construction nodes can be first screened out from the candidate construction nodes based on the status in the information of the candidate construction nodes, and then based on the available candidate construction nodes, the information of the candidate construction nodes and the association relationship between the candidate construction nodes can be screened to obtain the information of the available candidate construction nodes and the association relationship between the candidate construction nodes.

[0091] Furthermore, the construction node associated with the node name of the candidate construction node and / or the node address of the candidate construction node (i.e., the first task label) can be searched from the information of the available candidate construction nodes and the association relationship between the candidate construction nodes, that is, the construction node associated with the first task label can be searched.

[0092] In some embodiments, when the first task tag is the task identifier of the continuous integration task, the association relationship between the candidate task tag and the candidate build node is the association relationship between the candidate task identifier and the candidate build node. Furthermore, the build node associated with the task identifier of the continuous integration task can be determined from the association relationship between the candidate task identifier and the candidate build node, that is, the build node associated with the first task tag can be determined.

[0093] The search result is used to indicate whether there is a construction node associated with the first task tag. If there is a construction node associated with the first task tag, the search result is used to indicate that there is a construction node associated with the first task tag. If there is no construction node associated with the first task tag, the search result is used to indicate that there is no construction node associated with the first task tag.

[0094] In step 1022, based on the search result, a method for selecting a node cluster for processing the continuous integration task is determined.

[0095] In some embodiments, step 1022 can be implemented in the following way: when the search result indicates that there is a build node associated with the first task tag, it can be determined that the method for selecting the node cluster for processing the continuous integration task is the fourth selection method, wherein the fourth selection method is used to select the node cluster according to the build node associated with the first task tag.

[0096] When the search structure representation does not contain a build node associated with the first task tag, it can be determined that the node cluster for processing the continuous integration task is selected in at least one of the first selection method, the second selection method, and the third selection method.

[0097] Among them, the first selection method is a method of selecting a node cluster based on the task label and the cluster label of the node cluster. The second selection method is a method of selecting a node cluster based on the requirement information of the continuous integration task and the information of the node cluster. The third selection method is a method of selecting a node cluster based on the score of the node cluster determined by the information of the node cluster.

[0098] In some embodiments, when the search structure representation does not contain a build node associated with the first task tag, the node cluster for processing the continuous integration task may be selected in the first selection method, the second selection method, or the third selection method.

[0099] In some embodiments, when the search structure representation does not contain a build node associated with the first task tag, the node cluster for processing the continuous integration task may be selected in any two of the first selection method, the second selection method, and the third selection method.

[0100] For the case where the selection method is any two of the first selection method, the second selection method, and the third selection method, if the first selection method exists, the third node cluster can be selected from the node cluster according to the first selection method, and then the first node cluster can be selected from the third node cluster according to the second selection method or the third selection method.

[0101] If the first selection method does not exist, the selection order between the second selection method and the third selection method can be set according to actual usage requirements. For example, the fourth node cluster can be selected from the node cluster according to the second selection method, and then the first node cluster can be selected from the fourth node cluster according to the third selection method.

[0102] For the cases where the selection method is the first selection method, the second selection method, and the third selection method, the third node cluster can be selected from the node cluster according to the first selection method first, and the selection order between the second selection method and the third selection method can be set according to actual usage requirements. For example, the fifth node cluster can be selected from the third node cluster according to the third selection method, and then the first node cluster can be selected from the fifth node cluster according to the second selection method.

[0103] Through steps 1021 and 1022, different selection methods for the node cluster that processes the continuous integration task can be determined based on the search results, which can improve the accuracy of determining the selection method. The selection method is used to select the first node cluster from the node cluster, which can improve the accuracy of selecting the first node cluster.

[0104] Continue to see Figure 3 In step 103, based on the selection method, a first node cluster adapted to the continuous integration task is selected from multiple node clusters.

[0105] For ease of understanding, step 103 is described below for different selection methods. For the case where the selection method is the first selection method, the first selection method has a first selection sub-method and a second selection sub-method. If the selection method is the first selection sub-method in the first selection method, a node cluster whose task label is consistent with the cluster label of the node cluster can be determined as the first node cluster adapted to the continuous integration task.

[0106] In some embodiments, the number of node clusters is greater than or equal to 2, and multiple node clusters used to process continuous integration tasks all have cluster labels. The cluster labels include multiple types. The cluster labels are labels used to characterize one or more of the resources that the node cluster can provide, the region where it is located, the node type provided, isolation information, and environmental information.

[0107] The resources that the node cluster can provide may include whether it has a GPU, whether it supports exclusive GPU, the number of CPU cores, the CPU processing speed, the size of available memory, the type of available content, etc., which may specifically correspond to the description of the resource demand information. The cluster label corresponding to the resources that the node cluster can provide may be consistent with the second task label corresponding to the resource demand information, which will not be described in detail here.

[0108] The region can be the region where each node in the node cluster is located, and the cluster label can be the identifier of the region where each node is located, or can be "multiple regions" or a general term for the regions where each node is located. The provided node type is the type of each node in the node cluster, and the cluster label can be the type of each node, or can be an identifier for different node types, or can be "multiple node types".

[0109] Isolation information includes business isolation information and user isolation information. For business isolation information, the cluster tag can be the ID of the application. For user isolation information, the cluster tag can be the ID of the user. Environment information can be a development environment or a test environment, and the cluster tag can correspond to development or test.

[0110] The cluster label corresponds to the second task label, and the description of the second task label can be replaced by equivalent, which will not be repeated here. After determining that the selection method is the first selection sub-method in the first selection method and determining the second task label, the second task label of the continuous integration task and the cluster label of each node cluster can be compared to see if they are consistent. If the second task label is consistent with the cluster label, the node cluster whose second task label is consistent with the cluster label of the node cluster can be determined as the first node cluster adapted for the continuous integration task.

[0111] If the second task label and the cluster label are inconsistent, the second selection sub-method can be used to determine the first node cluster. That is, there can be an execution order between the first selection sub-method and the second selection sub-method. If the first selection sub-method fails to determine the first node cluster, the second selection sub-method is used to determine the first node cluster.

[0112] If the selection method is the second selection sub-method in the first selection method, the cluster sub-label of the node cluster can be determined based on the identifier of the node cluster and the cluster label of the node cluster, and the node cluster whose second task label is consistent with the cluster sub-label can be determined as the first node cluster adapted for the continuous integration task.

[0113] In some embodiments, the association relationship between candidate cluster labels and candidate cluster sub-labels can be obtained based on the identification of the node cluster, and then based on the cluster label, the cluster sub-label of the cluster label can be determined from the association relationship between the candidate cluster label and the candidate cluster sub-labels, so that the cluster sub-label of the node cluster can be obtained.

[0114] For example, if the cluster label is "multi-node type", the association relationship between the candidate cluster label and the candidate cluster sub-label can be obtained according to the name of the node cluster (a type of identifier), and then according to "multi-node type", one or more cluster sub-labels of the cluster label can be determined from the association relationship between the candidate cluster label and the candidate cluster sub-label. The cluster sub-labels include macos and windows.

[0115] After obtaining the cluster sub-label, you can compare the second task label of the continuous integration task with the cluster sub-label of each node cluster to see if they are consistent. If the second task label is inconsistent with the cluster sub-label, the node cluster corresponding to the cluster sub-label is not processed, or for the continuous integration task, the node cluster corresponding to the cluster sub-label can be deleted.

[0116] If the second task label is consistent with the cluster sub-label, the node cluster whose task label is consistent with the cluster sub-label is determined as the first node cluster adapted to the continuous integration task. For example, the cluster sub-labels include macos (cluster sub-label 1) and windows (cluster sub-label 2). If the second task label is windows, the second task label is consistent with cluster sub-label 2. Therefore, the node cluster corresponding to cluster sub-label 2 can be used as the first node cluster.

[0117] For the case where the selection method is the first selection method, the following three steps can be performed. In the first step, the second task label of the continuous integration task can be compared with the cluster label of each node cluster to see if they are consistent. If the second task label is consistent with the cluster label, the node cluster whose second task label is consistent with the cluster label of the node cluster is determined as the first node cluster adapted for the continuous integration task.

[0118] If the second task label is inconsistent with the cluster label, in a second step, the cluster sub-label of the node cluster can be obtained based on the identifier of the node cluster and the cluster label of the node cluster, and then the cluster sub-label of the node cluster can be obtained based on the identifier of the node cluster and the cluster label of the node cluster.

[0119] In the third step, the second task label of the continuous integration task can be compared with the cluster sub-label of each node cluster to see if they are consistent. If the second task label is consistent with the cluster sub-label, the node cluster whose second task label is consistent with the cluster sub-label is determined as the first node cluster adapted to the continuous integration task. If the second task label is inconsistent with the cluster sub-label, the node cluster corresponding to the cluster sub-label is not processed, or the node cluster corresponding to the cluster sub-label can be deleted for the continuous integration task.

[0120] In some embodiments, the number of second task labels is one or more, and the number of cluster labels of a node cluster is one or more. Different situations are explained below. When the number of second task labels is equal to the number of cluster labels of a node cluster, the first step to the third step above can be executed to obtain and determine the first node cluster.

[0121] In the case where the number of second task labels is less than the number of cluster labels of a node cluster, for the first step above, it can be determined whether there is a consistent cluster label for each second task label. If both exist, the second task label is considered to be consistent with the cluster label. If not, the second to third steps above can be executed.

[0122] After obtaining the cluster sub-label, for the third step above, it can be determined whether each second task label has a consistent cluster sub-label. If both exist, the second task label is considered to be consistent with the cluster sub-label. If not, the node cluster corresponding to the cluster sub-label is not processed, or for continuous integration tasks, the node cluster corresponding to the cluster sub-label can be deleted.

[0123] In the case where the number of second task labels is greater than the number of cluster labels of a node cluster, the first step above may not be performed (i.e., the first sub-selection method may not be adopted), but the second step above may be performed (i.e., the second sub-selection method may be adopted). After obtaining the cluster sub-labels, if the number of cluster sub-labels is less than the number of task labels, the node cluster corresponding to the cluster sub-label is not processed, or for continuous integration tasks, the node cluster corresponding to the cluster sub-label may be deleted.

[0124] If the number of cluster sub-labels is greater than or equal to the number of second task labels, determine whether each second task label has a consistent cluster sub-label. If both exist, the second task label is considered to be consistent with the cluster sub-label. If both do not exist, the node cluster corresponding to the cluster sub-label is not processed, or for continuous integration tasks, the node cluster corresponding to the cluster sub-label can be deleted. In this way, the first node cluster can be accurately determined.

[0125] If the selection method is the second selection method, see Figure 5 , Figure 5 This is a flowchart of the task scheduling method provided in the embodiment of the present application. Figure 3 , the following combination Figure 5 The method of selecting the first node cluster when the selection method is the second selection method is described.

[0126] In step 1031A, if the selection method is the second selection method, for each node cluster, the requirement information of the continuous integration task in each dimension is compared with the information of the node cluster to obtain a comparison result in each dimension.

[0127] In step 1032A, based on the comparison result, a first node cluster suitable for the continuous integration task is selected from the multiple node clusters.

[0128] In some embodiments, the node cluster information may include node cluster information of one or more dimensions, for example, the node cluster information may include one or more of the node cluster information of the GPU dimension, the node cluster information of the CPU dimension, and the node cluster information of the memory dimension. Of course, it may also include node cluster information of the network bandwidth dimension, node cluster information of the dependency dimension, node cluster information of the service quality dimension, etc., which may be set in combination with specific usage requirements.

[0129] The node cluster information of one dimension may include multiple sub-information. For example, the node cluster information of the GPU dimension may include model sub-information, architecture sub-information, computing capability sub-information, etc. The node cluster information of the CPU dimension may include processing speed sub-information, multi-threading capability sub-information, core number sub-information, etc. The node cluster information of the memory dimension may include performance requirement sub-information, memory bandwidth sub-information, and memory sub-information.

[0130] The method for determining the requirement information is described below. In some embodiments, the requirement information of the continuous integration task can be determined in response to a task processing request of the continuous integration task, wherein the requirement information can be pre-set by a user, and the requirement information can also be requirement information associated with the type of the continuous integration task determined from the association relationship between the candidate type and the candidate requirement information based on the type of the continuous integration task.

[0131] The requirement information of the continuous integration task has multiple dimensions, that is, the requirement information may include requirement information of one or more dimensions. For example, the requirement information may include one or more of the requirement information of the GPU dimension, the requirement information of the CPU dimension, and the requirement information of the memory dimension.

[0132] The demand information of one dimension may include multiple demand sub-information. For example, the demand information may include one or more of the demand information of the GPU dimension, the demand information of the CPU dimension, and the demand information of the memory dimension. The description of the demand sub-information may be equivalent to the description of the resource sub-information, which will not be repeated here.

[0133] There are multiple ways to compare the requirement information of the continuous integration task in each dimension with the information of the node cluster, which are described below. The first comparison method and the method of selecting the first node cluster under the first comparison method are described below.

[0134] In some embodiments, for the first comparison method, for each node cluster, it can be determined whether the dimensions of the demand information and the node cluster information are consistent. If the number of dimensions of the node cluster information is greater than or equal to the number of dimensions of the demand information, it can be compared whether the dimensions of the demand information are consistent with the dimensions of the node cluster information, that is, it is determined whether each dimension of the demand information has a consistent dimension of the node cluster information.

[0135] When the dimension of the requirement information is inconsistent with the dimension of the node cluster information, or when the number of dimensions of the node cluster information is less than the number of dimensions of the requirement information, it means that the node cluster information cannot meet the requirement information. Therefore, the comparison result can indicate that the node cluster information cannot meet the requirement information, and further indicates that the node cluster corresponding to the node cluster information is not the first node cluster adapted to the continuous integration task.

[0136] When the dimension of the demand information is consistent with the dimension of the node cluster information, for each demand information and node cluster information of the same dimension, it is possible to compare whether the resource sub-information under the dimension meets the demand sub-information. If the resource sub-information under each dimension meets the demand sub-information, it means that the node cluster information can meet the demand information. Therefore, the comparison result can indicate that the node cluster information can meet the demand information.

[0137] If there is a node cluster whose sub-information under each dimension meets the required sub-information, then the node cluster can be determined as the first node cluster adapted to the continuous integration task. If there are multiple node clusters whose sub-information under each dimension meets the required sub-information, then the first node cluster adapted to the continuous integration task can be selected according to the pre-set selection conditions. The pre-set selection conditions can be the largest memory, the highest number of CPU cores, etc., which can be set according to actual usage requirements.

[0138] If there is no node cluster whose sub-information under each dimension satisfies the required sub-information, the first weight of each dimension and / or the second weight of each required sub-information can be obtained. Correspondingly, the dimensions whose first weights are less than the first weight threshold can be deleted, and / or the sub-information whose second weights are less than the second weight threshold can be deleted to obtain updated required sub-information.

[0139] Then, for each updated requirement information and node cluster information of the same dimension, the sub-information of the node cluster under the dimension can be compared to see whether it meets the updated requirement sub-information. If the sub-information of the node cluster under each dimension meets the updated requirement sub-information, it means that the information of the node cluster can meet the requirement information. Therefore, the comparison result can indicate that the information of the node cluster can meet the requirement information. Based on the comparison result, the first node cluster that is suitable for the continuous integration task can be selected from multiple node clusters.

[0140] If there is no node cluster whose sub-information under each dimension satisfies the required sub-information, the number of dimensions whose sub-information of the node cluster satisfies the required sub-information can also be obtained, and the number of dimensions whose sub-information of the node cluster satisfies the required sub-information can be used as the comparison result, and the comparison result can indicate that the information of the node cluster partially satisfies the required information. Then, among the multiple node clusters, the node cluster whose sub-information of the node cluster satisfies the required sub-information in the largest number of dimensions can be determined as the first node cluster adapted to the continuous integration task.

[0141] For the first comparison method, a node cluster whose comparison result representation information meets the required information can be selected from multiple node clusters as the first node cluster adapted to the continuous integration task. In this way, the precise first node cluster can be determined. The first node cluster is used to process the continuous integration task, which can improve the efficiency of processing the continuous integration task.

[0142] The second comparison method and the method of selecting the first node cluster under the second comparison method are described below. The dimension priority of the demand information of each dimension can be obtained. The dimension priority can be preset or determined according to the type of continuous integration task, which is reasonable.

[0143] For each dimension of demand information, the demand information of the dimension can be compared with the information of the node cluster of the same dimension to obtain the comparison sub-result of the demand information of the dimension, which is equivalent to obtaining the comparison sub-result of the demand information of each dimension. The comparison sub-result is used to characterize whether the information of the node cluster of a single dimension can meet the demand information.

[0144] After obtaining the comparison sub-results, the comparison sub-results of each demand information can be combined to determine the comparison result. If each comparison sub-result represents that the information of the node cluster can meet the demand information, then the comparison result can represent that the information of the node cluster can meet the demand information, that is, the comparison result represents that the information meets the demand information whose dimension priority meets the priority condition. For this case, the method of selecting the first node cluster can be specifically referred to the description of the first comparison method.

[0145] If there is a comparison sub-result that indicates that the information of the node cluster cannot meet the required information, the comparison result can be determined according to the dimension priority and the comparison sub-result. If the comparison sub-results whose dimension priority is greater than the priority threshold all indicate that the information of the node cluster can meet the required information, then it is determined that the comparison result can indicate that the information of the node cluster meets the required information whose dimension priority meets the priority condition. Among them, the priority condition is that the priority is greater than the priority threshold.

[0146] If the comparison result indicates that the number of node clusters whose information satisfies the dimension priority and whose demand information meets the priority condition is 1, the node cluster may be determined as the first node cluster adapted to the continuous integration task.

[0147] If the comparison result represents that the information of the node clusters satisfies the dimension priority and the number of node clusters whose demand information meets the priority conditions is multiple, the dimension priority order can be obtained, and the dimension priority order is used to determine in turn whether the information of the node clusters represented by the comparison sub-results can meet the demand information. The node cluster with the most demand information represented by the comparison sub-results in the dimension priority order will be used as the first node cluster for adapting the continuous integration task, which is equivalent to being able to select the first node cluster for adapting the continuous integration task from multiple node clusters.

[0148] For example, the comparison result A of node cluster A and the comparison result B of node cluster both represent that the information of the node clusters meets the requirement information that the dimension priority meets the priority condition. The comparison result A has two comparison sub-results, namely, the comparison sub-result A1 and the comparison sub-result A2, and the dimension priority of the comparison sub-result A1 is greater than the dimension priority of the comparison sub-result A2. The comparison result B has two comparison sub-results, namely, the comparison sub-result B1 and the comparison sub-result B2, and the dimension priority of the comparison sub-result B1 is equal to the dimension priority of the comparison sub-result A1 and greater than the dimension priority of the comparison sub-result B2. The dimension priority of the comparison sub-result B2 is less than the dimension priority of the comparison sub-result A2, and the dimension priority of the comparison sub-result A1 is greater than the priority threshold.

[0149] If the information of the node cluster of the corresponding dimension represented by the comparison sub-result A1 can meet the required information, but the information of the node cluster of the corresponding dimension represented by the comparison sub-result B1 cannot meet the required information, then node cluster A is the node cluster that can meet the most required information according to the dimension priority order and the information of the node cluster represented by the comparison sub-result can meet the required information. Therefore, node cluster A can be selected as the first node cluster.

[0150] If comparison sub-result A1 and comparison sub-result B1 both represent that the information of the node cluster of the corresponding dimension can meet the required information, comparison sub-result A2 represents that the information of the node cluster of the corresponding dimension cannot meet the required information, but comparison sub-result B2 represents that the information of the node cluster of the corresponding dimension can meet the required information, then node cluster B is the node cluster whose information can meet the most required information according to the dimension priority order and whose comparison sub-result represents that the node cluster information can meet the most required information, so node cluster B can be selected as the first node cluster.

[0151] If comparison sub-result A1 and comparison sub-result B1 both represent that the information of the node cluster of the corresponding dimension can meet the required information, and comparison sub-result A2 and comparison sub-result B2 both represent that the information of the node cluster of the corresponding dimension can meet the required information, since the priority of comparison sub-result B2 is lower than the dimension priority of comparison sub-result A2, it can be seen that node cluster A is the node cluster with the most required information according to the dimension priority order and the comparison sub-result represents that the information of the node cluster can meet the required information. Therefore, node cluster A can be selected as the first node cluster.

[0152] For the second comparison method, a node cluster whose comparison result representation information meets the dimension priority and the requirement information that meets the priority conditions can be selected from multiple node clusters as the first node cluster adapted to the continuous integration task. In this way, the precise first node cluster can be determined. The first node cluster is used to process the continuous integration task, which can improve the efficiency of processing the continuous integration task.

[0153] If the selection method is the third selection method, see Figure 6 , Figure 6 This is a flowchart of the task scheduling method provided in the embodiment of the present application. Figure 4 , the following combination Figure 6 The method of selecting the first node cluster when the selection method is the third selection method is described.

[0154] In step 1031B, if the selection method is the third selection method, for each node cluster, a method for determining the score of the node cluster is determined according to the information of the node cluster.

[0155] The node cluster information includes resource types, which can be one or more of node resource types, node types, region types, isolation types, environment types, and node types with cache. Each resource type includes multiple resource subtypes. For example, the node resource type includes GPU subtype, CPU subtype, and memory subtype. The node type includes Linux subtype, macOS subtype, and Windows subtype.

[0156] In some embodiments, for each node cluster, a score determination method of the node cluster is determined according to the resource type. The score determination method includes a score calculation algorithm and a weight corresponding to each resource type, and the weight corresponding to each resource type may include a weight of the resource type and a weight of a resource subtype under the resource type.

[0157] In step 1032B, a first score of the node cluster is calculated based on the score determination method and the information of the node cluster.

[0158] After obtaining the score determination method, the first score of the node cluster can be calculated based on the score determination method and the information of the node cluster. In some embodiments, the information of the node cluster can be converted into a score, and then the weight corresponding to each resource type is multiplied by the score under the resource type to obtain a score product, and then the score calculation algorithm can be used to process the score product, so as to calculate the first score of the node cluster.

[0159] The way of converting the information of the node cluster into a score can be set according to actual use requirements. For example, the memory is x, which can be converted into a score a. In some embodiments, the score calculation algorithm can be a weighted sum, a simulated annealing algorithm, a genetic algorithm, an evolutionary algorithm, etc.

[0160] The following is an example of how to calculate the first score. Resource types include resource type a and resource type b. Resource type b includes resource subtype b1 and resource subtype b2. The information corresponding to resource type a can be converted into score 1, and the information corresponding to resource subtype b1 and resource subtype b2 can be converted into scores, corresponding to scores 2 and 3.

[0161] The weights corresponding to the resource types include weight a of resource type a, weight b of resource type b, weight b1 of resource subtype b1 under resource type b, and weight b2 of resource subtype b2 under resource type b. Score 1 can be multiplied by weight a to obtain product 1, score 2 can be multiplied by weight b1 to obtain product 2, and score 3 can be multiplied by weight b2 to obtain product 3. The calculation formula is to add product 2 and product 3 to obtain a sum value 1, multiply the sum value 1 by weight b to obtain product 4, and add product 3 and product 4 to obtain the first score.

[0162] In some embodiments, the information of the node cluster may be filtered according to the resource type, and the resource type of the filtered information of the node cluster may be used as the first resource type. The score determination method of the node cluster may be determined according to the first resource type.

[0163] The following describes a method for filtering node cluster information based on resource types. In order to improve the efficiency of processing continuous integration tasks, the information of node clusters other than node resource types and cached node types can be deleted. That is, the filtered node cluster information is the information of node clusters of node resource types and / or the information of node clusters of cached node types.

[0164] The first resource type may be a node resource type and / or a node type with a cache. According to the first resource type, a score determination method for the node cluster is determined. The score determination method includes a score calculation algorithm and a weight corresponding to each first resource type. The weight corresponding to each first resource type may include a weight of the first resource type and a weight of the first resource subtype under the first resource type. The calculation method can refer to the description of the first score of the above-mentioned calculation node cluster, which will not be repeated here.

[0165] In some embodiments, since the node cluster includes multiple nodes, the node cluster can obtain the node score of each working node included in the node cluster according to a preset conversion rule, and send the node score and the resource type for calculating the node score to the server.

[0166] After obtaining the node scores and resource types sent by each node cluster, the server may determine a score determination method for the node cluster according to the resource type, and further calculate a first score for the node cluster according to the node scores and the resource type for calculating the node scores.

[0167] The node cluster calculates the node score by converting the information corresponding to the node into a score, and then multiplying the weight corresponding to each resource type by the score under the resource type to obtain the node score. In some embodiments, the node score can also be calculated by the server.

[0168] The following example illustrates the method of calculating the first score of a node cluster. The node scores include score 4, score 5, and score 6. The nodes corresponding to score 4, score 5, and score 6 are all different. The resource type for calculating score 4 and score 5 is memory type, specifically memory utilization. Score 6 is a node type with cache. The score determination method includes the weight of the memory type and the weight of the node type with cache. The score calculation algorithm is weighted sum. Therefore, score 4 can be multiplied by the weight of the memory type to obtain product 5, score 5 can be multiplied by the weight of the memory type to obtain product 6, score 6 can be multiplied by the weight of the node type with cache to obtain product 7, and product 5, product 6, and product 7 are added together to obtain the first score of the node cluster. In this way, an accurate first score can be obtained.

[0169] In step 1033B, based on the first scores of the node clusters, a first node cluster suitable for the continuous integration task is selected from the multiple node clusters.

[0170] After calculating the first score of the node cluster, a first node cluster adapted to the continuous integration task can be selected from the node cluster based on the first score of the node cluster. In some embodiments, any node cluster whose first score is greater than a preset score threshold can be selected as the first node cluster adapted to the continuous integration task.

[0171] In some embodiments, the node clusters may be sorted based on the first score to obtain a node cluster order, thereby selecting the node cluster that is first in the node cluster order as the first node cluster that is adapted to the continuous integration task. In this way, the first node cluster that is adapted to the continuous integration task can be accurately determined, and the continuous integration task can be scheduled to the first node cluster, which can improve the accuracy of task scheduling.

[0172] If the selection method is the fourth selection method, the cluster where the construction node associated with the first task label is located can be determined, and the cluster where the construction node corresponding to the first task label is located can be selected from multiple node clusters as the first node cluster adapted for the continuous integration task. In this way, the first node cluster adapted for the continuous integration task can be accurately determined, and the continuous integration task can be scheduled to the first node cluster, which can improve the accuracy of task scheduling.

[0173] In step 104, the continuous integration task is scheduled to the first node cluster.

[0174] In some embodiments, step 104 may be implemented in the following manner: after selecting the first node cluster, the server may obtain the address of the first node cluster, thereby scheduling the continuous integration task to the first node cluster.

[0175] In some embodiments, an agent node is configured in each node cluster, and the server can obtain the address of the agent node, and then send a task scheduling request for the continuous integration task to the agent node according to the address of the agent node.

[0176] The proxy node is connected to the cluster interface (API) of the node cluster. After receiving the task scheduling request of the continuous integration task, the proxy node can call the cluster interface to create a pipeline task of the continuous integration task according to the task parameters of the continuous integration task. Among them, the task parameters of the continuous integration task may include the name of the application, the warehouse address, the build command, the image name, etc.

[0177] The node cluster includes multiple worker nodes, and a worker node includes one or more container groups (pods). In the process of creating a pipeline task for a continuous integration task, you can also add a container group that executes the pipeline task for the continuous integration task, and schedule the container group to the worker node through the scheduler. Start the container group to execute the pipeline task for the continuous integration task. The scheduler and cluster interface are located in the master node of the node cluster.

[0178] In some embodiments, a pipeline task is to execute multiple subtasks included in a continuous integration task in a preset order. For example, a continuous integration task may include a code pulling subtask, a dependency pulling subtask, a compilation and construction subtask, and an upload product subtask. The pipeline task is to first execute the pulling subtask, then the dependency pulling subtask, then the compilation and construction subtask, and finally the upload product subtask.

[0179] In some embodiments, the pipeline task of the agent node calling the cluster interface to create a continuous integration task can be implemented by a continuous integration tool. The continuous integration tool can be Tekton, Jenkins, Argo, etc., which can be specifically set according to actual usage requirements.

[0180] For example, the continuous integration tool may be Tekton. For a continuous integration task, a task resource (Task) in Tekton may be created. A task resource includes multiple steps (Step), and one step corresponds to one subtask.

[0181] For a continuous integration task, you can create a container group to complete the continuous integration task. In other words, you can use a container group to process the continuous integration task. For each subtask in a continuous integration task, you can create a container group to complete the subtask. In other words, for a continuous integration task, you can create multiple container groups. The number of container groups is the same as the number of subtasks. In other words, you can use multiple container groups to process the continuous integration task.

[0182] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of the task scheduling system provided in the embodiment of the present application. Figure 2 , the following combination Figure 7 The implementation of the task scheduling system and step 104 is described. The scheduling server 200 is a server, and the scheduling server 200 can obtain the address of the proxy node, for example, the address of the proxy node of the node cluster 400-1 can be pulled, or the address of the proxy node of the node cluster 400-m can be pulled. m is an integer greater than or equal to 2.

[0183] The scheduling server 200 may send a task scheduling request for a continuous integration task to the proxy node according to the address of the proxy node. For example, the scheduling server 200 may send a task scheduling request for a continuous integration task to the proxy node of the node cluster 400-1 according to the address of the proxy node of the node cluster 400-1. Or, for another example, the scheduling server 200 may send a task scheduling request for a continuous integration task to the proxy node of the node cluster 400-m according to the address of the proxy node of the node cluster 400-m.

[0184] The node cluster 400-1 includes p working nodes, where p is an integer greater than or equal to 2. Figure 7 Only worker node 1 and worker node p are shown in FIG. Worker node 1 includes 5 container groups. Figure 7 Only container groups 1 and 5 are shown in FIG. 1 . The working node p includes 7 container groups. Figure 7 Only container group 1 and container group 7 are shown.

[0185] The node cluster 400-m includes q working nodes, where q is an integer greater than or equal to 2. Figure 7 Only worker node 1 and worker node q are shown in FIG. Worker node 1 includes 8 container groups. Figure 7 Only container groups 1 and 8 are shown in FIG. 1 . The working node q includes three container groups. Figure 7 Only container group 1 and container group 3 are shown.

[0186] The proxy node of node cluster 400-1 is connected to the cluster interface of node cluster 400-1, and the proxy node of node cluster 400-m is connected to the cluster interface of node cluster 400-m. The following example illustrates the processing flow after the proxy node receives a task scheduling request in conjunction with the proxy node of node cluster 400-1. The process executed by the proxy node of node cluster 400-m is consistent with the process executed by the proxy node of node cluster 400-1, and will not be repeated here.

[0187] After receiving the task scheduling request of the continuous integration task, the agent node of the node cluster 400 - 1 may call the cluster interface of the node cluster 400 - 1 to create a pipeline task of the continuous integration task according to the task parameters of the continuous integration task.

[0188] The cluster interface of node cluster 400-1 is connected to the continuous integration tool, and a pipeline task can be created through the continuous integration tool. In the process of creating the pipeline task of the continuous integration task, container group 1-container group 5 for executing the pipeline task of the continuous integration task can also be added, and container group 1-container group 5 can be scheduled to working node 1. By starting container group 1-container group 5, the pipeline task of the continuous integration task can be executed.

[0189] In some embodiments, if the fourth selection method is used to determine the first node cluster, that is, the node cluster where the construction node is located is used as the first node cluster for adapting the continuous integration task, affinity configuration information can be included in the task scheduling request, and then the affinity configuration information can be configured in the container group, thereby scheduling the container group to the construction node in the first node cluster.

[0190] In some embodiments, when scheduling the container group to the build node, since the build node with the cache can call the cache, the time for executing the continuous integration task can be reduced.

[0191] In some embodiments, the container group can also be scheduled to a working node with the maximum node score or a node score greater than a preset node score threshold based on the node score, thereby preventing continuous integration tasks with a number greater than the preset threshold from being located on the same node in the same cluster.

[0192] The task scheduling method provided by the embodiment of the present application can determine the selection method of the node cluster based on the task label of the continuous integration task, and the association relationship between the pre-set candidate task label and the candidate construction node, and then can select the first node cluster adapted to the continuous integration task from multiple node clusters based on the selection method, so that the continuous integration task can be scheduled to the first node cluster. The present application involves multiple node clusters. In the process of performing task scheduling, the first node cluster can be selected from multiple node clusters. Compared with the solution of scheduling tasks with a single node cluster, the fault tolerance rate of the present application is higher. In addition, the present application can determine the screening method based on the task label of the continuous integration task, and then can select the first node cluster based on the screening method, which can improve the accuracy of task scheduling and can effectively utilize multiple node clusters.

[0193] Furthermore, the task scheduling method provided in the embodiment of the present application can meet the requirements of different continuous integration tasks by being compatible with different continuous integration tasks. This means that compared with the related art, the present application can be applied to continuous integration tasks corresponding to multiple different types of task tags.

[0194] In addition, the task scheduling method provided in the embodiment of the present application can accurately determine the cached build nodes corresponding to the continuous integration task, so as to accurately determine the first node cluster adapted for the continuous integration task. Since the cached build nodes can call the cache, the time for executing the continuous integration task can be reduced.

[0195] The task scheduling method provided in the embodiment of the present application can also first determine the node cluster selection method through the task label, so as to select the first node cluster from the node cluster according to the selection method, thereby avoiding that the continuous integration tasks with a number greater than the preset threshold are located in the same cluster, or on the same node in the same cluster, and effectively utilizing the resources of the node cluster.

[0196] For example, faced with the problem that working nodes in different regions are not located in the same node cluster and GPUs of different architectures cannot belong to the same node cluster, the relevant technology cannot meet the continuous integration tasks that require the architecture of the region and GPU. The task scheduling method provided in the embodiment of the present application can realize the scheduling of continuous integration tasks, thereby realizing the processing of continuous integration tasks.

[0197] The proxy node also has the following uses, which are described in detail below. In some embodiments, the proxy node can send information about the node cluster to the server. In some embodiments, the proxy node can calculate the node score of each working node in the Kubernetes cluster where the proxy node is located, and send the node score and the resource type of the node score to the server. The embodiment of the present application provides a server-agent architecture that can solve the management problem of multiple Kubernetes clusters.

[0198] See also Figure 8 , Figure 8 This is a flowchart of the task scheduling method provided in the embodiment of the present application. Figure 5 , the number of node clusters is multiple, for the case where the selection method is the third selection method, in Figure 3 After step 104 shown, steps 105 to 107 may also be performed.

[0199] In step 105, if the execution failure information sent by the first node cluster is received, for each node cluster, based on the adjustment information in the execution failure information, the score determination method and the information of the node cluster, a second score of the node cluster is calculated.

[0200] The execution failure information is used to indicate that the first node cluster failed to execute the continuous integration task. In some embodiments, the execution failure information is information sent by the agent node of the first node cluster to the server, and the adjustment information in the failure information is determined by the agent node according to the number of execution failures recorded by the continuous integration task.

[0201] In some embodiments, the adjustment information may be a score adjustment weight, and the first score calculated by the score determination method and the information of the node cluster may be multiplied by the score adjustment weight, so as to calculate the second score of the node cluster. The greater the number of execution failures, the smaller the score adjustment weight, and accordingly, the smaller the second score of the node cluster. In this way, the score of the node cluster can be reduced, which is equivalent to reducing the probability of the node cluster being selected.

[0202] In some embodiments, the adjustment information may also be a new weight corresponding to each resource type, and the weight corresponding to each resource type included in the score determination method may be replaced with the new weight corresponding to each resource type, thereby calculating the second score of the node cluster based on the score calculation algorithm, the new weight corresponding to each resource type, and the information of the node cluster. The specific calculation method may be equivalent to the method for calculating the first score of the node cluster, which will not be described in detail here.

[0203] The greater the number of execution failures, the smaller the new weight corresponding to each resource type, and accordingly, the smaller the second score of the node cluster. In this way, the score of the node cluster can be reduced, which is equivalent to reducing the probability of the node cluster being selected.

[0204] In some embodiments, the adjustment information may also be for the adjustment value of the weight corresponding to each resource type, and based on the adjustment value of the weight corresponding to each resource type, the weight corresponding to each resource type in the score determination method may be adjusted to obtain the adjustment weight corresponding to each resource type, and then based on the score calculation algorithm, the adjustment weight corresponding to each resource type and the information of the node cluster, the second score of the node cluster is calculated. The specific calculation method can be equivalent to the method of calculating the first score of the node cluster, which will not be repeated here.

[0205] The greater the number of execution failures, the smaller the adjustment weight corresponding to each resource type, and accordingly, the smaller the second score of the node cluster. In this way, the score of the node cluster can be reduced, which is equivalent to reducing the probability of the node cluster being selected.

[0206] In step 106 , based on the second score of the node cluster, a second node cluster suitable for the continuous integration task is selected from the multiple node clusters.

[0207] Step 106 and Figure 3 The method of selecting the first node cluster in step 103 is consistent. For details, please refer to the corresponding description of step 103, and the corresponding content of step 103 is equivalently replaced, which is not repeated here.

[0208] In step 107, the continuous integration task is scheduled to the second node cluster. Figure 3 The method of scheduling the continuous integration task to the first node cluster in step 107 is consistent. For details, please refer to the corresponding description of step 107, and the corresponding content of step 107 is equivalently replaced, which is not repeated here.

[0209] In some embodiments, before executing step 105, the number of execution failures can be compared with a failure threshold. If the number of execution failures is greater than or equal to the failure threshold, steps 105-107 can be executed. If the number of execution failures is less than the failure threshold, the node cluster order obtained by sorting the node clusters based on the first score can be used as the second node cluster, thereby scheduling the continuous integration task to the second node cluster.

[0210] In the case where the selection method is the fourth selection method, if the construction node corresponding to the first task tag does not exist in the pre-stored association relationship between the candidate task tag and the candidate construction node, or if the execution failure information of the first node cluster selected by the fourth selection method is received, at least one of the first selection method-the third selection method can be used to select a node cluster (equivalent to the second node cluster), and the continuous integration task can be scheduled to the second node cluster selected by the first selection method.

[0211] By determining the second node cluster in the above manner, it is possible to schedule the continuous integration task to the second node cluster when the first node cluster is unavailable or the first node cluster is unable to execute the continuous integration task. This can improve the fault tolerance rate and avoid the risk of the continuous integration task being unable to be executed due to cluster unavailability. In other words, the task scheduling system provided in the embodiment of the present application has fault tolerance and high availability.

[0212] Combine the following Figure 9-11 The task scheduling method provided in the embodiment of the present application is illustrated by way of example. Fig. 9 This is a flowchart of the task scheduling method provided in the embodiment of the present application. Figure 6 , Fig.10 is a schematic diagram of the structure of the server provided in the embodiment of the present application, Fig.11 This is a flowchart of the task scheduling method provided in the embodiment of the present application. Figure 7 .

[0213] See also Fig. 9 In step 901, in response to the task processing request of the continuous integration task, the task tag of the continuous integration task is determined. The task tag includes a first task tag and a second task tag. For step 901, see Figure 3 The description of step 101 is not repeated here.

[0214] In step 902, the node cluster information sent by the proxy node is received.

[0215] Step 902 can be implemented in the following way: the server can obtain the online list of the proxy node and send a node cluster information acquisition request to the online proxy node. After receiving the node cluster information acquisition request, the proxy node can obtain the information of the Kubernetes cluster where the proxy node is located (i.e., the node cluster information) in real time, and send the node cluster information to the server.

[0216] In some embodiments, the proxy node may also calculate the node score of each working node in the Kubernetes cluster where the proxy node is located, and may also send the node score and the resource type of the node score to the server.

[0217] In step 903, based on the first task label, a first node cluster adapted to the continuous integration task is selected from a plurality of node clusters for processing the continuous integration task.

[0218] For step 903, based on the first task tag, the construction node corresponding to the first task tag can be determined from the association relationship between the pre-stored candidate task tags and the candidate construction nodes. From multiple node clusters for processing continuous integration tasks, the node cluster where the construction node is located is determined, and the node cluster where the construction node is located is used as the first node cluster adapted for the continuous integration task.

[0219] In step 904, the continuous integration task is scheduled to the first node cluster.

[0220] For step 904, see Figure 3 If the first node cluster is not selected through step 903, or if the first node cluster is determined through step 904 and execution failure information sent by the first node cluster is received after step 904, step 905 is executed.

[0221] In step 905, at least one of the first selection method, the second selection method, and the third selection method is adopted to select a first node cluster adapted to the continuous integration task from multiple node clusters.

[0222] For step 905, the first selection method is to select a node cluster based on the task label and the cluster label of the node cluster. The second selection method is to select a node cluster based on the requirement information of the continuous integration task and the information of the node cluster. The third selection method is to select a node cluster based on the score of the node cluster determined by the information of the node cluster.

[0223] If the first node cluster is determined by step 905, and after step 904, the execution failure information sent by the first node cluster is received, the execution Figure 8For details of steps 105 to 107, please refer to Figure 8 The description of steps 105 to 107 is not repeated here.

[0224] In some embodiments, there is an execution order between the first selection method and the fourth selection method, and the execution order can be set according to actual usage requirements. Fig.10 The scheduling server 200 (i.e., the server) includes a pre-filter 1001, an extended filter 1002, a pre-classifier 1003, and an extended classifier 1004, wherein the pre-filter 1001 selects the first node cluster using the fourth selection method, and the pre-filter 1001 can also select the sixth cluster node using the first selection method and the second selection method.

[0225] The extended filter 1002 is used for providing an extended filtering method based on the sixth node cluster selected by the pre-filter 1001 according to actual usage requirements. For example, the extended filtering method requires exclusive GPU so that the seventh node cluster can be selected.

[0226] The pre-classifier 1003 is used to select the eighth node cluster from the seventh node cluster in the third selection method. The extended classifier 1004 is used to perform further classification methods based on the eighth node cluster selected by the pre-classifier 1003. The extended classifier 1004 can calculate scores by using simulated annealing algorithm, genetic algorithm, evolutionary algorithm, etc.

[0227] For example, see Fig.11 In step 1101, the server receives the task processing request and the node cluster information sent by the proxy node. In step 1102, the server screens the node clusters through a pre-filter to obtain the first node cluster or the sixth node cluster.

[0228] With respect to step 1102 , if the pre-filter can select the first node cluster using the fourth selection method, if the pre-filter cannot determine the first node cluster using the fourth selection method, the first selection method and the second selection method can be used to select the sixth node cluster.

[0229] In step 1103, if the first node cluster is obtained, the continuous integration task is scheduled to the first node cluster. After step 1103, if the continuous integration task is successfully executed, the process ends; if the continuous integration task fails, step 1108 is executed.

[0230] In step 1104, if the sixth node cluster is obtained, the server screens the sixth node cluster through the extended filter to obtain the seventh node cluster. In step 1105, the server calculates the first score of each seventh node cluster through the pre-classifier.

[0231] In step 1106, the server calculates a third score of the seventh node cluster by using the extended classifier. The third score is a score calculated by using a simulated annealing algorithm, a genetic algorithm, or an evolutionary algorithm. In step 1107, the server selects a first node cluster from the seventh node cluster according to the first score or the third score. After step 1107, step 1103 may be performed.

[0232] In step 1108, if the execution failure information sent by the first node cluster is received, the server selects a second node cluster from the screened node clusters according to the first score or the third score, and schedules the continuous integration task to the second node cluster.

[0233] The following is a description of an exemplary structure of the task scheduling device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the task scheduling device 455 of the memory 450 may include:

[0234] A response module 4551 is used to respond to a task processing request of a continuous integration task and determine a task tag of the continuous integration task;

[0235] A selection module 4552 is used to determine a selection method of a node cluster for processing the continuous integration task based on the task label and the association relationship between the preset candidate task labels and the candidate build nodes;

[0236] The selection module 4552 is further configured to select, based on the selection method, a first node cluster adapted to the continuous integration task from a plurality of node clusters;

[0237] The scheduling module 4553 is used to schedule the continuous integration task to the first node cluster.

[0238] In some embodiments, the selection module 4552 is also used to search for the construction node associated with the task tag from the association relationship between the pre-set candidate task tags and the candidate construction nodes to obtain the search result; based on the search result, determine the selection method of the node cluster for processing the continuous integration task.

[0239] In some embodiments, the selection module 4552 is also used to, if the selection method is the first selection method, determine the node cluster whose task label is consistent with the cluster label of the node cluster as the first node cluster adapted to the continuous integration task; or, based on the identifier of the node cluster and the cluster label of the node cluster, determine the cluster sub-label of the node cluster, and determine the node cluster whose task label is consistent with the cluster sub-label as the first node cluster adapted to the continuous integration task.

[0240] In some embodiments, the selection module 4552 is also used to compare the demand information of the continuous integration task in each dimension with the information of the node cluster for each node cluster if the selection method is the second selection method, to obtain a comparison result for each dimension; based on the comparison result, select a first node cluster that is adapted to the continuous integration task from the multiple node clusters.

[0241] In some embodiments, the selection module 4552 is also used to select, from the multiple node clusters, a node cluster whose comparison result represents that the information satisfies the requirement information, as the first node cluster adapted to the continuous integration task; or, from the multiple node clusters, select a node cluster whose comparison result represents that the information satisfies the requirement information whose dimension priority meets the priority condition, as the first node cluster adapted to the continuous integration task.

[0242] In some embodiments, the selection module 4552 is also used to determine, for each of the node clusters, a score determination method for the node cluster based on information of the node cluster if the selection method is the third selection method; calculate a first score for the node cluster based on the score determination method and information of the node cluster; and select a first node cluster that is adapted to the continuous integration task from a plurality of the node clusters based on the first score of the node cluster.

[0243] In some embodiments, the selection module 4552 is further configured to, after scheduling the continuous integration task to the first node cluster, if execution failure information sent by the first node cluster is received, calculate a second score of the node cluster for each node cluster based on the adjustment information in the execution failure information, the score determination method, and information of the node cluster; and select a second node cluster adapted to the continuous integration task from the plurality of node clusters based on the second score of the node cluster;

[0244] The scheduling module 4553 is further used to schedule the continuous integration task to the second node cluster.

[0245] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the task scheduling method described in the embodiment of the present application.

[0246] The present application embodiment provides a computer-readable storage medium, in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will be caused to execute the task scheduling method provided by the present application embodiment, for example, Figure 3 The task scheduling method is shown.

[0247] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0248] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0249] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0250] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0251] The task scheduling method provided by the embodiment of the present application can determine the selection method of the node cluster based on the task label of the continuous integration task, and the association relationship between the pre-set candidate task label and the candidate construction node, and then can select the first node cluster adapted to the continuous integration task from multiple node clusters based on the selection method, so that the continuous integration task can be scheduled to the first node cluster. The present application involves multiple node clusters. In the process of performing task scheduling, the first node cluster can be selected from multiple node clusters. Compared with the solution of scheduling tasks with a single node cluster, the fault tolerance rate of the present application is higher. In addition, the present application can determine the screening method based on the task label of the continuous integration task, and then can select the first node cluster based on the screening method, which can improve the accuracy of task scheduling and can effectively utilize multiple node clusters.

[0252] Furthermore, the task scheduling method provided in the embodiment of the present application can meet the requirements of different continuous integration tasks by being compatible with different continuous integration tasks. This means that compared with the related art, the present application can be applied to continuous integration tasks corresponding to multiple different types of task tags.

[0253] In addition, the task scheduling method provided in the embodiment of the present application can accurately determine the construction node of the cache corresponding to the continuous integration task, so as to accurately determine the first node cluster adapted to the continuous integration task, because the construction node with the cache can call the cache, so the time for executing the continuous integration task can be reduced. The above is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements and improvements made within the spirit and scope of the present application are all included in the scope of protection of the present application.

Claims

1. A task scheduling method, characterized in that: The method comprises: In response to a task processing request of a continuous integration task, determining a task tag of the continuous integration task; Based on the task label and the association between the preset candidate task labels and the candidate build nodes, determine a method for selecting a node cluster for processing the continuous integration task; Based on the selection method, selecting a first node cluster adapted to the continuous integration task from multiple node clusters; The continuous integration task is scheduled to the first node cluster.

2. The method according to claim 1, characterized in that The method of determining the selection method of the node cluster for processing the continuous integration task based on the task label and the association relationship between the preset candidate task label and the candidate build node includes: From the association relationship between the preset candidate task tags and the candidate construction nodes, searching for the construction node associated with the task tag to obtain a search result; Based on the search result, a method for selecting a node cluster for processing the continuous integration task is determined.

3. The method according to claim 1, characterized in that The selecting, based on the selection method, a first node cluster adapted to the continuous integration task from a plurality of node clusters includes: If the selection method is the first selection method, the node cluster whose task label is consistent with the cluster label of the node cluster is determined as the first node cluster adapted to the continuous integration task; or, Based on the identifier of the node cluster and the cluster label of the node cluster, a cluster sub-label of the node cluster is determined, and a node cluster having the task label consistent with the cluster sub-label is determined as a first node cluster adapted to the continuous integration task.

4. The method according to claim 1, characterized in that: The selecting, based on the selection method, a first node cluster adapted to the continuous integration task from a plurality of node clusters includes: If the selection method is the second selection method, for each of the node clusters, the requirement information of the continuous integration task in each dimension is compared with the information of the node cluster to obtain a comparison result in each dimension; Based on the comparison result, a first node cluster adapted to the continuous integration task is selected from the multiple node clusters.

5. The method according to claim 4, characterized in that The selecting, based on the comparison result, a first node cluster adapted to the continuous integration task from the plurality of node clusters comprises: From the plurality of node clusters, a node cluster whose information is represented by the comparison result to satisfy the required information is selected as a first node cluster adapted to the continuous integration task; or, From the multiple node clusters, a node cluster whose comparison result represents that the information satisfies the requirement information of the dimension priority and meets the priority condition is selected as the first node cluster adapted to the continuous integration task.

6. The method according to claim 1, characterized in that The selecting, based on the selection method, a first node cluster adapted to the continuous integration task from a plurality of node clusters includes: If the selection method is the third selection method, for each of the node clusters, a score determination method of the node cluster is determined according to the information of the node cluster; Calculate a first score of the node cluster based on the score determination method and the information of the node cluster; Based on the first score of the node cluster, a first node cluster suitable for the continuous integration task is selected from the multiple node clusters.

7. The method according to claim 6, characterized in that After scheduling the continuous integration task to the first node cluster, the method further includes: If the execution failure information sent by the first node cluster is received, for each of the node clusters, based on the adjustment information in the execution failure information, the score determination method, and the information of the node cluster, calculating a second score of the node cluster; Based on the second score of the node cluster, selecting a second node cluster adapted to the continuous integration task from the multiple node clusters; The continuous integration task is scheduled to the second node cluster.

8. A task scheduling device, characterized in that: The device comprises: A response module, configured to respond to a task processing request of a continuous integration task and determine a task tag of the continuous integration task; A selection module, used to determine a selection method for a node cluster for processing the continuous integration task based on the task label and a pre-set association relationship between candidate task labels and candidate build nodes; The selection module is further used to select a first node cluster adapted to the continuous integration task from multiple node clusters based on the selection method; A scheduling module is used to schedule the continuous integration task to the first node cluster.

9. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; The processor is used to implement the task scheduling method described in any one of claims 1 to 7 when executing the computer executable instructions or computer programs stored in the memory.

10. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer programs are executed by a processor, the task scheduling method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer programs are executed by a processor, the task scheduling method according to any one of claims 1 to 7 is implemented.

Images