Resource allocation method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a resource allocation method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps that in response to an arrangement operation for an assembly line, an assembly line task is constructed and obtained, and the assembly line task comprises at least one execution step, target assembly line components associated with all the execution steps and target machine architectures corresponding to all the target assembly line components; obtaining a resource group corresponding to each execution step in the pipeline task, wherein the resource group comprises at least one building machine; target building machines corresponding to the assembly line tasks are determined from all the resource groups, corresponding task instructions are issued to all the target building machines based on the execution steps corresponding to all the resource groups, service software corresponding to the target assembly line assemblies is deployed in the target building machines, and the service software corresponds to a target machine architecture. By adopting the method, the resource utilization rate and the assembly line task execution efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a resource allocation method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] A build machine is a computer or server dedicated to compiling, building, and testing software code.

[0003] The traditional pipeline scheduling method is based on Jenkins' master (primary build machine) or slave (slave build machine) mode. This method can specify in the script to use the master or slave to execute pipeline tasks, or it can randomly assign tasks to a build machine.

[0004] In the process of specifying the build machine to execute the pipeline task in the above method, due to the diversity of pipeline tasks, it is easy to have the problem that the selected build machine does not meet the execution environment required by the pipeline task, resulting in the failure of the pipeline task to be executed smoothly, resulting in low task execution efficiency and waste of builder machine resources. Summary of the invention

[0005] Based on this, it is necessary to provide a resource allocation method, device, computer equipment, computer-readable storage medium and computer program product that can improve the resource utilization of the builder and improve the task execution efficiency in response to the above technical problems.

[0006] In a first aspect, the present application provides a resource allocation method, comprising:

[0007] In response to the orchestration operation for the pipeline, a pipeline task is constructed, wherein the pipeline task includes at least one execution step, a target pipeline component associated with each execution step, and a target machine architecture corresponding to each target pipeline component;

[0008] Obtain a resource group corresponding to each execution step in the pipeline task, wherein the resource group includes at least one builder;

[0009] Determine the target build machine corresponding to the pipeline task from each of the resource groups, and issue corresponding task instructions to each of the target build machines based on the execution steps corresponding to each of the resource groups. The target build machine is deployed with service software corresponding to the target pipeline component, and the service software corresponds to the target machine architecture.

[0010] In one embodiment, determining the target build machine corresponding to the pipeline task from each of the resource groups includes:

[0011] For any of the execution steps, determine at least one service software associated with the target pipeline component corresponding to the execution step, and use the build machine with the at least one service software deployed in the resource group corresponding to the execution step as a candidate build machine corresponding to the pipeline task;

[0012] A target build machine is determined from the candidate build machines based on the resource data of the candidate build machines.

[0013] In one embodiment, determining a target build machine from the candidate build machines based on the resource data of the candidate build machines includes:

[0014] Determine a target candidate builder machine whose resource data satisfies an execution condition from among the candidate builder machines;

[0015] A target build machine having the lowest load among the target candidate build machines is determined from the target candidate build machines.

[0016] In one embodiment, before issuing corresponding task instructions to each target builder based on the execution steps corresponding to each resource group, the method further includes:

[0017] Locking the target build machine;

[0018] After issuing corresponding task instructions to each target builder based on the execution steps corresponding to each resource group, the method further includes:

[0019] After receiving a successful response message for the task instruction, the lock of the target builder is released.

[0020] In one embodiment, the method further comprises:

[0021] In response to a grouping operation on the build machine, determining at least one resource group;

[0022] For any build machine in the resource group, after determining the service software corresponding to the build machine, issue an installation instruction to the build machine, wherein the installation instruction includes an installation package and an installation script of the service software, and the installation instruction is used to instruct the build machine to run the installation script to deploy the service software on the build machine based on the installation package;

[0023] In the maintenance data corresponding to the resource group, the correspondence between the construction machine and the deployed service software is recorded.

[0024] In one embodiment, in the component maintenance data, the correspondence between the component and the service software is recorded, and the determining of at least one service software associated with the target pipeline component corresponding to the execution step includes:

[0025] At least one service software associated with the target pipeline component corresponding to the execution step is determined from the component maintenance data.

[0026] In one embodiment, determining a target candidate builder whose resource data satisfies the execution condition from among the candidate builder machines includes:

[0027] If there is no target candidate build machine whose resource data satisfies the execution condition among the candidate build machines, displaying an execution condition adjustment page, wherein the execution condition is displayed on the execution condition adjustment page;

[0028] In response to the modification operation on the execution condition, the target candidate build machine is continuously determined from the candidate build machines based on the modified execution condition.

[0029] In one embodiment, the method further comprises:

[0030] If there is no target candidate builder whose resource data satisfies the execution condition among the candidate builder machines, then for any of the candidate builder machines, the data to be scheduled of the candidate builder machine is constructed based on the task data of each executed pipeline task in the candidate builder machine and the resource data of the candidate builder machine;

[0031] Acquire task data of the pipeline task, and input the task data of the pipeline task and the to-be-scheduled data of each candidate builder into a pre-trained scheduling model to obtain a scheduling plan;

[0032] The executed pipeline tasks in each of the candidate build machines are scheduled based on the scheduling scheme, and a target build machine is determined from the candidate build machines that have completed task scheduling.

[0033] In a second aspect, the present application further provides a resource allocation device, including:

[0034] An orchestration module, configured to construct a pipeline task in response to an orchestration operation on the pipeline, wherein the pipeline task includes at least one execution step, a target pipeline component associated with each execution step, and a target machine architecture corresponding to each target pipeline component;

[0035] An acquisition module, used to acquire a resource group corresponding to each execution step in the pipeline task, wherein the resource group includes at least one construction machine;

[0036] A resource allocation module is used to determine the target build machine corresponding to the pipeline task from each of the resource groups, and issue corresponding task instructions to each of the target build machines based on the execution steps corresponding to each of the resource groups. The target build machine is deployed with service software corresponding to the target pipeline component, and the service software corresponds to the target machine architecture.

[0037] In one embodiment, the resource allocation module is further configured to:

[0038] For any of the execution steps, determine at least one service software associated with the target pipeline component corresponding to the execution step, and use the build machine with the at least one service software deployed in the resource group corresponding to the execution step as a candidate build machine corresponding to the pipeline task;

[0039] A target build machine is determined from the candidate build machines based on the resource data of the candidate build machines.

[0040] In one embodiment, the resource allocation module is further configured to:

[0041] Determine a target candidate builder machine whose resource data satisfies an execution condition from among the candidate builder machines;

[0042] A target build machine having the lowest load among the target candidate build machines is determined from the target candidate build machines.

[0043] In one embodiment, the device further comprises:

[0044] A lock module, used to lock the target build machine;

[0045] The lock module is further used to release the lock of the target construction machine after receiving a successful response message for the task instruction.

[0046] In one embodiment, the device further comprises a maintenance module, which is used to:

[0047] In response to a grouping operation on the build machine, determining at least one resource group;

[0048] For any build machine in the resource group, after determining the service software corresponding to the build machine, issue an installation instruction to the build machine, wherein the installation instruction includes an installation package and an installation script of the service software, and the installation instruction is used to instruct the build machine to run the installation script to deploy the service software on the build machine based on the installation package;

[0049] In the maintenance data corresponding to the resource group, the correspondence between the construction machine and the deployed service software is recorded.

[0050] In one embodiment, in the component maintenance data, the correspondence between the component and the service software is recorded, and the resource allocation module is further used to:

[0051] At least one service software associated with the target pipeline component corresponding to the execution step is determined from the component maintenance data.

[0052] In one embodiment, the resource allocation module is further configured to:

[0053] If there is no target candidate build machine whose resource data satisfies the execution condition among the candidate build machines, displaying an execution condition adjustment page, wherein the execution condition is displayed on the execution condition adjustment page;

[0054] In response to the modification operation on the execution condition, the target candidate build machine is continuously determined from the candidate build machines based on the modified execution condition.

[0055] In one embodiment, the apparatus further comprises a scheduling module for executing:

[0056] If there is no target candidate builder whose resource data satisfies the execution condition among the candidate builder machines, then for any of the candidate builder machines, the data to be scheduled of the candidate builder machine is constructed based on the task data of each executed pipeline task in the candidate builder machine and the resource data of the candidate builder machine;

[0057] Acquire task data of the pipeline task, and input the task data of the pipeline task and the to-be-scheduled data of each candidate builder into a pre-trained scheduling model to obtain a scheduling plan;

[0058] The executed pipeline tasks in each of the candidate build machines are scheduled based on the scheduling scheme, and a target build machine is determined from the candidate build machines that have completed task scheduling.

[0059] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above resource allocation methods when executing the computer program.

[0060] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements any of the above resource allocation methods when executed by a processor.

[0061] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements any of the above resource allocation methods when executed by a processor.

[0062] The resource allocation method, apparatus, computer device, computer-readable storage medium and computer program product described above can construct a pipeline task in response to an orchestration operation for a pipeline, wherein the pipeline task includes at least one execution step, a target pipeline component associated with each execution step, and a target machine architecture corresponding to each target pipeline component. Obtain a resource group corresponding to each execution step in the pipeline task, wherein the resource group includes at least one builder. Determine the target builder corresponding to the pipeline task from each resource group, and issue corresponding task instructions to each target builder based on the execution steps corresponding to each resource group. The target builder is deployed with service software corresponding to the target pipeline component, and the service software corresponds to the target machine architecture. By using the resource allocation method, apparatus, computer equipment, computer-readable storage medium, and computer program product provided in the embodiments of the present application, when arranging pipeline tasks, the target pipeline components and target machine architecture corresponding to the pipeline tasks can be set. When performing resource allocation, the target build machine that meets the pipeline task execution environment can be automatically matched based on the service software associated with the target pipeline components and the target machine architecture, thereby ensuring that the pipeline tasks can be executed smoothly, improving the execution efficiency of the pipeline tasks, and improving the resource utilization of the build machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0064] Figure 1 An application environment diagram of a resource allocation method in an embodiment;

[0065] Figure 2 A schematic diagram of a flow chart of a resource allocation method in an embodiment;

[0066] Figure 3 A schematic diagram of a process for maintaining a builder in one embodiment;

[0067] Figure 4 A schematic diagram of a process for determining a target build machine in one embodiment;

[0068] Figure 5 A schematic diagram of a process of determining a target build machine from candidate build machines in one embodiment;

[0069] Figure 6 A schematic diagram of a process of determining a target candidate build machine from candidate build machines in one embodiment;

[0070] Figure 7 A schematic diagram of a process of determining a target build machine from candidate build machines in one embodiment;

[0071] Figure 8 is a schematic diagram of a resource allocation method in one embodiment;

[0072] Fig. 9 is a structural block diagram of a resource allocation device in an embodiment;

[0073] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0075] The resource allocation method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the builder 102 communicates with the resource allocation platform 104 via a network. The data storage system can store data that the resource allocation platform 104 needs to process. The data storage system can be integrated on the resource allocation platform 104, or placed on a cloud or other network server.

[0076] The resource allocation platform 104 can manage the service software required to execute the pipeline tasks in a unified manner, including software installation packages, installation scripts, environment variables, etc., and support service software of multiple machine architectures. The resource allocation platform 104 can also maintain pipeline component information and record information such as service software associated with pipeline components. At the same time, the resource allocation platform 104 can also maintain builder information, record information such as service software installed in the builder 102, monitor the resource load of the builder, and the amount of build tasks. After the pipeline task is orchestrated, the target builder that meets the pipeline execution environment can be automatically matched based on the pipeline components involved in the pipeline task and the corresponding machine architecture, based on the locally maintained builder information and pipeline component information, and the pipeline task can be issued to the target builder.

[0077] The resource allocation platform 104 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and servers. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, and projection devices. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0078] In an exemplary embodiment, Figure 2 As shown, a resource allocation method is provided, which is applied to Figure 1 The resource allocation platform 104 in FIG. 1 is taken as an example to illustrate, including the following steps 202 to 206. Among them:

[0079] Step 202, in response to the orchestration operation for the pipeline, construct a pipeline task, the pipeline task including at least one execution step, target pipeline components associated with each execution step, and a target machine architecture corresponding to each target pipeline component.

[0080] In the embodiments of the present application, a pipeline task generally refers to a series of continuous, automated work steps, which are executed in a predetermined order to complete a specific task or process a series of data. The user can manually arrange the pipeline task in the interface, or arrange the pipeline task by command line, or arrange the pipeline task by uploading a pipeline task document. The embodiments of the present application do not specifically limit the arrangement operation for the pipeline.

[0081] Exemplary, take the manual arrangement of pipeline tasks as an example. First, you can create a pipeline task, including creating a pipeline task name, description, generating an ID (Identification) corresponding to the pipeline task, etc. After creating the pipeline task, you can enter the content configuration interface of the pipeline task, in which you can sequentially configure the various execution steps included in the pipeline task and the dependencies between the various execution steps, including sequentially setting the detailed information of each execution step, such as the step name, command, input and output, etc., and setting the target pipeline component associated with each execution step and the corresponding target machine architecture. For example, you can determine which pipeline components are needed for each execution step, such as compilers, test tools, deployment tools, etc., and determine the corresponding target machine architecture, such as x86, ARM, cloud environment, etc., to ensure that the pipeline components are compatible with the architecture.

[0082] Exemplarily, the user can give priority to setting the target machine architecture corresponding to the execution step. When setting the target pipeline component corresponding to the execution step, the resource allocation platform can give priority to displaying the pipeline component corresponding to the target machine architecture to the user, and the user can select the target pipeline component corresponding to the execution step from the displayed pipeline components.

[0083] Alternatively, the user can search for the target pipeline component through the search box. When the target pipeline component does not exist in the pipeline components locally maintained by the resource allocation platform, the user can enter the component function description interface. The user can enter the component function description information in the component function description interface. The resource allocation platform can search for matching pipeline components from the locally maintained pipeline components based on the component function description information (the similarity between the function description information of the matching pipeline component and the component function description information is greater than or equal to the similarity threshold), and recommend the matching pipeline component to the user.

[0084] Alternatively, if the target pipeline component that matches the component function description information does not exist in the resource allocation platform, a corresponding prompt message can be generated and displayed. The prompt message may include a creation link. By triggering the creation link, the component creation page can be entered, and the user can create the target pipeline component on the component creation page.

[0085] After the pipeline task is created and the target pipeline components and the target machine architecture are associated for each execution step in the pipeline task, the pipeline task is constructed.

[0086] Step 204, obtaining a resource group corresponding to each execution step in the pipeline task, wherein the resource group includes at least one builder.

[0087] In the embodiment of the present application, the resource allocation platform locally maintains the builder information. The builder can be divided into multiple resource groups, each resource group includes at least one builder, and the resource allocation platform can maintain the builder information according to the granularity of the resource group. In an exemplary embodiment, refer to Figure 3 As shown, the above method may further include the following steps 302 to 306, wherein:

[0088] Step 302: In response to a grouping operation on a builder, determine at least one resource group.

[0089] In an embodiment of the present application, the build machines can be grouped according to dimensions such as architecture type, usage team, and resource amount. For example, the build machines can be divided into multiple resource groups corresponding to different architecture types, or the build machines can be divided into multiple resource groups corresponding to different usage teams, or the build machines can be divided into multiple resource groups corresponding to different resource amounts, etc. The embodiment of the present application does not make specific limitations on the grouping method of the build machines.

[0090] Step 304, for any build machine in the resource group, after determining the service software corresponding to the build machine, send an installation instruction to the build machine, the installation instruction includes the installation package and installation script of the service software, and the installation instruction is used to instruct the build machine to run the installation script to deploy the service software on the build machine based on the installation package.

[0091] In an embodiment of the present application, the resource allocation platform can locally perform unified management of the service software required to execute pipeline tasks, such as maven, node, python, jdk, kaniko, etc., including software installation packages, installation scripts, environment variables, etc., and can support multiple machine architectures, such as x86, aarch64, etc.

[0092] After the resource groups are divided, for any build machine in the resource group, the corresponding service software can be selected for the build machine, and the installation instruction can be sent to the build machine based on the installation package, installation script, and environment dependency information of the selected service software. After receiving the installation instruction, the build machine can parse the installation package, installation script, and environment dependency information of the service software from the installation instruction, and execute the installation script to install the service software.

[0093] Step 306: Record the correspondence between the builder and the deployed service software in the maintenance data corresponding to the resource group.

[0094] In an embodiment of the present application, after receiving information indicating that the build machine has completed the installation of the service software, the correspondence between the build machine and the deployed service software can be recorded in the maintenance data of the resources corresponding to the build machine, including information such as the ID of the build machine, the installed service software, and the corresponding machine architecture.

[0095] Compared with offline manual installation and maintenance of service software, the resource allocation method provided in the embodiment of the present application is a resource allocation platform that can uniformly manage service software without the need for manual initialization of the builder, thereby unifying standards, reducing the error rate of human operations, and improving work efficiency.

[0096] In the embodiment of the present application, after the pipeline task is constructed, a corresponding resource group can be selected for each step in the pipeline task. For example, the corresponding resource group can be selected based on the machine architecture corresponding to the execution step, or based on the team corresponding to the execution step.

[0097] Step 206, determine the target build machine corresponding to the pipeline task from each resource group, and issue corresponding task instructions to each target build machine based on the execution steps corresponding to each resource group. The target build machine is deployed with service software corresponding to the target pipeline component, and the service software corresponds to the target machine architecture.

[0098] In an embodiment of the present application, after selecting the resource group corresponding to each execution step, the target build machine that meets the environment required for the execution step can be determined from each resource group based on the target pipeline component and target machine architecture associated with the execution step. Subsequently, in the pipeline task, the task instruction corresponding to the execution step can be issued to the target build machine.

[0099] Exemplarily, a build machine that has the service software corresponding to the target pipeline component installed and supports the target machine architecture can be searched from the maintenance information corresponding to the resource group, and the found build machine can be used as the target build machine.

[0100] The above resource allocation method can be constructed in response to the orchestration operation for the pipeline, wherein the pipeline task includes at least one execution step, a target pipeline component associated with each execution step, and a target machine architecture corresponding to each target pipeline component. Obtain the resource group corresponding to each execution step in the pipeline task, and the resource group includes at least one construction machine. Determine the target construction machine corresponding to the pipeline task from each resource group, and issue corresponding task instructions to each target construction machine based on the execution steps corresponding to each resource group, and the target construction machine is deployed with service software corresponding to the target pipeline component, and the service software corresponds to the target machine architecture. Using the resource allocation method provided in the embodiment of the present application, when arranging the pipeline task, the target pipeline component and the target machine architecture corresponding to the pipeline task can be set, and when performing resource allocation, the target construction machine that meets the pipeline task execution environment can be automatically matched based on the service software and the target machine architecture associated with the target pipeline component, so as to ensure that the pipeline task can be executed smoothly, improve the execution efficiency of the pipeline task, and improve the resource utilization of the construction machine.

[0101] In an exemplary embodiment, the resource allocation platform may record the correspondence between components and service software in the component maintenance data, for example, recording the name and ID of the component, the name, ID, and supported machine architecture of each service software associated with the component.

[0102] In an exemplary embodiment, referring to Figure 4 As shown, determining the target build machine corresponding to the pipeline task from each resource group may include the following steps 402 to 404, wherein:

[0103] Step 402: for any execution step, determine at least one service software associated with the target pipeline component corresponding to the execution step, and use the build machine with at least one service software deployed in the resource group corresponding to the execution step as a candidate build machine corresponding to the pipeline task;

[0104] Step 404: determine a target build machine from the candidate build machines based on the resource data of the candidate build machines.

[0105] In an embodiment of the present application, taking an execution step as an example, determining at least one service software associated with a target pipeline component corresponding to the execution step may include: determining at least one service software associated with a target pipeline component corresponding to the execution step from component maintenance data.

[0106] That is, for the execution step, the target pipeline component and target machine architecture associated with the execution step can be determined, and the service software (hereinafter referred to as the target service software for clarity of description) associated with the target pipeline component and supporting the target machine architecture can be further searched from the component maintenance data.

[0107] The build machine with all target service software installed is determined from the resource group as a candidate build machine. The resource allocation platform detects the resource data of each build machine in real time, and the resource data may include but is not limited to load data, build task volume data, etc. The resource data of each candidate build machine can be obtained, and a build machine with low load and / or low task volume can be selected from the candidate build machines as the target build machine.

[0108] In an exemplary embodiment, referring to Figure 5 As shown, in step 404, determining the target build machine from the candidate build machines based on the resource data of the candidate build machines may include the following steps 502 to 504, wherein:

[0109] Step 502, determining a target candidate builder whose resource data meets the execution condition from the candidate builder;

[0110] Step 504 , determining a target build machine from the target candidate build machines, where the target build machine has the lowest load among the target candidate build machines.

[0111] In the embodiment of the present application, due to the diversity of pipeline tasks, the focus of resources will be biased. For example, if the pipeline task has a large amount of data calculation, it will pay more attention to CPU resources, or if the pipeline task involves a lot of data reading and writing, it will pay more attention to memory resources. Therefore, when constructing a pipeline task, the execution conditions corresponding to the pipeline task can be pre-set based on its inclination towards resource data, or the execution conditions corresponding to each execution step can be set.

[0112] Exemplarily, the execution condition may include conditions such as CPU and memory usage being lower than a threshold. That is, after determining the candidate build machines corresponding to the pipeline task, first select a target candidate build machine that meets the execution condition from the candidate build machines, and further select a build machine with the lowest comprehensive load from the target candidate build machines as the target build machine, wherein the comprehensive load is calculated by combining the load conditions of various resources of the build machine, for example, by weighted summing up the loads of various resources.

[0113] In an exemplary embodiment, referring to Figure 6 As shown, in step 502, a target candidate builder whose resource data meets the execution condition is determined from the candidate builder, specifically including the following steps 602 to 604, wherein:

[0114] Step 602: If there is no target candidate builder whose resource data satisfies the execution condition among the candidate builder machines, an execution condition adjustment page is displayed, and the execution condition is displayed in the execution condition adjustment page;

[0115] Step 604 : In response to the modification operation on the execution condition, continue to determine a target candidate builder from the candidate builder based on the modified execution condition.

[0116] In the embodiment of the present disclosure, if there is no target candidate builder that meets the execution condition among the candidate builder machines, a prompt message may be fed back to the user, prompting the user that there is currently no builder that meets the execution condition, and asking the user whether to adjust the execution condition. For example, the prompt message may include an adjustment control, and the user may trigger the adjustment control to jump to the execution condition adjustment page, which may include the current execution condition. In response to the user's modification of the current execution condition, the modified execution condition may be obtained, and the target candidate builder may continue to be matched from the candidate builder machines based on the modified execution condition.

[0117] In an exemplary embodiment, when there is no target candidate builder machine that meets the execution condition among the candidate builder machines, the execution condition may not be adjusted. Figure 7 As shown, the above method further includes the following steps 702 to 706, wherein:

[0118] Step 702: If there is no target candidate builder whose resource data satisfies the execution condition among the candidate builder machines, then for any candidate builder machine, the data to be scheduled of the candidate builder machine is constructed based on the task data of each executed pipeline task in the candidate builder machine and the resource data of the candidate builder machine;

[0119] Step 704, obtaining task data of the pipeline task, and inputting the task data of the pipeline task and the data to be scheduled of each candidate builder into the pre-trained scheduling model to obtain a scheduling plan;

[0120] Step 706 , schedule the executed pipeline tasks in each candidate builder based on the scheduling scheme, and determine the target builder from the candidate builder that has completed the task scheduling.

[0121] In an embodiment of the present disclosure, when there is no target candidate builder machine that meets the execution conditions among the candidate builder machines, the priority of the pipeline task can be obtained. The priority can be set when building the pipeline task, or it can be determined based on the execution conditions of the pipeline task or the identity of the user who created the pipeline task. This is not specifically limited in the embodiments of the present application.

[0122] For each candidate build machine, the data to be scheduled for each candidate build machine can be constructed based on the task data of each executed pipeline task in each candidate build machine and the resource data of the candidate build machine. In the data to be scheduled for any candidate build machine, the task data may include the task description data corresponding to each executed pipeline task (the task currently executed in the candidate build machine), the task description data may include the service software information, priority, execution time, execution conditions, resource usage, etc. associated with the associated pipeline component, and the resource data may include the various resource data of the candidate build machine and the deployed service software information, etc.

[0123] The task data (including priority and execution conditions) of the current pipeline task and the data to be scheduled of each candidate builder can be input into a pre-trained scheduling model. The scheduling model can output a corresponding scheduling plan. The scheduling plan may include the executed pipeline task to be scheduled (hereinafter referred to as the task to be scheduled), the candidate builder information that executes the executed pipeline task to be scheduled (hereinafter referred to as the first candidate builder), and the target candidate builder information to be scheduled (hereinafter referred to as the second candidate builder), where the priority of the task to be scheduled is lower than the priority of the current pipeline task.

[0124] The resource allocation platform can schedule the tasks to be scheduled from the first candidate builder to the second candidate builder for execution. After the scheduling is completed, the first candidate builder will meet the execution conditions of the current pipeline task, and the second candidate builder can meet the execution conditions of each executed pipeline task. The first candidate builder can be directly used as the target builder for the current pipeline task.

[0125] Among them, the training process of the scheduling model may include: building a resource group, collecting a large amount of historical task data and resource data of candidate build machines for the resource group, including: task description data (including service software information associated with the task, priority, execution time, execution conditions, occupied resources, etc.) and resource data (including various resource data of the candidate build machine (such as CPU, memory, storage, etc.) and deployed service software information), building a sample pipeline task, and obtaining sample data based on the sample pipeline task and the collected data, inputting the sample data into the initial scheduling model, outputting the corresponding scheduling plan (sample task to be scheduled, first sample build machine, second sample build machine), after executing the scheduling plan, determining whether the first sample build machine meets the execution conditions of the sample pipeline task, whether the second sample build machine meets the execution conditions of the executed pipeline task in the second sample build machine, and whether the priority of the sample task to be scheduled is lower than the priority of the sample pipeline task, adjusting the initial scheduling model based on the determination result, and continuing to iteratively train the initial scheduling model until the training is completed to obtain the trained scheduling model.

[0126] The initial scheduling model can adopt models such as decision trees, random forests, neural networks, and reinforcement learning. After selecting the model, it is necessary to design the structure and parameters of the model to ensure that the model can effectively learn the laws of task scheduling. During the training process, the initial scheduling model will learn how to optimally schedule tasks and allocate resources based on the input task data and resource data. When adjusting the network parameters of the initial scheduling model, the loss function, optimization algorithm selection, hyperparameter tuning, etc. can be used for adjustment, which is not specifically limited in the embodiments of the present application. After the training is completed, the performance of the scheduling model can be evaluated. Evaluation indicators may include but are not limited to accuracy, resource utilization, task execution time, etc.

[0127] In an exemplary embodiment, before issuing corresponding task instructions to each target builder based on the execution steps corresponding to each resource group, the embodiment of the present application may further include the following steps:

[0128] Lock the target build machine;

[0129] After issuing corresponding task instructions to each target builder based on the execution steps corresponding to each resource group, the following steps may also be included:

[0130] After receiving a successful response message for the task instruction, the lock of the target builder is released.

[0131] In an embodiment of the present application, after determining the target builder corresponding to the pipeline task, the target builder can be locked. After the target builder is locked, the resource allocation platform will no longer issue other pipeline tasks to the target builder until the task instruction corresponding to the current pipeline task is issued to the target builder. After receiving the successful response message of the task instruction issuance, the lock of the target builder will be released, so that the target builder can ensure that it can meet the operation requirements of the pipeline task before the current pipeline task is issued, and at the same time, after executing the pipeline task, it can make full use of the remaining resources.

[0132] In order to enable those skilled in the art to better understand the embodiments of the present application, the embodiments of the present application are described below through specific examples.

[0133] Reference Figure 8 As shown, the embodiment of the present application may include the following steps 1 to 7, wherein:

[0134] Step 1: The resource allocation platform can centrally manage the service software required to perform tasks, such as Maven, Node, Python, JDK, Kaniko and other service software. It can maintain the installation packages, installation scripts, environment variables, etc. of the service software, and support software of multiple machine architectures, such as x86, aarch64, etc., for easy use.

[0135] Step 2: The resource allocation platform can maintain pipeline components and establish association information between each pipeline component and the corresponding service software, such as maven, node, django and other associated service software information.

[0136] Step 3: The resource allocation platform can maintain the build machines, manage the build machines in groups, select the service software for each build machine, and automatically install the various service software required by the build machine. The installation process of the service software can include: first sending the software package and installation script package to the build machine, then decompressing the installation script package, and finally executing the installation script to install the software on the execution machine, and the installation result is sent to the front end through the MQ message.

[0137] Step 4: Orchestrate the pipeline, select the corresponding components and execution environment groups, and associate the pipeline with the components and build environment.

[0138] Step 5: Execute the pipeline task and select the resource group. The system will count the required service software based on the components associated with the pipeline, and compare it with the build machines in the selected resource group. It will mainly compare the service software and select the qualified machines: match the build machine corresponding to the pipeline task based on the service software associated with the component and the software installed on the build machine.

[0139] Step 6: Select a suitable build machine based on the filtered resource load and the number of build tasks to lock it to prevent multiple concurrent build machines from occupying it at the same time, and then send task instructions to the build machine.

[0140] Step 7: Release the lock after receiving the successful message of task instruction issuance.

[0141] The resource allocation method provided in the embodiment of the present application can manage the service software required for executing the task in a unified manner. The operator can directly use the system to complete the initialization of the builder, unify the standards, reduce the error rate of human operation, and improve work efficiency. And the resource group is selected for executing the pipeline task. The system counts the required service software according to the components associated with the pipeline task, and compares it with the machine situation in the selected execution group. The service software situation is mainly compared to screen out the qualified machine. The traditional selection of the builder is generally a direct manual method. In this way, the operator is not clear about the actual use of the current builder, which may cause the task to be unable to execute, the machine load is too high, and other problems, affecting production efficiency. In the embodiment of the present application, the user can select a builder group, and the system automatically assigns the task to a specific machine. There are many screening principles, including the machine software installation situation, load situation, and task construction quantity situation. The system selects an optimal machine to execute the task, ensuring that it can still run smoothly and stably under the situation of multi-concurrency of tasks, which can improve the construction efficiency and task execution efficiency, and ensure the robustness of the high-concurrency multi-task system.

[0142] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0143] Based on the same inventive concept, the embodiment of the present application also provides a resource allocation device for implementing the resource allocation method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more resource allocation device embodiments provided below can refer to the limitations on the resource allocation method above, and will not be repeated here.

[0144] In an exemplary embodiment, Fig. 9 As shown, a resource allocation device is provided, including: an arrangement module 902, an acquisition module 904 and a resource allocation module 906, wherein:

[0145] The orchestration module 902 is used to construct a pipeline task in response to the orchestration operation for the pipeline, wherein the pipeline task includes at least one execution step, a target pipeline component associated with each execution step, and a target machine architecture corresponding to each target pipeline component;

[0146] An acquisition module 904 is used to acquire a resource group corresponding to each execution step in the pipeline task, wherein the resource group includes at least one construction machine;

[0147] The resource allocation module 906 is used to determine the target build machine corresponding to the pipeline task from each of the resource groups, and issue corresponding task instructions to each of the target build machines based on the execution steps corresponding to each of the resource groups. The target build machine is deployed with service software corresponding to the target pipeline component, and the service software corresponds to the target machine architecture.

[0148] By using the resource allocation device provided in the embodiment of the present application, when arranging pipeline tasks, the target pipeline components and target machine architecture corresponding to the pipeline tasks can be set. When performing resource allocation, the target build machine that meets the pipeline task execution environment can be automatically matched based on the service software associated with the target pipeline components and the target machine architecture, thereby ensuring that the pipeline tasks can be executed smoothly, improving the execution efficiency of the pipeline tasks, and improving the resource utilization of the build machine.

[0149] In one embodiment, the resource allocation module 906 is further configured to:

[0150] For any of the execution steps, determine at least one service software associated with the target pipeline component corresponding to the execution step, and use the build machine with the at least one service software deployed in the resource group corresponding to the execution step as a candidate build machine corresponding to the pipeline task;

[0151] A target build machine is determined from the candidate build machines based on the resource data of the candidate build machines.

[0152] In one embodiment, the resource allocation module 906 is further configured to:

[0153] Determine a target candidate builder machine whose resource data satisfies an execution condition from among the candidate builder machines;

[0154] A target build machine having the lowest load among the target candidate build machines is determined from the target candidate build machines.

[0155] In one embodiment, the device further comprises:

[0156] A lock module, used to lock the target build machine;

[0157] The lock module is further used to release the lock of the target construction machine after receiving a successful response message for the task instruction.

[0158] In one embodiment, the device further comprises a maintenance module, which is used to:

[0159] In response to a grouping operation on the build machine, determining at least one resource group;

[0160] For any build machine in the resource group, after determining the service software corresponding to the build machine, issue an installation instruction to the build machine, wherein the installation instruction includes an installation package and an installation script of the service software, and the installation instruction is used to instruct the build machine to run the installation script to deploy the service software on the build machine based on the installation package;

[0161] In the maintenance data corresponding to the resource group, the correspondence between the construction machine and the deployed service software is recorded.

[0162] In one embodiment, in the component maintenance data, the correspondence between the component and the service software is recorded, and the resource allocation module 906 is further used to:

[0163] At least one service software associated with the target pipeline component corresponding to the execution step is determined from the component maintenance data.

[0164] In one embodiment, the resource allocation module is further configured to:

[0165] If there is no target candidate build machine whose resource data satisfies the execution condition among the candidate build machines, displaying an execution condition adjustment page, wherein the execution condition is displayed on the execution condition adjustment page;

[0166] In response to the modification operation on the execution condition, the target candidate build machine is continuously determined from the candidate build machines based on the modified execution condition.

[0167] In one embodiment, the apparatus further comprises a scheduling module for executing:

[0168] If there is no target candidate builder whose resource data satisfies the execution condition among the candidate builder machines, then for any of the candidate builder machines, the data to be scheduled of the candidate builder machine is constructed based on the task data of each executed pipeline task in the candidate builder machine and the resource data of the candidate builder machine;

[0169] Acquire task data of the pipeline task, and input the task data of the pipeline task and the to-be-scheduled data of each candidate builder into a pre-trained scheduling model to obtain a scheduling plan;

[0170] The executed pipeline tasks in each of the candidate build machines are scheduled based on the scheduling scheme, and a target build machine is determined from the candidate build machines that have completed task scheduling.

[0171] Each module in the resource allocation device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0172] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.10As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a resource allocation method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0173] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0174] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0175] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0176] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0177] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0178] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0179] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0180] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A resource allocation method, characterized in that: The method comprises: In response to the orchestration operation for the pipeline, a pipeline task is constructed, wherein the pipeline task includes at least one execution step, a target pipeline component associated with each execution step, and a target machine architecture corresponding to each target pipeline component; Obtain a resource group corresponding to each execution step in the pipeline task, wherein the resource group includes at least one builder; Determine the target build machine corresponding to the pipeline task from each of the resource groups, and issue corresponding task instructions to each of the target build machines based on the execution steps corresponding to each of the resource groups. The target build machine is deployed with service software corresponding to the target pipeline component, and the service software corresponds to the target machine architecture.

2. The method according to claim 1, characterized in that Determining a target build machine corresponding to the pipeline task from each of the resource groups includes: For any of the execution steps, determine at least one service software associated with the target pipeline component corresponding to the execution step, and use the build machine with the at least one service software deployed in the resource group corresponding to the execution step as a candidate build machine corresponding to the pipeline task; A target build machine is determined from the candidate build machines based on the resource data of the candidate build machines.

3. The method according to claim 2, characterized in that The step of determining a target build machine from the candidate build machines based on the resource data of the candidate build machines includes: Determine a target candidate builder machine whose resource data satisfies an execution condition from among the candidate builder machines; A target build machine having the lowest load among the target candidate build machines is determined from the target candidate build machines.

4. The method according to any one of claims 1 to 3, characterized in that Before issuing corresponding task instructions to each target builder based on the execution steps corresponding to each resource group, the method further includes: Locking the target build machine; After issuing corresponding task instructions to each target builder based on the execution steps corresponding to each resource group, the method further includes: After receiving a successful response message for the task instruction, the lock of the target builder is released.

5. The method according to any one of claims 1 to 3, characterized in that The method further comprises: In response to a grouping operation on the build machine, determining at least one resource group; For any build machine in the resource group, after determining the service software corresponding to the build machine, issue an installation instruction to the build machine, wherein the installation instruction includes an installation package and an installation script of the service software, and the installation instruction is used to instruct the build machine to run the installation script to deploy the service software on the build machine based on the installation package; In the maintenance data corresponding to the resource group, the correspondence between the construction machine and the deployed service software is recorded.

6. The method according to claim 2, characterized in that In the component maintenance data, the correspondence between the component and the service software is recorded, and the determining of at least one service software associated with the target pipeline component corresponding to the execution step includes: At least one service software associated with the target pipeline component corresponding to the execution step is determined from the component maintenance data.

7. The method according to claim 3, characterized in that The step of determining a target candidate builder whose resource data satisfies the execution condition from among the candidate builder machines comprises: If there is no target candidate build machine whose resource data satisfies the execution condition among the candidate build machines, displaying an execution condition adjustment page, wherein the execution condition is displayed on the execution condition adjustment page; In response to the modification operation on the execution condition, the target candidate build machine is continuously determined from the candidate build machines based on the modified execution condition.

8. The method according to claim 3, characterized in that The method further comprises: If there is no target candidate builder whose resource data satisfies the execution condition among the candidate builder machines, then for any of the candidate builder machines, the data to be scheduled of the candidate builder machine is constructed based on the task data of each executed pipeline task in the candidate builder machine and the resource data of the candidate builder machine; Acquire task data of the pipeline task, and input the task data of the pipeline task and the to-be-scheduled data of each candidate builder into a pre-trained scheduling model to obtain a scheduling plan; The executed pipeline tasks in each of the candidate build machines are scheduled based on the scheduling scheme, and a target build machine is determined from the candidate build machines that have completed task scheduling.

9. A resource allocation device, characterized in that: The device comprises: An orchestration module, configured to construct a pipeline task in response to an orchestration operation on the pipeline, wherein the pipeline task includes at least one execution step, a target pipeline component associated with each execution step, and a target machine architecture corresponding to each target pipeline component; An acquisition module, used to acquire a resource group corresponding to each execution step in the pipeline task, wherein the resource group includes at least one construction machine; A resource allocation module is used to determine the target build machine corresponding to the pipeline task from each of the resource groups, and issue corresponding task instructions to each of the target build machines based on the execution steps corresponding to each of the resource groups. The target build machine is deployed with service software corresponding to the target pipeline component, and the service software corresponds to the target machine architecture.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images