Resource matching method, device and system based on cloud computing

Abstract

The present invention discloses a resource matching method, device and system based on cloud computing, which relates to the field of computer technology. The main purpose of the present invention is to determine the resources required for a job by standardizing the format of collecting job information, and then match the computing resources of the cluster as a whole. The main technical scheme of the present invention is: obtaining job description information for a job from a target cluster; extracting resource configuration information in the job description information using a preset job model, wherein the resource configuration information at least includes the processing resource requirement information and processing process association information of the job; counting the computing resources required for the job in the target cluster according to the resource configuration information; and determining whether to adjust the computing resources currently configured in the target cluster according to the preset strategy and the computing resources required for the job.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a resource matching method, device and system based on cloud computing. Background Art

[0002] In recent years, with the rapid development of cloud computing, the demand for high-performance computing in the cloud has increased, and the amount of computing tasks has increased. Automatic scaling solutions that use elastic computing resources on the cloud have emerged. The automatic scaling solution for elastic computing resources means that users can scale system resources based on CPU utilization, memory usage, etc., and efficiently utilize the processing power of the cluster.

[0003] However, for HPC (High Performance Computing) users who use schedulers to manage jobs, the same cluster often serves multiple HPC users. These HPC users may not use the same scheduler based on their own needs and usage habits. The resource usage statistics of jobs in the cluster, such as CPU and memory usage, collected by these different types of schedulers often cannot accurately reflect the actual resource requirements of the jobs. For example, for jobs that occupy exclusive nodes, it will lead to insufficient resource expansion, while for jobs with dependencies, it will lead to excessive resource expansion. It can be seen that the existing automatic scaling solutions for elastic computing resources cannot effectively match the computing resources required for jobs in the scenarios where multiple schedulers are used for HPC users, which leads to the problem of low efficiency in cluster job processing. Summary of the invention

[0004] In view of the above problems, the present invention proposes a resource matching method, device and system based on cloud computing, the main purpose of which is to determine the resources required for the job by standardizing the format of collecting job information, and then overall match the computing resources of the cluster.

[0005] In order to achieve the above object, the present invention mainly provides the following technical solutions:

[0006] In one aspect, the present invention provides a resource matching method based on cloud computing, which specifically includes:

[0007] Get the job description information for the job from the target cluster;

[0008] Extracting resource configuration information from the job description information using a preset job model, wherein the resource configuration information at least includes processing resource requirement information and processing process association information of the job;

[0009] Counting the computing resources required for the jobs in the target cluster according to the resource configuration information;

[0010] Determine whether to adjust the computing resources currently configured in the target cluster according to a preset strategy and the computing resources required by the job.

[0011] On the other hand, the present invention provides a resource matching device based on cloud computing, specifically comprising:

[0012] An acquisition unit, used to acquire job description information for a job from a target cluster;

[0013] An extraction unit, configured to extract resource configuration information from the job description information acquired by the acquisition unit by using a preset job model, wherein the resource configuration information at least includes processing resource requirement information and processing process association information of the job;

[0014] A statistics unit, configured to count the computing resources required for the jobs in the target cluster according to the resource configuration information obtained by the extraction unit;

[0015] The matching unit is used to determine whether to adjust the computing resources currently configured in the target cluster according to a preset strategy and the computing resources required for the job obtained by the statistical unit.

[0016] On the other hand, the present invention provides a resource matching system based on cloud computing, the system comprising a computing node, a management node;

[0017] The computing node is used to process the job in the target cluster and send the job description information of the job to the management node;

[0018] The management node is used to adjust the computing resources currently configured in the target cluster according to the industry description information and the preset strategy, and execute the above-mentioned resource matching method based on cloud computing.

[0019] On the other hand, the present invention provides a processor, which is used to run a program, and the program executes the above-mentioned cloud computing-based resource matching method when running.

[0020] On the other hand, the present invention provides an electronic device, comprising a processor and a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by the processor, the above-mentioned cloud computing-based resource matching method is implemented.

[0021] Through the above technical scheme, the present invention provides a resource matching method, device and system based on cloud computing, which mainly obtains corresponding job description information from the target cluster in units of jobs when automatically scaling the computing resources of the cluster, and extracts resource configuration information with a unified format through a preset job model. The resource configuration information contains processing resource demand information and processing process association information of the job, wherein the processing resource demand information is used to represent the job's own demand for computing resources, and the processing process association information is used to represent the demand for the degree of computing resource sharing between jobs. Based on these resource configuration information in a unified format, the computing resources required for the target cluster to process the job can be calculated more quickly and accurately from multiple dimensions, and then the computing resource expansion or reduction operation to be performed on the target cluster is determined based on the current computing resources of the target cluster and the preset strategy configured by the user, thereby improving the rationality of scaling the computing resources of the target cluster, thereby improving the processing efficiency of the target cluster for jobs.

[0022] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0024] Figure 1 A flow chart of a resource matching method based on cloud computing proposed in an embodiment of the present invention is shown;

[0025] Figure 2 A flowchart of another resource matching method based on cloud computing proposed in an embodiment of the present invention is shown;

[0026] Figure 3 A block diagram showing a resource matching device based on cloud computing proposed in an embodiment of the present invention is shown;

[0027] Figure 4 A block diagram showing another resource matching device based on cloud computing proposed in an embodiment of the present invention is shown;

[0028] Figure 5 A flow chart showing the execution principle of the cloud computing-based resource matching method proposed in an embodiment of the present invention;

[0029] Figure 6 A block diagram showing the steps of executing the resource matching method based on cloud computing proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0031] The embodiment of the present invention provides a resource matching method based on cloud computing. The method optimizes and improves the scaling solution of cloud computing resources for computing clusters, and improves the effectiveness and rationality of resource matching by accurately representing the computing resources required for jobs in the target cluster. The specific steps of this method are as follows: Figure 1 As shown, the method includes:

[0032] Step 101: Obtain job description information for the job from the target cluster.

[0033] The job description information obtained in this step refers to all the information submitted by the user for the job when submitting the job, including the demand for the job processing resources and the associated information of the processing. Since different users use different types of schedulers in the embodiment of the present invention, the description methods of the jobs are also different. This step is to directly obtain all the relevant information of the job, so as to extract the information required in the embodiment of the present invention, and avoid the omission of information acquisition due to format problems.

[0034] It should be noted that, in actual application, the embodiments of the present invention may also restrict the description information of the job submitted by the user, requiring the user to explain the value of the specified information parameter of the job, so as to ensure the comprehensiveness of the obtained job description information. The specified information parameter is customized according to the specific application scenario, and this embodiment does not make specific limitations on this.

[0035] Step 102: extract resource configuration information from the job description information using a preset job model.

[0036] In this step, the preset job model is pre-set based on the specific scenario applied by the target cluster, and is used to organize the job description information corresponding to the jobs in different formats submitted by different users in the cluster, and unify the representation of the job's demand for computing resources. Its input is the job description information obtained in the previous step, and the output is the resource configuration information in a unified format. The resource configuration information at least includes the processing resource demand information of the job and the processing process association information, wherein the processing resource demand information refers to the computing resources required for the job itself when processing, such as the memory required for processing the job, the number of CPU cores, etc.; the processing process association information refers to whether the job is associated with other jobs when processing the job, such as the dependency relationship of the job, that is, the processing of job A needs to be performed according to the calculation result of job B, so that job A and job B do not need to allocate computing resources to them at the same time. In other words, when describing the demand for computing resources of the job in this step, the resource configuration information is expressed at least from different dimensions such as the association relationship between the job itself and the job, that is, the demand for computing resources of the job is expressed from multiple dimensions, and in addition, it can also include the time dimension, such as the waiting time of the job. By representing jobs in multiple dimensions, we can more accurately describe the job's demand for computing resources, thereby more accurately counting the target cluster's demand for computing resources.

[0037] The preset job model in this step can be understood as a formatted template for resource configuration information. Different target clusters can set one or more templates according to their own needs to analyze jobs submitted by different users and extract information therein to generate resource configuration information.

[0038] Step 103: Count the computing resources required for the jobs in the target cluster according to the resource configuration information.

[0039] Since each job in the target cluster is represented in the form of resource configuration information, and the resource requirements of the job itself and the relationship between jobs are represented from multiple dimensions, this step can accurately count the demand for computing resources based on the resource configuration information of the jobs in the target cluster.

[0040] The statistics of jobs in the target cluster in this step can refer to all jobs in the target cluster, or jobs that have not been processed and are in a queued state. The purpose of counting all jobs is to determine the overall computing resources required by the target cluster to process these jobs. Whether to perform scaling operations needs to be determined by comparing the currently configured computing resources. The purpose of counting jobs in a queued state is to calculate the incremental demand for computing resources of the target cluster. This method can more quickly determine whether the target cluster needs to perform a capacity expansion operation. However, in the absence of queued jobs, this method is also difficult to determine whether the target cluster needs to perform a capacity reduction operation.

[0041] Step 104: Determine whether to adjust the computing resources currently configured in the target cluster according to the preset strategy and the computing resources required by the job.

[0042] In this step, the specific demand for computing resources of the target cluster can be determined by comparing the computing resources required by the job with the computing resources currently configured in the target cluster, and the addition of preset policies is used to control the degree of adjustment of computing resources. That is, through the control of preset policies, the total amount of shrinking and expanding the target cluster can be limited to avoid the occurrence of unlimited shrinking and expanding. Among them, the computing resources currently configured in the target cluster include the computing resources in use and the unused computing resources in the target cluster. When the computing resources required by the job are the computing resources required by all jobs, the currently configured computing resources are the sum of the used and unused computing resources. When the computing resources required by the job are the computing resources required by the unprocessed jobs, the currently configured computing resources can refer to the unused computing resources.

[0043] In addition, the preset strategy is also used to further verify the computing resources required for the job, and the preset strategy is used to check whether the computing resources required for the job counted in step 103 are reasonable. If not, the computing resources required for the job can be corrected according to the preset strategy. In this way, the system administrator can continuously update the specific method of computing resource statistics through the preset strategy.

[0044] From the description of the above embodiments, it can be seen that a resource matching method based on cloud computing provided by an embodiment of the present invention is to process the jobs in the target cluster into resource configuration information represented in multiple dimensions in the same way, so as to count the computing resources required by the target cluster, and then perform controllable scaling adjustment on the required computing resources according to the preset strategy combined with the computing resources currently configured by the target cluster, so as to improve the efficiency of the target cluster in processing jobs.

[0045] In practical applications, the resource matching method described in the embodiment of the present invention can be deployed separately as an independent service. It can be deployed in the cloud or in the target cluster. By starting the service, the demand for computing resources of the target cluster can be effectively counted, and combined with its own existing computing resources, it can be determined through artificial preset strategies whether to perform corresponding scaling operations on the target cluster, thereby realizing dynamic management of the computing resources of the target cluster.

[0046] Corresponding to the above Figure 1 The following description of the embodiment shown will be combined with Figure 5 The flowchart shown in the figure illustrates a resource matching method based on cloud computing proposed by the present invention. Specifically, Figure 5The computing nodes 1-3 and the management node constitute the target cluster in this embodiment, wherein the computing nodes are used to process process jobs submitted by users, such as computing jobs proposed by scientific computing software such as meteorology, mechanics, and molecular dynamics; the management node is used to obtain the job description information corresponding to the jobs to be processed by each computing node in the target cluster, which can be an independent node in the target cluster, or can be set on a device where a computing node is located. A preset job model is set in the management node, and the preset job model extracts the resource configuration information in each job description information, and reports the resource configuration information to the automatic scaling service in the cloud. The automatic scaling service is used to statistically process the computing resources required for the jobs in the target cluster, and determine whether to adjust the computing resources currently configured in the target cluster according to a preset strategy, wherein, Figure 5 The computing instances in the table represent the computing resources that the cloud can provide. When the automatic scaling service determines that the target cluster needs to be expanded, the management node is used to notify the computing instances provided for the target cluster so that the target cluster can quickly process the queued jobs. When the automatic scaling service determines that the target cluster needs to be reduced, the management node is also used to obtain the right to use some computing resources in the target cluster. Figure 5 In the figure, the auto-scaling service and computing instance enclosed by the dotted box are cloud resources, especially the computing instance, which can be flexibly applied for and released. The auto-scaling service provides dynamic allocation of computing resources for the cluster it serves, thereby optimizing the use of computing resources.

[0047] Furthermore, for the above Figure 1 The resource matching method based on cloud computing, the embodiment of the present invention will explain in detail the determination of the target cluster computing resource requirements and how to determine the specific way to scale the target cluster computing resources. The specific steps are as follows Figure 2 As shown, including:

[0048] Step 201: Obtain job description information for the job from the target cluster.

[0049] In this example, the jobs in the target cluster are submitted by users using different schedulers, and the jobs submitted by different users will be added to the corresponding schedulers to determine the priority of job processing according to their own needs. Therefore, to obtain the jobs in the target cluster, you can read them from the schedulers of different users. To do this, you need to first determine the schedulers corresponding to the users in the target cluster, and then obtain the jobs that need to be processed in the target cluster from these schedulers, and further obtain the corresponding job description information based on the determined jobs.

[0050] Step 202: Obtain load information of the target cluster.

[0051] This step can be performed synchronously with step 201 regularly, and the load information of the target cluster is obtained while obtaining the job description information. The load information refers to the computing resources currently configured by the target cluster that have been used to process the job.

[0052] Since the target cluster is generally composed of multiple nodes, different nodes have different types according to their functions, and the corresponding load information representations will also be different. Therefore, the embodiment of the present invention also normalizes the load information through a preset load model to obtain the resource occupancy information corresponding to each node. The load model is similar to the above-mentioned preset job model. Its function is to uniformly represent the load conditions of different nodes in the target cluster. Its input is the load information reported by each node, and its output is the resource occupancy information in a unified format. The main contents of the resource occupancy information include: node identification, node type, node CPU and memory parameters and their respective usage, etc., the job identification of the node, etc.

[0053] The purpose of obtaining resource occupancy information is to determine the computing resources currently configured in the target cluster. At the same time, it is also possible to determine whether the target cluster has releasable computing resources based on the target cluster's demand for computing resources based on the resource configuration information, that is, to count the computing resources in the target cluster that are temporarily not needed to execute job processing. If there are releasable computing resources, a request to release computing resources can be sent to the cloud, and the cloud will allocate new jobs to these computing resources for processing, thereby fully utilizing the computing resources.

[0054] Step 203: extract resource configuration information from the job description information using a preset job model.

[0055] The specific implementation process of this step is: first determine the preset operation model according to the identifier of the target cluster, that is, first determine the preset operation model, wherein the identifier of the target cluster is determined based on the functions of the target cluster, such as the applications installed in the target cluster, the types of operations to be processed, etc. Then, use the preset operation model to extract the corresponding information value from the operation description information according to the preset formatting information, and generate resource configuration information with a unified format according to the obtained information value.

[0056] The resource configuration information mainly includes: job identification information, processing resource requirement information and processing process related information. Specifically, the processing resource requirement information may include whether the job occupies the node processing exclusively, the number of nodes requested by the job, the number of cores requested by the job on each node, the amount of memory requested by the job, etc., while the processing process related information may include the dependent job identification information, the dependent job processing status, etc.

[0057] Step 204: Count the computing resources required for the jobs in the target cluster according to the resource configuration information.

[0058] Specifically, when counting the computing resources required for a job, you can use the preset strategy to accurately count the computing resources required for the target cluster. For example, by identifying whether a job is exclusively processing a node, you can effectively prevent the remaining computing resources of the node that will be exclusively occupied by the job from being counted. For example, by identifying whether a job has dependent jobs and the processing status of the dependent jobs, you can determine whether it is necessary to expand computing resources for the job, thereby avoiding the waste of resources caused by invalid expansion of computing resources. It can be seen that the preset strategy can accurately determine whether each job needs to expand computing resources and how much computing resources need to be expanded.

[0059] Since the preset strategy can be continuously updated, and the resource configuration information corresponding to the jobs in the target cluster has a unified format, the demand for computing resources of the target cluster can be further quickly and accurately counted.

[0060] Step 205: Determine whether to adjust the computing resources currently configured in the target cluster according to the preset strategy and the computing resources required by the job.

[0061] First, the computing resources required by the target cluster are counted based on the computing resources required by the job, that is, the computing resource requirements of the target cluster are counted by all the jobs in the target cluster. At the same time, the computing resources currently configured in the target cluster are counted based on the load information obtained.

[0062] Afterwards, determine whether the computing resources required by the target cluster are greater than the currently configured computing resources. The purpose of this judgment is to determine whether the computing required by the currently unprocessed jobs in the target cluster can be satisfied by the currently idle computing resources of the target cluster. If the idle computing resources are insufficient, that is, greater than the currently configured computing resources, it is necessary to determine the capacity expansion request for the target cluster according to the preset strategy. Conversely, when there are more idle computing resources, it is possible to determine the capacity reduction request for the target cluster according to the preset strategy.

[0063] When determining the scaling request for the target cluster according to the preset strategy, it is necessary to obtain the preset strategy for the target cluster, that is, different target clusters can set corresponding strategies, which include the maximum scaling value of the target cluster. By judging whether the difference between the computing resources required by the target cluster and the currently configured computing resources is greater than the maximum scaling value, if so, the scaling value requested can be the maximum scaling value, otherwise, the scaling value requested is the difference. In other words, the preset strategy can be used to control the degree of scaling of the computing resources of the target cluster, thereby improving the rationality of computing resource matching.

[0064] Through the steps of the above-mentioned embodiments and the corresponding diagrammatic illustrations, it can be seen that the cloud computing-based resource matching method proposed in the embodiment of the present invention realizes the accurate determination of the computing resources required for the entire target cluster through the unified and standardized representation of the job requirements in the target cluster. Compared with the method of requesting computing resource expansion for queued jobs in the prior art, the embodiment of the present invention can more effectively integrate the computing resources configured by the target cluster, determine the degree of computing resource demand of different jobs, such as exclusive nodes, dependent jobs, etc., to determine whether to make a request for scaling in or out, and the specific values ​​of scaling in or out, so as to realize efficient utilization of cloud computing resources.

[0065] According to the above Figure 2 The embodiments, in combination with Figure 5 The flowchart shown in FIG. 1 specifically illustrates the specific steps of the resource matching method based on cloud computing in the target cluster processing operation process. Figure 6 As shown, including:

[0066] First, the user submits a job to the target cluster. Assuming that all computing nodes in the target cluster are in working state, the job is in a queued state.

[0067] Second, the auto-scaling service will periodically send query requests for the workload and cluster computing resource usage status to the management node of the target cluster.

[0068] Third, the management node obtains the job description information and load information of each computing node in the target cluster according to the query request, that is, steps 201 and 202. The load information is the usage status of the computing resources currently configured in the cluster.

[0069] Fourth, the preset job model in the management node extracts the resource configuration information in the job description information, and reports the resource configuration information and load information to the automatic scaling service. The automatic scaling service counts the computing resources required for the jobs in the target cluster, and determines whether computing resources need to be adjusted according to the preset strategy, that is, steps 203-205, and generates corresponding scaling instructions.

[0070] Fifth, the automatic scaling service sends the scaling command feedback to the management node.

[0071] Sixth, the management node executes the scaling command. When executing the scaling command, it determines the computing resources provided by the cloud, that is, the computing instances, and schedules the jobs queued in the target cluster to be processed by the computing instances. When executing the scaling command, it obtains the free computing resources or computing instances in the target cluster and releases them to the jobs to be processed in other clusters.

[0072] Seventh, the target cluster distributes the queued jobs to the corresponding computing instances for processing and obtains the computing results fed back by the computing instances.

[0073] Finally, the target cluster feeds back the calculation results to the user.

[0074] Furthermore, as a response to the above Figure 1 , 2 In order to realize the method shown in the figure, the embodiment of the present invention provides a resource matching device based on cloud computing. The main purpose of the device is to determine the resources required for the job by standardizing the format of collecting job information, and then match the computing resources of the cluster as a whole. For the sake of ease of reading, this device embodiment will no longer repeat the details of the above method embodiments one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the above method embodiments. The device is as follows Figure 3 As shown, specifically including:

[0075] An acquisition unit 31, configured to acquire job description information for a job from a target cluster;

[0076] An extraction unit 32 is used to extract resource configuration information from the job description information obtained by the acquisition unit 31 using a preset job model, wherein the resource configuration information at least includes processing resource requirement information and processing process association information of the job;

[0077] A statistics unit 33, configured to count the computing resources required for the jobs in the target cluster according to the resource configuration information obtained by the extraction unit 32;

[0078] The matching unit 34 is used to determine whether to adjust the computing resources currently configured in the target cluster according to a preset strategy and the computing resources required for the job obtained by the statistical unit 33 .

[0079] Further, such as Figure 4 As shown, the extraction unit 32 includes:

[0080] A determination module 321, configured to determine the preset operation model according to the identifier of the target cluster;

[0081] An extraction module 322, configured to extract corresponding information values ​​from the job description information according to preset formatting information using the preset job model determined by the determination module 321;

[0082] The generating module 323 is used to generate resource configuration information having a unified format according to the information value obtained by the extracting module 322 .

[0083] Furthermore, the resource configuration information specifically includes: job identification information, processing resource requirement information and processing process related information; wherein, the processing resource requirement information includes whether the node is exclusively processed, the number of requested nodes, the number of requested cores for each node, and the amount of memory requested; the processing process related information includes dependent job identification information.

[0084] Further, such as Figure 4 As shown, the matching unit 34 includes:

[0085] A statistics module 341 is used to count the computing resources required by the target cluster according to the computing resources required by the job;

[0086] A judgment module 342 is used to judge whether the computing resources required by the target cluster obtained by the statistical module 341 are greater than the currently configured computing resources;

[0087] The matching module 343 is used to determine the expansion request for the target cluster according to the preset strategy when the judgment module 342 determines that the computing resources are greater than the currently configured ones; otherwise, determine the reduction request for the target cluster according to the preset strategy.

[0088] Furthermore, the matching module 343 is specifically used to obtain a preset strategy for the target cluster, wherein the preset strategy includes a maximum expansion value for the target cluster; determine whether the difference between the computing resources required by the target cluster and the currently configured computing resources is greater than the maximum expansion value; if so, the requested expansion value is the maximum expansion value, otherwise, the requested expansion value is the difference.

[0089] Further, such as Figure 4 As shown, the device also includes:

[0090] The acquisition unit 31 is further used to acquire load information of the target cluster, where the load information includes load information of each node;

[0091] A processing unit 35 is used to process the load information obtained by the acquisition unit 31 using a preset load model to obtain resource occupancy information corresponding to each node;

[0092] The statistical unit 33 is further used to count the computing resources that can be released by the target cluster according to the resource occupancy information obtained by the processing unit 35 and the resource configuration information obtained by the extraction unit 32 .

[0093] Further, such as Figure 4 As shown, the device also includes:

[0094] The sending unit 36 ​​is configured to send a request to release computing resources to the cloud when the statistical unit 33 determines that there are releasable computing resources in the target cluster.

[0095] Furthermore, the acquisition unit 31 is further configured to acquire job description information of jobs in an unprocessed state.

[0096] Further, such as Figure 4 As shown, the acquisition unit 31 also includes:

[0097] A determination module 311, used to determine a scheduler corresponding to a user in the target cluster;

[0098] The acquisition module 312 is used to acquire the job description information corresponding to the job that needs to be processed in the target cluster from the scheduler determined by the determination module 311.

[0099] Furthermore, an embodiment of the present invention also provides a resource matching system based on cloud computing, which includes a computing node and a management node, wherein the computing node is used to process jobs in a target cluster and send the job description information of the job to the management node, and the management node is used to adjust the computing resources currently configured in the target cluster according to the job description information and preset strategies.

[0100] Specifically, Figure 5 , 6 As shown, the management node is used to implement statistics and adjustment of the computing resources of the target cluster by calling the automatic scaling service in the cloud. The automatic scaling service uses preset decisions to analyze the job description information and load information of the target cluster to determine whether the computing resources in the target cluster have reached the optimal usage state, and generates corresponding scaling instructions based on the analysis results to adjust the computing resources of the target cluster.

[0101] In addition, an embodiment of the present invention further provides a processor, the processor is used to run a program, wherein the program executes the above Figure 1 or Figure 2 The embodiment shown in the embodiment provides a resource matching method based on cloud computing.

[0102] In addition, an embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by the processor, the above Figure 1 or Figure 2 The embodiment shown in the embodiment provides a resource matching method based on cloud computing.

[0103] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0104] It is understandable that the related features in the above methods and devices can be referenced to each other. In addition, the "first", "second" and the like in the above embodiments are used to distinguish the embodiments, but do not represent the advantages and disadvantages of the embodiments.

[0105] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0106] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the preferred embodiment of the present invention.

[0107] In addition, the memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory.

[0108] (flash RAM), the memory includes at least one memory chip.

[0109] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0110] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0111] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0113] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0114] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0115] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0116] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0117] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0118] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A resource matching method based on cloud computing, applied to the management node of the target cluster, the method include: According to the query request sent by the cloud, obtain the job description information for the job from the target cluster; Determine a preset job model according to the identifier of the target cluster, and extract corresponding information values ​​from the job description information using the preset job model according to preset formatting information; Generate resource configuration information in a unified format according to the obtained information value, wherein the resource configuration information at least includes processing resource requirement information of the job and processing process association information, wherein the processing process association information refers to whether the job is associated with other jobs when processing the job; Sending the resource configuration information to the cloud, so that the cloud counts the computing resources required for the job in the target cluster according to the resource configuration information; and counts the computing resources required for the target cluster according to the computing resources required for the job; Determining whether to adjust the computing resources currently configured for the target cluster according to a preset strategy based on a size relationship between the computing resources required by the target cluster and the computing resources currently configured for the target cluster; Execute the scaling instructions sent by the cloud.

2. The method according to claim 1, It is characterized in that The resource configuration information includes: job identification information, processing resource requirement information and processing process related information; wherein, The processing resource requirement information includes whether to exclusively occupy the node processing, the number of nodes requested, the number of cores requested for each node, and the amount of memory requested; The processing process associated information includes dependent job identification information.

3. The method according to claim 1, It is characterized in that According to the size relationship between the computing resources required by the target cluster and the computing resources currently configured for the target cluster, determining whether to adjust the computing resources currently configured for the target cluster according to a preset strategy includes: Determine whether the computing resources required by the target cluster are greater than the currently configured computing resources; If it is greater than, determining a capacity expansion request for the target cluster according to a preset strategy; If it is less than, a scaling-down request for the target cluster is determined according to a preset strategy.

4. The method according to claim 3, It is characterized in that Determining a capacity expansion request for the target cluster according to a preset strategy includes: Acquire a preset strategy for the target cluster, wherein the preset strategy includes a maximum expansion value of the target cluster; Determine whether the difference between the computing resources required by the target cluster and the currently configured computing resources is greater than the maximum expansion value; If so, the value requested for expansion is the maximum expansion value; otherwise, the value requested for expansion is the difference.

5. The method according to claim 1, It is characterized in that When obtaining job description information for the job from the target cluster, the method further includes: Obtaining load information of the target cluster, wherein the load information includes load information of each node; Processing the load information using a preset load model to obtain resource occupancy information corresponding to each node; The computing resources that can be released by the target cluster are counted according to the resource occupancy information and the resource configuration information.

6. The method according to claim 5, It is characterized in that The method further comprises: When there are releasable computing resources in the target cluster, a request to release the computing resources is sent to the cloud.

7. The method according to claim 1, It is characterized in that Get the job description information for the job from the target cluster, including: Get the job description information of the job in the unprocessed state.

8. The method according to claim 1, It is characterized in that Get the job description information for the job from the target cluster, including: Determine a scheduler corresponding to a user in the target cluster; Obtain job description information corresponding to the job that needs to be processed in the target cluster from the scheduler.

9. A resource matching device based on cloud computing, applied to the management node of a target cluster, the device include: An acquisition unit, used for acquiring job description information for a job from a target cluster according to a query request sent by the cloud; An extraction unit, configured to determine a preset job model according to the identifier of the target cluster, and extract corresponding information values ​​from the job description information using the preset job model according to preset formatting information; Generate resource configuration information in a unified format according to the obtained information value, the resource configuration information at least including the processing resource demand information of the job and the processing process association information, the processing process association information refers to whether the job is associated with other jobs when processing the job; send the resource configuration information to the cloud, and execute the scaling instruction sent by the cloud, the resource configuration information is used by the cloud to count the computing resources required by the jobs in the target cluster; count the computing resources required by the target cluster according to the computing resources required by the jobs; According to the size relationship between the computing resources required by the target cluster and the computing resources currently configured for the target cluster, it is determined according to a preset policy whether to adjust the computing resources currently configured for the target cluster.

10. A resource matching system based on cloud computing, the system comprising computing nodes, management nodes and a cloud; The computing node is used to process the job in the target cluster and send the job description information of the job to the management node; The management node is used to obtain job description information for the job from the target cluster according to the query request sent by the cloud; Determine a preset job model according to the identifier of the target cluster, and extract corresponding information values ​​from the job description information using the preset job model according to preset formatting information; Generate resource configuration information in a unified format according to the obtained information value, wherein the resource configuration information at least includes processing resource requirement information of the job and processing process association information, wherein the processing process association information refers to whether the job is associated with other jobs when processing the job; execute the scaling instruction sent by the cloud; The cloud is used to send the query request to the management node, and count the computing resources required for the job in the target cluster according to the resource configuration information sent by the management node; count the computing resources required for the target cluster according to the computing resources required for the job; According to the size relationship between the computing resources required by the target cluster and the computing resources currently configured for the target cluster, determine whether to adjust the computing resources currently configured for the target cluster according to a preset strategy, and send a scaling instruction to the management node.

11. A processor, It is characterized in that The processor is used to run a program, and when the program is run, the cloud computing-based resource matching method according to any one of claims 1 to 8 is executed.

12. An electronic device, It is characterized in that The electronic device includes a processor and a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by the processor, the cloud computing-based resource matching method as described in any one of claims 1 to 8 is implemented.

Images