Resource management method, device and equipment

Abstract

The present invention discloses a resource management method, device and equipment, the method comprising: receiving a resource application request from a resource user, matching each first tag information in a first tag group carried by the resource application request with tag information associated with each preset resource, and obtaining a first matching resource that successfully matches the first tag group; detecting whether the available resource quantity of each first matching resource is not less than the requested resource quantity in the resource application request; if so, determining to pass the resource application of the resource user, and locking the available resource quantity of each first matching resource according to the requested resource quantity. The present invention identifies and divides resources through tag information, thereby improving the flexibility of resource allocation.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a resource management method, device and equipment. Background Art

[0002] Yarn (YetAnotherResourceNegotiator) is a resource management system that uses a resource manager to manage and allocate all resources in the cluster. Specifically, the resources in the cluster are divided according to the queue dimension, and each node is specified in advance to which queue it belongs. When applying for resources, resource users (users) can specify which queue's resources to use. If the number of resources currently applied for by the user does not exceed the limit, the application can be successful. This resource management method can only fix the granularity of control in advance, such as queues, users and other dimensions, and cannot support dynamic adjustment and insertion of other granularities according to actual needs. As a result, after the resource allocation strategy is formulated, it is impossible to divide and summarize resources at any time, and the resource management method is not flexible enough. Summary of the invention

[0003] The main purpose of the present invention is to provide a resource management method, device and equipment, aiming to solve the technical problems that the existing resource management system cannot divide and aggregate resources at any time and the resource management method is not flexible enough.

[0004] To achieve the above object, the present invention provides a resource management method, which comprises the following steps:

[0005] Receive a resource application request from a resource user, match each piece of first tag information in a first tag group carried in the resource application request with each piece of preset tag information, and obtain a target tag number associated with the preset tag information that successfully matches the first tag information;

[0006] The resources associated with the target tag number in each preset resource are taken as candidate resources, and it is detected whether each matchable preset tag information associated with the candidate resource has tag information that is successfully matched in the first tag group;

[0007] If all of them exist, the candidate resource is used as the first matching resource that successfully matches the first tag group, and it is detected whether the available resource quantity of each of the first matching resources is not less than the requested resource quantity in the resource application request;

[0008] If so, it is determined that the resource application of the resource user is approved, and the available resource quantity of each of the first matching resources is locked according to the applied resource quantity.

[0009] Optionally, the step of matching each piece of first tag information in the first tag group carried by the resource application request with each piece of preset tag information to obtain a target tag number associated with the preset tag information successfully matched with each piece of the first tag information includes:

[0010] For each piece of first label information in the first label group carried in the resource application request, extract each label value in each piece of the first label information;

[0011] Searching for each of the tag values ​​in a preset quick query table, and obtaining a tag number associated with the found tag value in the preset quick query table;

[0012] If the tag number is found in the preset tag table, the tag number is used as the target tag number, wherein a row of records in the preset tag table includes a preset tag information and a tag number associated with the preset tag information, and a row of records in the preset quick query table includes a tag value in a preset tag information and a tag number associated with the preset tag information.

[0013] Optionally, the first tag group includes a user tag information for uniquely identifying the resource user, and after the step of locking the available resource quantity of each first matching resource according to the applied resource quantity, the method further includes:

[0014] Add a new resource and use the requested resource quantity as the locked resource quantity of the new resource;

[0015] The newly added resource is associated with each of the first tag information in the first tag group.

[0016] Optionally, after the step of locking the available resource quantity of each first matching resource according to the applied resource quantity, the method further includes:

[0017] receiving resource usage report information from the resource user, matching each piece of second tag information in a second tag group carried in the resource usage report information with tag information associated with each preset resource, and obtaining a second matching resource that successfully matches the second tag group, wherein the second tag group is the same as the first tag group;

[0018] Detecting whether the locked resource quantity of each of the second matching resources is not less than the actual used resource quantity in the resource usage reporting information;

[0019] If so, convert the locked resource quantity in each of the second matching resources into the used resource quantity according to the actually used resource quantity.

[0020] Optionally, after the step of converting the locked resource quantity in each second matching resource into the used resource quantity according to the actually used resource quantity, the method further includes:

[0021] Receive a resource recycling request from the resource user, match each piece of third tag information in a third tag group carried by the resource recycling request with tag information associated with each preset resource, and obtain a third matching resource that successfully matches the third tag group, wherein the third tag group is the same as the first tag information group;

[0022] Detecting whether the used resource quantity of each of the third matching resources is not less than the recycled resource quantity in the resource recycling request;

[0023] If so, the used resource quantity of each of the third matching resources is recycled according to the recycled resource quantity.

[0024] Optionally, after the step of recycling the used resource quantity of each of the third matching resources according to the recycled resource quantity, the method further includes:

[0025] Detecting whether the used resource quantity of the target resource associated with the user tag information in the third matching resource is zero;

[0026] If so, the target resource is deleted.

[0027] Optionally, the method further comprises:

[0028] A new resource is added according to the resource quantity information carried in the resource registration request, and each fourth tag information carried in the resource registration request is respectively associated with the newly added resource, wherein the resource registration request is initiated by a resource provider or generated according to resource usage restriction information set for a resource user, and when generated according to the resource usage restriction information, each fourth tag information is generated according to the restriction conditions in the resource usage restriction information and the resource quantity information is generated according to the restricted resource quantity in the resource usage restriction information.

[0029] Optionally, the method further comprises:

[0030] Receive a resource offline request, and match each fifth tag information carried by the resource offline request with the tag information associated with each preset resource, so as to search for a resource to be offline in each preset resource, wherein the resource to be offline is a resource associated with the same tag information as that corresponding to each fifth tag information;

[0031] If the resource to be taken offline is found, the resource quantity information of the resource to be taken offline and the association information between the resource to be taken offline and the tag information are deleted to take the resource offline.

[0032] To achieve the above object, the present invention further provides a resource management device, the device comprising:

[0033] A receiving module, used to receive a resource application request from a resource user, and match each piece of first tag information in a first tag group carried in the resource application request with each piece of preset tag information, to obtain a target tag number associated with the preset tag information that successfully matches the first tag information;

[0034] A first detection module is used to take resources associated with the target tag number in various preset resources as candidate resources, and detect whether all matching preset tag information associated with the candidate resources have tag information that is successfully matched in the first tag group;

[0035] A second detection module is used to, if all exist, use the candidate resource as a first matching resource that successfully matches the first tag group, and detect whether the available resource quantity of each of the first matching resources is not less than the requested resource quantity in the resource application request;

[0036] The determination module is used to, if yes, determine the resource application of the resource user and lock the available resource quantity of each of the first matching resources according to the applied resource quantity.

[0037] To achieve the above objectives, the present invention also provides a resource management device, which includes: a memory, a processor, and a resource management program stored in the memory and executable on the processor, and the resource management program implements the steps of the resource management method described above when executed by the processor.

[0038] In addition, to achieve the above-mentioned purpose, the present invention further proposes a computer-readable storage medium, on which a resource management program is stored, and when the resource management program is executed by a processor, the steps of the resource management method described above are implemented.

[0039] Compared with the existing resource management methods that can only fix the granularity of management and control in advance and cannot support dynamic adjustment and insertion of other granularities according to actual needs, in the present invention, by associating various resources with tag information, and having the resource user carry a tag group including one or more tag information in the resource application request when applying for resources, the resource manager matches the tag information in the tag group with the tag information associated with each resource to obtain matching resources that match the tag group, and when the number of available resources of the matching resources is not less than the number of applied resources in the resource application request, the resource user's resource application is passed, and the available resources of the matching resources are locked according to the number of applied resources, thereby realizing the identification and division of resources through tag information, and the flexible setting of tag information can divide resources from different granularities, and resources are allocated through tag matching in the resource application process, so that the resource manager can re-divide or summarize resources at any time by changing the tag information associated with the resources, thereby improving the flexibility of resource allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention;

[0041] Figure 2 It is a flowchart of the first embodiment of the resource management method of the present invention;

[0042] Figure 3 A schematic diagram of a table of resources and labels involved in an embodiment of the present invention;

[0043] Figure 4 Schematic diagram of functional modules of a preferred embodiment of the resource management device of the present invention.

[0044] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0046] like Figure 1 As shown, Figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0047] It should be noted that the resource management device in the embodiment of the present invention may be a smart phone, a personal computer, a server or other device, and no specific limitation is made herein.

[0048] like Figure 1As shown, the resource management device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0049] Those skilled in the art will understand that Figure 1 The device structure shown in the figure does not constitute a limitation on the resource management device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0050] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a resource management program. The operating system is a program that manages and controls the hardware and software resources of the device and supports the operation of the resource management program and other software or programs. Figure 1 In the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for establishing a communication connection with the server; and the processor 1001 can be used to call the resource management program stored in the memory 1005 and perform the following operations:

[0051] Receive a resource application request from a resource user, match each piece of first tag information in a first tag group carried in the resource application request with each piece of preset tag information, and obtain a target tag number associated with the preset tag information that successfully matches the first tag information;

[0052] The resources associated with the target tag number in each preset resource are taken as candidate resources, and it is detected whether each matchable preset tag information associated with the candidate resource has tag information that is successfully matched in the first tag group;

[0053] If all of them exist, the candidate resource is used as the first matching resource that successfully matches the first tag group, and it is detected whether the available resource quantity of each of the first matching resources is not less than the requested resource quantity in the resource application request;

[0054] If so, it is determined that the resource application of the resource user is approved, and the available resource quantity of each of the first matching resources is locked according to the applied resource quantity.

[0055] Furthermore, the step of matching each piece of first tag information in the first tag group carried by the resource application request with each piece of preset tag information to obtain a target tag number associated with the preset tag information successfully matched with each piece of the first tag information includes:

[0056] For each piece of first label information in the first label group carried in the resource application request, extract each label value in each piece of the first label information;

[0057] Searching for each of the tag values ​​in a preset quick query table, and obtaining a tag number associated with the found tag value in the preset quick query table;

[0058] If the tag number is found in the preset tag table, the tag number is used as the target tag number, wherein a row of records in the preset tag table includes a preset tag information and a tag number associated with the preset tag information, and a row of records in the preset quick query table includes a tag value in a preset tag information and a tag number associated with the preset tag information.

[0059] Further, the first tag group includes a user tag information for uniquely identifying the resource user. After the step of locking the available resource quantity of each first matching resource according to the applied resource quantity, the processor 1001 can also be used to call the resource management program stored in the memory 1005 to perform the following operations:

[0060] Add a new resource and use the requested resource quantity as the locked resource quantity of the new resource;

[0061] The newly added resource is associated with each of the first tag information in the first tag group.

[0062] Further, after the step of locking the available resource quantity of each of the first matching resources according to the applied resource quantity, the processor 1001 may also be used to call the resource management program stored in the memory 1005 to perform the following operations:

[0063] receiving resource usage report information from the resource user, matching each piece of second tag information in a second tag group carried in the resource usage report information with tag information associated with each preset resource, and obtaining a second matching resource that successfully matches the second tag group, wherein the second tag group is the same as the first tag group;

[0064] Detecting whether the locked resource quantity of each of the second matching resources is not less than the actual used resource quantity in the resource usage reporting information;

[0065] If so, convert the locked resource quantity in each of the second matching resources into the used resource quantity according to the actually used resource quantity.

[0066] Further, after the step of converting the locked resource quantity in each of the second matching resources into the used resource quantity according to the actually used resource quantity, the processor 1001 may also be used to call the resource management program stored in the memory 1005 to perform the following operations:

[0067] Receive a resource recycling request from the resource user, match each piece of third tag information in a third tag group carried by the resource recycling request with tag information associated with each preset resource, and obtain a third matching resource that successfully matches the third tag group, wherein the third tag group is the same as the first tag information group;

[0068] Detecting whether the used resource quantity of each of the third matching resources is not less than the recycled resource quantity in the resource recycling request;

[0069] If so, the used resource quantity of each of the third matching resources is recycled according to the recycled resource quantity.

[0070] Further, if so, after the step of recycling the used resource quantity in each of the third matching resources according to the recycled resource quantity, the processor 1001 may also be used to call the resource management program stored in the memory 1005 to perform the following operations:

[0071] Detecting whether the used resource quantity of the target resource associated with the user tag information in the third matching resource is zero;

[0072] If so, the target resource is deleted.

[0073] Furthermore, the processor 1001 may also be used to call the resource management program stored in the memory 1005 to perform the following operations:

[0074] A new resource is added according to the resource quantity information carried in the resource registration request, and each fourth tag information carried in the resource registration request is respectively associated with the newly added resource, wherein the resource registration request is initiated by a resource provider or generated according to resource usage restriction information set for a resource user, and when generated according to the resource usage restriction information, each fourth tag information is generated according to the restriction conditions in the resource usage restriction information and the resource quantity information is generated according to the restricted resource quantity in the resource usage restriction information.

[0075] Further, the processor 1001 may also be used to call the resource management program stored in the memory 1005 to perform the following operations:

[0076] Receive a resource offline request, and match each fifth tag information carried by the resource offline request with the tag information associated with each preset resource, so as to search for a resource to be offline in each preset resource, wherein the resource to be offline is a resource associated with the same tag information as that corresponding to each fifth tag information;

[0077] If the resource to be taken offline is found, the resource quantity information of the resource to be taken offline and the association information between the resource to be taken offline and the tag information are deleted to take the resource offline.

[0078] Based on the above structure, various embodiments of the resource management method are proposed.

[0079] Reference Figure 2 , Figure 2 It is a flowchart of the first embodiment of the resource management method of the present invention.

[0080] The embodiments of the present invention provide embodiments of the resource management method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than here. The execution subjects of each embodiment of the resource management method of the present invention may be devices such as smart phones, personal computers, and servers. For ease of description, the following embodiments are described with the resource manager (or simply referred to as the manager) as the execution subject. In this embodiment, the resource management method includes:

[0081] Step S10, receiving a resource application request from a resource user, matching each piece of first tag information in a first tag group carried in the resource application request with each piece of preset tag information, and obtaining a target tag number associated with the preset tag information successfully matched with the first tag information;

[0082] In this embodiment, resources refer to CPU resources, memory resources, hard disk resources, network resources, etc. required when a computer program is running. Multiple resources, i.e., preset resources, can be recorded in the resource manager in advance, and the size of the resource can be indicated by a resource quantity information. For example, the resource quantity information can include resource type, maximum resource quantity, minimum resource quantity, used resource quantity, remaining resource quantity (or usable resource quantity) and locked resource quantity. Among them, resource type can include resources such as CPU, memory, hard disk and network; maximum resource quantity is the total resource quantity of the resource; minimum resource quantity is the critical value of a resource that can be applied. When the remaining resource quantity is less than the minimum resource quantity, the resource can no longer be allocated to the resource user; used resource quantity is the resource quantity that has been allocated to the resource user for use; remaining resource quantity is the resource quantity that has not been allocated and locked; locked resource quantity is the resource quantity that has been allocated to the resource user but has not been actually used. The manager can distinguish various resources by resource number. One or more tags can be added to each resource in the resource manager in advance, that is, each resource can be associated with one or more tag information to identify the resource. In other implementations, the resource provider may also report its own resources to the resource manager and carry resource quantity information and tag information in the reported information. The resource manager adds a new resource based on the resource quantity information in the reported information and associates the resource with the tag information carried in the reported information.

[0083] Among them, a tag information may include a tag name and at least one tag value. The tag name refers to some descriptive words (such as color, type, height, etc.) that can be used to identify the characteristics of a certain transaction after the attributes existing on multiple things are abstracted. The tag value refers to the manifestation of a certain attribute actually possessed by a certain transaction (such as red, blue, large, small, high, medium, low, etc.). In this way, by specifying a series of tags, a smaller range can be matched from a group of things. The tag name involved in this embodiment can be any descriptive word that can be abstracted from the attributes existing in various resource providers or various resource users, for example, engine type (engineType), service alias (serverAlias), cluster name (cluster), user (user), etc. That is, the resource can be identified by any attribute of the provider or user of the resource, wherein the resource can be identified by the attribute of the resource provider, which can be used to limit who provides the resource, and the resource can be identified by the attribute of the resource user, which can be used to limit who can use the resource.

[0084] When a resource user needs to use a resource, he or she may send a resource application request to the resource manager, wherein the resource user is an object that needs to use a certain type of resource, such as a process. In this embodiment, the type of resource user is not limited. The resource user's resource application request may carry at least one tag information. Since multiple tag information may be carried, the combination of each tag information is called a tag group. To illustrate the difference, the tag group and tag information carried by the resource application request are referred to as the first tag group and the first tag information, respectively, in some descriptions. The tag information that the resource user needs to carry when generating a resource application request can be set in advance, or the management personnel can configure the tag information that each resource user needs to carry when generating a resource application request at any time as needed. Similarly, the tag information carried in the resource application request can be used to identify the resource user himself or herself, and can also be used to identify the resource provider.

[0085] The manager receives a resource application request from a resource user, extracts the first tag group in the resource application request, matches each piece of first tag information in the first tag group with the tag information associated with each preset resource, and finds a resource that successfully matches the first tag group. The resource that successfully matches the first tag group is referred to as the first matching resource below. Depending on the specific application scenario, the judgment rules for whether the tag group and the resource are successfully matched may be different; for example, each piece of tag information associated with a resource may correspond to each piece of tag information in the first tag group, and it can be determined that the resource is successfully matched with the first tag group; or, each piece of tag information associated with a resource can find the same tag information in the first tag group, and it can be determined that the resource is successfully matched with the first tag group, that is, the tag information associated with the resource is included in the first tag group, but the tag information in the first tag group may not necessarily be the tag information associated with the resource. It can be understood that in this embodiment, the judgment rules for whether the tag group and the resource are successfully matched are not limited.

[0086] In one embodiment, a specific rule is proposed for determining whether a tag group matches a resource successfully. That is, when all tag information associated with a resource can find the same tag information in the tag group, it can be determined that the resource matches the tag group successfully.

[0087] According to the judgment rule, the resource manager can first extract the first tag group carried in the resource application request, and match each first tag information in the first tag group with each preset tag information. Specifically, in one embodiment, the resource manager can compare each first tag information with each preset tag information one by one to determine whether there is the same preset tag information as the first tag information. The two tag information are the same, that is, the tag name and the tag value are the same; if there is the same preset tag information as the first tag information, it can be determined that the preset tag information matches the first tag information successfully, and the tag number (or tag ID) associated with the preset tag information is used as the target tag number. In other embodiments, the resource manager can also compare each tag value in each first tag information with the tag value of each preset tag information one by one to determine whether it is the same tag value as the tag value in the first tag information. If so, the tag number of the preset tag information corresponding to the same tag value is used as the target tag number; in this case, when a first tag information includes multiple tag values, matching with the first tag information may match multiple target tag numbers.

[0088] The resource manager uses each piece of first tag information to perform a match respectively, and obtains a target tag number associated with the preset tag information that successfully matches some of the first tag information. If multiple pieces of first tag information are successfully matched, multiple target tag numbers are finally obtained. In one embodiment, if each piece of first tag information is not successfully matched, the resource manager can determine that the resource application is not passed, that is, no resource matching the tag in the resource application request is found.

[0089] Among them, the preset tag information is the tag information associated with each resource recorded in advance in the resource manager. Each piece of preset tag information is distinguished by a tag number, that is, each piece of preset tag information is associated with a unique tag number, and the tag information is associated with the resource by associating the tag number with the resource number. Specifically, in one embodiment, a tag table can be used to store each piece of preset tag information, and a row of records in the tag table includes a preset tag information and a tag number associated with the preset tag information; a resource table is used to store each resource, and a row of records in the resource table includes a resource number of a resource and the resource quantity information of the resource; and a resource tag association table is used to store the association relationship between resources and tags, and a row of records in the resource tag association table includes a tag number and a resource number. It can be understood that the same tag information can be associated with multiple resources, and a resource can also be associated with multiple different tag information.

[0090] Furthermore, in one embodiment, tag information that does not participate in matching can be set in advance, or the resource user marks the first tag information that does not participate in matching in the resource application request, and the resource manager can not match the first tag information that does not participate in matching during matching. For example, when applying for resources, resource users generally carry an engine instance tag information, and the engine instance tag information is unique to each resource user and is generally not associated with resources, so the engine instance tag information can be set to not participate in matching to reduce unnecessary calculations.

[0091] Step S20, taking resources associated with the target tag number in various preset resources as candidate resources, and detecting whether all matching preset tag information associated with the candidate resources have tag information that is successfully matched in the first tag group;

[0092] After matching the target tag number, the resource manager can use the resource associated with the target tag number as a candidate resource. Specifically, in one embodiment, the resource tag association table can be searched to find the record where the target tag number is located, and the resource number stored in the record associated with the target tag number is extracted, and the resource number is the resource number of the candidate resource.

[0093] After obtaining the candidate resources, the resource manager can first find the preset tag information associated with the candidate resources that can be matched. Among the multiple preset tag information associated with a resource, tag information that does not participate in the matching can be preset. For example, the engine manager tag information mostly indicates the machine and port where the resource provider is located, and the resource user does not need to specify the range of machines and ports every time he applies, so the tag information can be set not to participate in the matching. The resource manager detects whether each of the preset tag information that can be matched associated with the candidate resource has a successfully matched tag information in the first tag group. Among them, a preset tag information has a successfully matched tag information in the first tag group, that is, there is a tag information in the first tag group that is the same as the preset tag information; a preset tag information does not have a successfully matched tag information in the first tag group, that is, each tag information in the first tag group is different from the preset tag information.

[0094] In one embodiment, the resource manager may first search the resource tag association table, find the record where the resource number of the candidate resource is located, and extract the tag number stored in association with the resource number in the record. It is understandable that when a resource is associated with multiple tag information, multiple tag numbers can be found based on the resource number of the candidate resource. According to each tag number found, the record where the tag number is located is found in the tag table, and the tag information stored in association with the tag number in the record is extracted, that is, each preset tag information associated with the candidate resource is extracted. For each preset tag information associated with the candidate resource that can be involved in the matching, the preset tag information is compared with each first tag information in the first tag group to determine whether there is tag information in the first tag group that is identical to the preset tag information.

[0095] Step S30: If all of them exist, the candidate resource is used as the first matching resource that successfully matches the first tag group, and it is detected whether the available resource quantity of each of the first matching resources is not less than the requested resource quantity in the resource application request;

[0096] If all the matching preset tag information associated with the candidate resource has successfully matched tag information in the first tag group, the resource manager can determine that the candidate resource successfully matches the first tag group, that is, the candidate resource is used as the first matching resource. It should be noted that when there are multiple candidate resources, each candidate resource is judged according to the above steps to determine whether it successfully matches the first tag group.

[0097] There may be one or more first matching resources that successfully match the first tag group. After determining the first matching resource, the resource manager can detect whether the available resource quantity of each first matching resource is not less than the requested resource quantity in the resource application request. Among them, the available resource quantity of each resource is recorded in the resource manager. When it is not applied for, the available resource quantity is the maximum resource quantity of the resource. When a part of the resource is applied for, the available resource quantity is correspondingly reduced, and the used resource quantity is correspondingly increased. The resource user's resource application request can carry the requested resource quantity, that is, the number of resources that need to be applied for, for example, 10 CPUs and 100G memory are required.

[0098] Step S40: If yes, determine to approve the resource application of the resource user, and lock the available resource quantity of each of the first matching resources according to the applied resource quantity.

[0099] If the resource manager detects that the available resource quantity of each first matching resource is not less than the requested resource quantity, it means that each first matching resource can be allocated to the resource user for use. At this time, the resource manager can determine that the resource user's resource application is approved, that is, the application is successful. After determining that the application is successful, the resource user can lock the available resource quantity of each first matching resource according to the requested resource quantity. Specifically, the locking method can be to subtract the requested resource quantity from the available resource quantity of the first matching resource and add the requested resource quantity to the locked resource quantity, so as to indicate that the part of the resources is locked and other resource users can no longer apply for it. After being informed that the application is approved, the resource user can actually use the requested resources.

[0100] Furthermore, in one embodiment, if the resource manager detects that the available resource quantity of each first matching resource is not all less than the applied resource quantity, that is, the available resource quantity of some first matching resources is less than the applied resource quantity, it means that the first matching resources cannot be allocated to the resource user for use. At this time, the resource manager can determine not to approve the resource user's resource application. Alternatively, in other embodiments, if the available resource quantity of a part of the first matching resources is less than the applied resource quantity, and the available resource quantity of another part of the first matching resources is not less than the applied resource quantity, the resource manager can first determine whether the first matching resources whose available resource quantity is not less than the applied resource quantity can be applied for separately; if so, the resource manager can determine to approve the application and feed back the tag information associated with the first matching resources to the resource user who initiated the resource application request, so that the resource user can actually use the resources associated with the tag information; the resource manager can also lock the available resource quantity of the first matching resources according to the applied resource quantity only. It should be noted that whether each resource can be applied for separately can be preset in the resource manager.

[0101] Compared with the existing resource management method, which can only fix the granularity of management and control in advance and cannot support dynamic adjustment and insertion of other granularities according to actual needs, in this embodiment, by associating various resources with tag information, and having the resource user carry a tag group including one or more tag information in the resource application request when applying for resources, the resource manager matches the tag information in the tag group with the tag information associated with each resource to obtain matching resources that match the tag group, and when the number of available resources of the matching resources is not less than the number of applied resources in the resource application request, the resource user's resource application is passed, and the available resources of the matching resources are locked according to the number of applied resources, thereby realizing the identification and division of resources through tag information, and the flexible setting of tag information can divide resources from different granularities, and resources are allocated through tag matching in the resource application process, so that the resource manager can re-divide or summarize resources at any time by changing the tag information associated with the resources, thereby improving the flexibility of resource allocation.

[0102] Moreover, due to the different scenarios targeted by clusters, resource allocation strategies are often very different. Resource users and providers have different definitions of resource attributes and are not in the same dimension (for example, Yarn resource providers control resources from the queue dimension, while users only focus on the CPU and memory information of a single node and a single process, and need to distinguish between the resources of the driver and the executor. The resources of the driver are not managed in the Yarn queue, which is very complicated). Therefore, the existing resource management method can only manage the resources of one cluster at a time. Each cluster must deploy its own resource manager, which increases operating costs and reduces the efficiency of resource allocation. In this embodiment, resources are divided by labeling resources, and resources are allocated by label matching, so that resources of different clusters can also be identified and allocated by the resource manager by marking labels, so that one resource manager can be used to manage resources for multiple clusters, without the need to deploy multiple resource managers, reducing operating costs and improving the efficiency of cross-cluster resource allocation.

[0103] Further, based on the above-mentioned first embodiment, a second embodiment of the resource management method of the present invention is proposed. In this embodiment, in step S10, each piece of first tag information in the first tag group carried by the resource application request is matched with each piece of preset tag information respectively, and the step of obtaining the target tag number associated with the preset tag information that successfully matches the first tag information includes:

[0104] Step S101, for each piece of first tag information in the first tag group carried by the resource application request, extract each tag value in each piece of the first tag information;

[0105] In one implementation, in order to improve the efficiency of tag matching and resource management, a fast query table may be used to back up each piece of tag information.

[0106] Specifically, a tag table can be used to store each preset tag information. A row of records in the tag table includes a preset tag information and a tag number associated with the preset tag information. In the tag table, a serialized format (such as json format) is generally used to store the tag values ​​in the tag information. A tag value includes a name and a content. The serialized format is not convenient for retrieval. For example, an engine type tag information includes a tag name "engineType" and two tag values. The name and content of one tag value are "engineType" and "io" respectively, and the name and content of the other tag value are "version" and "1" respectively. The two tag values ​​together constitute the engine type tag. The json format is used to represent the two tag values ​​as "{"engineType":"io","version":"1"}".

[0107] In order to facilitate retrieval and comparison, a quick query table is used to store a backup of the tag information in the tag table. A row of records in the quick query table includes a tag value in a preset tag information and a tag number associated with the preset tag information. The tag number in the quick query table is the same as the tag number used in the tag table. It can be understood that when a tag information includes multiple tag values, it is split into multiple rows of records in the quick query table. For example, for the two tag values ​​in the above engine type tag information, assuming that the tag number of the engine type tag information is 1, then the tag value of the engine type tag information is recorded in the quick query table as:

[0108] Tag ID Name of the value Value Content 1 engineType io 1 version 1

[0109] In one embodiment, the Figure 3 The resource table, tag table, quick query table and resource tag association table are constructed in this way. In the tag table, a row of records may also include a field with the number of values, that is, the tag information of the row of records includes several tag values, for example, the number of values ​​of the above engine type tag information is 2.

[0110] After receiving the resource application request, the resource manager may respectively extract each label value in each piece of first label information in the first label group carried in the resource application request.

[0111] Step S102, searching for each of the tag values ​​in a preset quick query table, and obtaining a tag number associated with the found tag value in the preset quick query table;

[0112] After extracting the tag value, the resource manager searches for each tag value in the quick query table. That is, it uses the name and content of the value in the tag value as conditions to search the quick query table, finds one or more rows of records with the same name and content of the value, and obtains the tag number in the found record (hereinafter referred to as the candidate number).

[0113] Step S103: If the tag number is found in the preset tag table, the tag number is used as the target tag number, wherein a row of records in the preset tag table includes a preset tag information and a tag number associated with the preset tag information, and a row of records in the preset quick query table includes a tag value in a preset tag information and a tag number associated with the preset tag information.

[0114] Search for the candidate number in the tag table. If the candidate number is found, use the candidate number as the target tag number. If the candidate number is not found, it means that the tag information corresponding to the tag number has been deleted from the tag table, but it may not have been deleted synchronously in the quick query table in time. At this time, the candidate number can be discarded and regarded as not matching the target tag number. It is understandable that a tag information may include multiple tag values, and a tag value may find multiple tag numbers in the quick query table, so there may be multiple target tag numbers found in the end.

[0115] In this embodiment, a label table is used to associate and store label numbers and label information, and a quick query table is used to back up and store label values ​​in each label information. The label value is searched through the quick query table, and then the label information is matched, thereby improving the efficiency of label information matching and the efficiency of resource management.

[0116] Further, based on the above-mentioned first and / or second embodiments, a third embodiment of the resource management method of the present invention is proposed. In this embodiment, after step S40, the method further includes:

[0117] Step S50, adding a new resource, and using the applied resource quantity as the locked resource quantity of the new resource;

[0118] Step S60: Associating the newly added resource with each piece of the first tag information in the first tag group respectively.

[0119] In this embodiment, after approving the resource application of the resource user, the resource manager may record the resource quantity applied for by the resource user, so as to manage the resource quantity during the subsequent actual use and resource recycling.

[0120] Specifically, a resource user can carry a tag information that can uniquely identify the resource user in a resource application request, that is, the first tag group includes a first tag information for uniquely identifying the resource user. Generally, a characteristic attribute that is different for each resource user is used as the tag information that serves as a unique identifier, for example, the resource user's engine instance tag information.

[0121] After the resource manager approves the resource application of the resource user, it can add a new resource, and use the number of resources applied for in the resource application request as the number of locked resources for the new resource, and associate the new resource with each piece of first tag information in the first tag group. Specifically, in one embodiment, the resource manager can add a new row of records in the resource table, automatically generate the resource number of the record according to the current sorting, fill the number of resources applied for as the data in the field of the number of locked resources in the record, and other fields can be left blank; find the tag number associated with each piece of first tag information in the first tag group in the tag table or the quick query table, if the first tag information is not recorded in the tag table, then first add a record of the first tag information in the tag table and the quick query table; store the tag number of each piece of first tag information in association with the resource number of the new resource in the resource tag association table, so as to associate the new resource with each piece of first tag information in the first tag group.

[0122] That is, the newly added resource is not a real available resource, but is only used to record the number of resources applied for by resource users. Since the resource is associated with the tag information used to uniquely identify the resource user, other resource users will not be matched with the resource when applying for it, and thus it will not be mistakenly used as an available resource.

[0123] Further, based on the above-mentioned first, second and / or third embodiments, a fourth embodiment of the resource management method of the present invention is proposed. In this embodiment, after step S40, the method further includes:

[0124] Step S70, receiving resource usage report information from the resource user, matching each piece of second tag information in the second tag group carried in the resource usage report information with tag information associated with each preset resource, and obtaining a second matching resource that successfully matches the second tag group, wherein the second tag group is the same as the first tag group;

[0125] After confirming that the resource application is approved, the resource user can actually use the resources according to the tag information and the number of resources applied for in the resource application request. After the resource user actually uses the resources, the resource user can send resource usage reporting information to the resource manager. The resource usage reporting information can carry a second tag group, and the second tag group includes one or more second tag information. The resource usage reporting information can also include the number of resources actually used by the resource user (hereinafter referred to as the actual number of resources used). It should be noted here that the second tag group is the same as the first tag group, that is, when the resource user reports the actually used resources, the tag information carried in the resource usage reporting information is the same as the tag information carried in the resource application request when applying for resources.

[0126] After receiving the resource usage report information, the resource manager can match each second tag information in the second tag group with the tag information associated with each preset resource to obtain a second matching resource that successfully matches the second tag group. Among them, the process of matching the second tag information with the tag information associated with each preset resource is similar to the process of matching the first tag information with the tag information associated with each preset resource in the above embodiment, and will not be repeated in detail here. Since the second tag group is the same as the first tag group, when there is no new resource item between the resource user making a resource application request and reporting the resource usage report information, the second matching resource is the same as the first matching resource. If a resource item for recording the number of resources applied by the resource user is added after the resource user makes a resource application request, since the resource item is associated with each first tag information in the first tag group, by matching each second tag information in the second tag group, in addition to being able to match the same resource item as the first matching resource, the resource item for recording the number of resources applied by the resource user can also be matched.

[0127] Step S80, detecting whether the locked resource quantity of each of the second matching resources is not less than the actual used resource quantity in the resource usage reporting information;

[0128] After matching and obtaining each second matching resource, the resource manager can detect whether the number of locked resources in each second matching resource is not less than the number of actually used resources in the resource usage report information. It can be understood that after the resource user passes the resource application, the resource is actually used according to the number of resources applied in the resource application request, that is, the actually used resources do not exceed the number of resources applied, and the resource manager locks the available resources of the first matching resource according to the number of resources applied, so under normal circumstances, the number of locked resources in each second matching resource is not less than the number of actually used resources, and if the number of locked resources in a second matching resource is less than the number of actually used resources, it means that the number of actually used resources reported by the resource user is incorrect.

[0129] Step S90: If yes, convert the locked resource quantity in each of the second matching resources into the used resource quantity according to the actually used resource quantity.

[0130] If the resource manager determines that the locked resource quantity of each second matching resource is not less than the actual used resource quantity in the resource usage report information, it means that the actual used resource quantity reported by the resource user is correct. At this time, the resource manager can convert the locked resource quantity in each second matching resource into the used resource quantity according to the actual used resource quantity. Specifically, the resource manager can subtract the actual used resource quantity from the locked resource quantity and add the actual used resource quantity to the used resource quantity to complete the conversion.

[0131] It is understandable that when the second matching resource includes a resource item for recording the number of resources applied for by the resource user, the number of resources actually used by the resource user is recorded by converting the number of locked resources of the resource item into the number of used resources. In one embodiment, the resource user can report the number of resources actually used to the resource manager multiple times, and the tag information carried each time is the same, so that the resource manager can find the same resource item each time, and then update the resource quantity information in the resource item to accurately record the usage of each resource, so as to facilitate accurate management of resources.

[0132] If the locked resource quantity of each second matching resource is not less than the actual number of used resources, that is, the locked resource quantity of some second matching resources is less than the actual number of used resources, then it means that the actual number of used resources carried by the resource user in the resource usage report information is incorrect. At this time, the resource manager can feedback an error alarm to the resource user and do not process the resource usage report information.

[0133] Furthermore, after step S90, the method further includes:

[0134] Step A10, receiving a resource recycling request from the resource user, matching each piece of third tag information in the third tag group carried by the resource recycling request with tag information associated with each preset resource, and obtaining a third matching resource that successfully matches the third tag group, wherein the third tag group is the same as the first tag information group;

[0135] After the resource user has finished using the resources, the resource user can send a resource recovery request to the resource manager. The resource recovery request can carry a third tag group, which includes one or more third tag information. The resource recovery request can also include the number of resources that the resource user requests to be recovered (hereinafter referred to as the recovered resource number). It should be noted that the third tag group is the same as the second tag group, that is, when the resource user requests to recover resources, the tag information carried in the resource recovery request is the same as the tag information carried in the resource application request when the resource user applies for resources.

[0136] After receiving the resource recovery request, the resource manager can match each third tag information in the third tag group with the tag information associated with each preset resource to obtain a third matching resource that successfully matches the third tag group. Among them, the process of matching the third tag information with the tag information associated with each preset resource is similar to the process of matching the first tag information with the tag information associated with each preset resource in the above embodiment, and will not be repeated in detail here. Similarly, since the third tag group is the same as the first tag group, when there is no new resource item between the resource user's resource application request and the resource recovery request, the third matching resource is the same as the first matching resource. If a resource item for recording the number of resources applied by the resource user is added after the resource user makes a resource application request, since the resource item is associated with each first tag information in the first tag group, by matching each third tag information in the third tag group, in addition to being able to match the same resource item as the first matching resource, the resource item for recording the number of resources applied by the resource user can also be matched.

[0137] Step A20, detecting whether the used resource quantity of each of the third matching resources is not less than the recycled resource quantity in the resource recycling request;

[0138] After matching and obtaining each third matching resource, the resource manager can detect whether the used resource quantity of each third matching resource is not less than the recycled resource quantity in the resource recycling request. It can be understood that after the resource user passes the resource application, the resource is actually used according to the requested resource quantity in the resource application request, that is, the actually used resources do not exceed the requested resource quantity, and the resource manager converts the locked resource quantity of the second matching resource into the used resource quantity according to the actually used resource quantity. Therefore, under normal circumstances, the used resource quantity of each third matching resource is not less than the recycled resource quantity. If the used resource quantity of a third matching resource is less than the recycled resource quantity, it means that the recycled resource quantity reported by the resource user is incorrect.

[0139] Step A30: If yes, then recycle the used resource quantity of each of the third matching resources according to the recycle resource quantity.

[0140] If the resource manager determines that the used resource quantity of each third matching resource is not less than the recycled resource quantity in the resource recycling request, it means that the recycled resource quantity reported by the resource user is correct. At this time, the resource manager can recycle the used resource quantity of each third matching resource according to the recycled resource quantity. Specifically, the resource manager can subtract the recycled resource quantity from the used resource quantity and add the recycled resource quantity to the usable resource quantity to complete resource recycling, so as to facilitate the resource user or other resource users to make the next resource application.

[0141] It is understandable that when the third matching resource includes a resource item for recording the number of resources applied for by the resource user, the number of resources reclaimed by the resource user is recorded by reclaiming the used number of resources of the resource item. In one embodiment, the resource user can send resource reclaim requests to the resource manager multiple times, and the tag information carried each time is the same, so that the resource manager can find the same resource item each time, and then update the resource quantity information in the resource item to accurately record the usage of each resource, so as to facilitate accurate management of resources.

[0142] If the used resource quantities of each third matching resource are not all less than the recycled resource quantities, that is, the used resource quantities of some third matching resources are less than the recycled resource quantities, it means that the recycled resource quantities carried by the resource user in the resource recycling request are incorrect. At this time, the resource manager can feedback an error alarm to the resource user and do not process the resource recycling request.

[0143] Furthermore, in one embodiment, after step A30, the method further includes:

[0144] Step A40, detecting whether the used resource quantity of the target resource associated with the user tag information in the third matching resource is zero;

[0145] Step A50: If yes, delete the target resource.

[0146] When the resource manager adds a resource item for recording the number of resources applied for by the resource user after making a resource application request, when the resource manager matches according to the third tag group in the resource recovery request, the third matching resource obtained by matching will include the resource item, which is the only resource associated with the user tag information in the third matching resource. After receiving the resource recovery request and recovering the used resources in the third matching resource according to the number of recovered resources, the resource manager can find the resource associated with the user tag information from the third matching resource. The resource item is the resource item used to record the number of resources applied for by the resource user, and the resource item is called the target resource for distinction; the resource manager detects whether the number of used resources of the target resource is zero; if so, it means that the resource user has recovered all the used resources. At this time, the resource manager can delete the target resource, for example, delete it from the resource table, and can delete the association between the target resource and the tag information.

[0147] In this embodiment, the resource user carries the same tag information in the resource application request, resource usage report information and resource recovery request, and the resource manager searches for the corresponding resource item by tag matching, and updates the resource quantity information of the resource item accordingly, so that the resource manager can manage resources through tag information in the entire resource management process of resource application, resource usage report and resource recovery, so that the role of flexible division of resources through tag information can run through the entire resource management process, further improving the flexibility of resource management.

[0148] Further, based on the above-mentioned first, second, third and / or fourth embodiments, a fifth embodiment of the resource management method of the present invention is proposed. In this embodiment, the method further includes:

[0149] Step A60, adding a new resource according to the resource quantity information carried in the resource registration request, and associating each fourth tag information carried in the resource registration request with the newly added resource, wherein the resource registration request is initiated by a resource provider or generated according to resource usage restriction information set for a resource user, and when generated according to the resource usage restriction information, each fourth tag information is generated according to the restriction conditions in the resource usage restriction information and the resource quantity information is generated according to the restricted resource quantity in the resource usage restriction information.

[0150] In this embodiment, a resource provider can send a resource registration request to the resource manager, or an administrator's operation of setting resource usage restriction information for a resource user can trigger a resource registration request in the resource manager. The resource registration request carries a resource quantity information and one or more tag information (referred to as fourth tag information for distinction), the resource quantity information indicating the resource quantity of the resource to be registered, and the tag information is tag information used to mark the resource. After receiving the resource registration request, the resource manager adds a new resource according to the resource quantity information carried in the request, that is, uses the resource quantity information as the resource quantity information of the newly added resource, and associates each fourth tag information with the newly added resource.

[0151] Specifically, in one embodiment, the resource manager can add a new row of records in the resource table, automatically generate a resource number for the record according to the current sorting, and fill in the resource quantity information in the resource registration request into each field of the record accordingly; the resource manager can search in the tag table or the quick query table whether tag information identical to the fourth tag information already exists. If not found, a new record of the fourth tag information is added to the tag table and the quick query table; the tag number of each piece of fourth tag information is associated with the resource number of the newly added resource and stored in the resource tag management table, thereby completing the registration of the resource in the resource registration request.

[0152] When a resource registration request is initiated by a resource provider, the resource items added according to the resource registration request are actually available resources. In this case, the resource quantity information carried in the resource registration request generally includes information on fields such as resource type, maximum resource quantity, minimum resource quantity, used resource quantity, remaining resource quantity (or available resource quantity) and locked resource quantity.

[0153] When the resource registration request is generated according to the resource usage restriction information set for the resource user, the resource item added according to the resource registration request is not an actually available resource, but is used to restrict the resource user who meets the tag information of the resource item in the process of tag information matching. Or it can be said that in this embodiment, the restriction information on the resource user is embodied in the form of virtual resources, so that in the process of resource application, resource usage reporting information reporting, and resource recovery, the resource manager only needs to execute the logic of tag information matching, and does not need to specifically execute the judgment logic of whether the resource applied by the resource user is restricted, which simplifies the management process of the resource manager and improves the resource management efficiency. In this case, each fourth tag information in the resource registration request can be generated according to the restriction conditions in the resource usage restriction information, and the resource quantity information in the resource registration request can be generated according to the limited resource quantity in the resource usage restriction information. Generally, when the restriction conditions include multiple conditions that are all satisfied, a tag information can be generated according to each condition, and then each tag information can be combined into a tag information, that is, multiple tag names are synthesized into a tag name by string concatenation, and multiple tag values ​​are connected in series as the tag value of a tag information. The resource type for which the resource quantity is limited is used as the resource type in the resource registration request, the resource quantity limited is used as the available resource quantity of the resource in the resource registration request, and other fields can be set to empty.

[0154] Furthermore, in one embodiment, the method further comprises:

[0155] Step A70, receiving a resource offline request, and matching each fifth tag information carried in the resource offline request with tag information associated with each preset resource, so as to search for a resource to be offline in each preset resource, wherein the resource to be offline is a resource associated with the same tag information as that corresponding to each fifth tag information;

[0156] Step A80: If the resource to be taken offline is found, the resource quantity information of the resource to be taken offline and the association information between the resource to be taken offline and the tag information are deleted to take the resource offline.

[0157] The resource provider can send a resource offline request to the resource manager. After receiving the resource offline request, the resource manager can offline the resource according to the resource offline request. Specifically, the resource offline request carries one or more tag information, hereinafter referred to as the fifth tag information for distinction. The resource manager matches each fifth tag information in the resource offline request with the tag information associated with each preset resource to find out whether there are resources to be offline. Specifically, when the tag information associated with a resource is completely identical to the tag information in the resource offline request, the resource is the resource to be offline; if the resource to be offline is found, the resource manager can delete the resource quantity information of the resource to be offline, and also delete the association between the resource to be offline and the tag information, thereby completing the offline of the resource. If the resource to be offline is not found, that is, there is no pre-resource associated with all the fifth tag information, the resource manager can feedback an error alarm to the sender of the resource offline request, or not process the resource offline request.

[0158] In one implementation, the process of resource registration, resource application, and resource recovery is described by way of example.

[0159] 1. Assume that there is a DEV cluster, in which there is a machine named provider01, which can provide 50 CPUs and 500G memory. It communicates with computing and storage-related applications such as resource managers through the engine manager on port 8080. Then, as a resource provider, it submits its resource-related information to the resource manager for storage. The machine sends an RPC message to the resource manager, which indicates the operation type, tag information, and resource quantity information in Json format. The specific format is:

[0160] {"RegisterResource":{"labels":[{"key":"cluster","value":"{"clusterName":"DEV"}"},{"key":"emInstance","value":"{"instance":"provider01:8080","serviceNa me":"engineConnManager"}"}],"resources":{…}}}.

[0161] A. After receiving the message, the resource manager identifies it as a resource registration request based on "RegisterResource", then reads the labels field to obtain the labels in JSON format. After parsing the key and value in the JSON format, it obtains the label information for the provider:

[0162]

[0163] These tag information are stored in the tag table in the manner described above, assuming that their row IDs are 001 and 002. The tag description is a field added for ease of understanding, and is not actually stored in the tag table.

[0164] B. The resource quantity information of the provider is stored as a row in the resource table, assuming that the row ID is 001. (The storage format of the resource itself is not limited in this embodiment and will not be described in detail again).

[0165] C. In the association table between tag information and resources, there will be the following records:

[0166] Tag ID Resource ID 001 001 002 001

[0167] 2. Assume that there is a yarn queue named IDE in the DEV cluster, which can provide 20 CPUs and 100G queue memory. The Yarn resource provider sends an RPC message to the resource manager, indicating the operation type, label information and resource quantity information of the yarn queue in Json format. The specific format is:

[0168] {"RegisterResource":{"labels":[{"key":"cluster","value":"{"clusterName":"DEV"}"},{"key":"yarn","value":"{"url":"0.0.0.0:8899","queueName":"IDE"}"}],"re sources":{…}}}.

[0169] Compared with the resource provider in step 1, this yarn queue is also on the Dev cluster, so their cluster cluster labels are the same. The difference is that the one in step 1 is an engine manager, which represents the local resources of a machine, so it has the emInstance label, indicating its machine name and port; the one in step 2 is a yarn queue, so it has the yarn label, indicating the service interface address and queue name of this yarn.

[0170] A. Parse the json using the same steps as described in step 1. The tag information for the provider is:

[0171]

[0172] The cluster label reuses the record with ID 001, and the record with another label is assumed to be 003.

[0173] B. The resource quantity information of the provider is stored as a row in the resource table, assuming that the row ID is 002.

[0174] C. In the association table between tag information and resources, there will be the following records:

[0175] Tag ID Resource ID 001 002 003 002

[0176] 3. Assume that user hduser001 is set up. When it starts the spark engine in the name of BI, it can only use 5 CPUs and 5G memory at most. (The resource usage limit is actually similar to a provider, but it is not a real provider. However, when processing the logic of resource application, there is no need to distinguish between the two).

[0177] A. The tag information generated according to this setting is:

[0178]

[0179] Since it is a combination of tags and cannot be separated and combined with resources separately, it generates a combined tag as follows:

[0180]

[0181] B. Add a new resource in the resource table based on this setting. Assume the row ID is 003.

[0182] C. In the association table between tag information and resources, there will be the following records:

[0183] Tag ID Resource ID 004 003

[0184] 4. At this time, a resource user starts to apply for resources from the resource manager. Assume that it is a user named hduser001, who is going to start a spark engine in the name of BI on port 10001 of machine consumer01, and is going to apply for resources of the DEV or SIT environment, which are 10 CPUs, 10G memory, and 2 CPUs and 2G memory of the IDE queue.

[0185] The resource user sends an RPC message to the resource manager in the following format:

[0186] {"RequestResource":{"labels":[{"Key":"user","value":"{"userName":"hduser001"}"},{"Key":"engineInstance","value":"{"instance":"consumer01:10001","serv iceName":"engineConn"}"},{"Key":"creator","value":"{"creator":"BI"}"},{"Key":"engineType","value":"{"engineType":"spark","version":"*"}"},{"Key":"Yarn","va lue":"{"url":"*","queueName":"IDE"}"},{"Key":"cluster","value":"{"clusterName":"DEV"}"},{"Key":"cluster","value":"{"clusterName":"SIT"}"}],"resources":{…}}}

[0187] This format is similar to the resource registration format described above, except that it starts with "RequestResource" to identify it as a resource application message, labels are the labels carried by the application, and resources are the resources that the user is going to apply for.

[0188] A. The tag information carried by this resource application is as follows:

[0189]

[0190] Since you can apply for resources from two clusters, two cluster tags are included.

[0191] B. After receiving the above resource request RPC message, the resource manager will filter out qualified resources based on the tag information. Specifically, it will traverse each value name and value content of each tag to find the corresponding tag and resource and find a large candidate resource pool, such as:

[0192] ① Taking the user tag as an example, in the resource application, there is a tag with {"userName":"hduser001"}.

[0193] a. First, find a record from the quick query table with [value name is "userName" and value content is "hduser001"] as the condition (according to the assumption in step 3, this ID is 004);

[0194] b. Then, based on the tag ID of this record, find the tag that has a foreign key association with this ID from the tag table, that is, the combined tag in step 3;

[0195] c. Finally, after confirming that the queried tag actually exists, find the corresponding resource ID through the resource tag association table (based on the assumption in step 3, this ID is 003), and then find the resource with a foreign key association with this ID from the resource table, that is, the resource corresponding to this user tag is matched.

[0196] ② Similarly, in the above resource application, the user, creator, or engineType tags can match the resources set in step 3. The cluster tag can match the resources of the resource provider in steps 1 and 2. The yarn tag can also match the resources in step 2. I will not list all the possibilities here.

[0197] C. Among the candidate resources, match the tags associated with each resource to see if a match can be found in the tags applied for the resource.

[0198] ① For example, the resource matched by the user tag in the previous step has user, creator, and engineType tags. These tags and values ​​must exist in the tags carried by the resource application at the same time to be considered a successfully matched resource. In the above example, you can see that the resource can be matched successfully.

[0199] ② There are also cases where the matching fails: Assume that the resource provided in step 3 has an additional tag of engineType=hive when it is reported, but this tag is not carried by this resource application, so it cannot be matched by this application. If you want the resource provided in step 3 to be matched by the tag of engineType=hive, you can adjust the resource allocation strategy by first offline the resource, and then add the tag of engineType=hive in the registration RPC message when re-registering.

[0200] D. Resource tags. You can choose not to participate in this step of comparison during the reporting stage. For example, the emInstance tag usually indicates the machine and port where the resource provider is located. You do not need to specify the machine and port range for each application. However, if the resource application is actively accompanied by an emInstance tag, you can still use this tag to add the corresponding resource to the candidate resource pool in the previous step.

[0201] E. After finding a match, compare the number of requested resources with the number of available resources (such as the number of CPUs and memory). If the available resources of all matching resources are not less than the number of requested resources, the request is approved, and the resource manager locks the resources according to the tag and waits for actual use:

[0202] ① According to the number of resources applied for, add a new record to the resource table and obtain the ID of this record.

[0203] ②Record the resource record ID and all the tag IDs carried by the resource application in the association table. (Since the engineInstance tag is unique to each engine, the resources recorded here will not be mistakenly used as available resources)

[0204] F. After the resources are used up, the resource manager recycles the resources.

[0205] ① Before closing, the resource user (usually a process) will notify the resource manager to recycle resources through RPC messages. The format is:

[0206] {"ReleaseResource":{"labels":[…(label content is consistent with the one passed when applying for resources)],"resources":{…}}}

[0207] ② After receiving the resource recycling request, the resource manager finds the tag ID according to the key and value of each tag, and then finds the associated resource record. It subtracts the number of resources carried in the RPC message from the number of resources used, and then updates the resource record.

[0208] ③ For the resource records corresponding to the engineInstance tag, delete them when recycling.

[0209] In addition, the embodiment of the present invention also provides a resource management device, referring to Figure 4 , the device comprises:

[0210] The receiving module 10 is used to receive a resource application request from a resource user, and match each piece of first tag information in the first tag group carried in the resource application request with each piece of preset tag information, to obtain a target tag number associated with the preset tag information that successfully matches the first tag information;

[0211] A first detection module 20 is used to take resources associated with the target tag number in various preset resources as candidate resources, and detect whether all matching preset tag information associated with the candidate resources have tag information that is successfully matched in the first tag group;

[0212] A second detection module 30 is used to, if all exist, use the candidate resource as the first matching resource that successfully matches the first tag group, and detect whether the available resource quantity of each of the first matching resources is not less than the requested resource quantity in the resource application request;

[0213] The determination module 40 is configured to, if yes, determine that the resource application of the resource user is approved, and lock the available resource quantity of each of the first matching resources according to the applied resource quantity.

[0214] Furthermore, the receiving module 10 includes:

[0215] An extraction unit, configured to extract, for each piece of first label information in the first label group carried in the resource application request, each label value in each piece of the first label information;

[0216] A search unit, used to search for each of the tag values ​​in a preset quick query table, and obtain a tag number associated with the found tag value in the preset quick query table;

[0217] A determination unit is used to use the label number as the target label number if the label number is found in the preset label table, wherein a row of records in the preset label table includes a piece of preset label information and a label number associated with the preset label information, and a row of records in the preset quick query table includes a label value in a piece of preset label information and a label number associated with the preset label information.

[0218] Furthermore, the first tag group includes a piece of user tag information for uniquely identifying the resource user, and the device further includes:

[0219] The first adding module is used to add a new resource and use the applied resource quantity as the locked resource quantity of the new resource;

[0220] An associating module is used to associate the newly added resource with each of the first tag information in the first tag group.

[0221] Further, the receiving module 10 is further used to: receive resource usage reporting information of the resource user, match each second tag information in the second tag group carried by the resource usage reporting information with the tag information associated with each preset resource, and obtain a second matching resource that successfully matches the second tag group, wherein the second tag group is the same as the first tag group;

[0222] The second detection module 30 is further used to detect whether the locked resource quantity of each second matching resource is not less than the actual used resource quantity in the resource usage reporting information;

[0223] The device also includes:

[0224] A conversion module is used to convert the locked resource quantity in each second matching resource into the used resource quantity according to the actually used resource quantity.

[0225] Further, the receiving module 10 is also used to: receive a resource recycling request from the resource user, match each piece of third tag information in the third tag group carried by the resource recycling request with the tag information associated with each preset resource, and obtain a third matching resource that successfully matches the third tag group, wherein the third tag group is the same as the first tag information group;

[0226] The second detection module 30 is further used to: detect whether the used resource quantity of each of the third matching resources is not less than the recycled resource quantity in the resource recycling request;

[0227] The device also includes:

[0228] A recycling module is used to recycle the used resource quantity of each of the third matching resources according to the recycled resource quantity.

[0229] Furthermore, the detection module 30 is further used to: detect whether the used resource quantity of the target resource associated with the user tag information in the third matching resource is zero;

[0230] The device also includes:

[0231] A deletion module is used to delete the target resource if yes.

[0232] Furthermore, the device also includes:

[0233] The second adding module is used to add a new resource according to the resource quantity information carried by the resource registration request, and associate each fourth tag information carried by the resource registration request with the newly added resource, wherein the resource registration request is initiated by the resource provider or generated according to the resource usage restriction information set for the resource user. When generated according to the resource usage restriction information, each fourth tag information is generated according to the restriction conditions in the resource usage restriction information and the resource quantity information is generated according to the restricted resource quantity in the resource usage restriction information.

[0234] Further, the receiving module 10 is also used to: receive a resource offline request, and match each fifth tag information carried by the resource offline request with the tag information associated with each preset resource, so as to search for a resource to be offline in each preset resource, wherein the resource to be offline is a resource associated with the same tag information as that corresponding to each fifth tag information;

[0235] The device also includes:

[0236] The offline module is used to delete the resource quantity information of the resource to be offline and the association information between the resource to be offline and the tag information if the resource to be offline is found, so as to offline the resource to be offline.

[0237] The extended contents of the specific implementation methods of the resource management device of the present invention are basically the same as those of the above-mentioned resource management method embodiments, and will not be elaborated here.

[0238] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a resource management program is stored. When the resource management program is executed by a processor, the steps of the resource management method described below are implemented.

[0239] The various embodiments of the resource management device and computer-readable storage medium of the present invention may refer to the various embodiments of the resource management method of the present invention, and will not be described in detail here.

[0240] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0241] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0242] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0243] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A resource management method, characterized in that: The method comprises the following steps: Receive a resource application request from a resource user, match each piece of first tag information in a first tag group carried in the resource application request with each piece of preset tag information, and obtain a target tag number associated with the preset tag information that successfully matches the first tag information; The resources associated with the target tag number in each preset resource are taken as candidate resources, and it is detected whether each matchable preset tag information associated with the candidate resource has tag information that is successfully matched in the first tag group; If all of them exist, the candidate resource is used as the first matching resource that successfully matches the first tag group, and it is detected whether the available resource quantity of each of the first matching resources is not less than the requested resource quantity in the resource application request; If so, determine to approve the resource application of the resource user, and lock the available resource quantity of each of the first matching resources according to the applied resource quantity; receiving resource usage report information from the resource user, matching each piece of second tag information in a second tag group carried in the resource usage report information with tag information associated with each preset resource, and obtaining a second matching resource that successfully matches the second tag group, wherein the second tag group is the same as the first tag group; Detecting whether the locked resource quantity of each of the second matching resources is not less than the actual used resource quantity in the resource usage reporting information; If so, convert the locked resource quantity in each of the second matching resources into the used resource quantity according to the actually used resource quantity.

2. The resource management method according to claim 1, characterized in that: The step of matching each piece of first tag information in the first tag group carried by the resource application request with each piece of preset tag information to obtain a target tag number associated with the preset tag information successfully matched with each piece of the first tag information comprises: For each piece of first label information in the first label group carried in the resource application request, extract each label value in each piece of the first label information; Searching for each of the tag values ​​in a preset quick query table, and obtaining a tag number associated with the found tag value in the preset quick query table; If the tag number is found in the preset tag table, the tag number is used as the target tag number, wherein a row of records in the preset tag table includes a preset tag information and a tag number associated with the preset tag information, and a row of records in the preset quick query table includes a tag value in a preset tag information and a tag number associated with the preset tag information.

3. The resource management method according to any one of claims 1 or 2, characterized in that: The first tag group includes a user tag information for uniquely identifying the resource user, and after the step of locking the available resource quantity of each first matching resource according to the applied resource quantity, the step further includes: Add a new resource and use the requested resource quantity as the locked resource quantity of the new resource; The newly added resource is associated with each of the first tag information in the first tag group.

4. The resource management method according to claim 1, characterized in that: After the step of converting the locked resource quantity in each of the second matching resources into the used resource quantity according to the actually used resource quantity, the method further includes: Receive a resource recycling request from the resource user, match each piece of third tag information in a third tag group carried by the resource recycling request with tag information associated with each preset resource, and obtain a third matching resource that successfully matches the third tag group, wherein the third tag group is the same as the first tag information group; Detecting whether the used resource quantity of each of the third matching resources is not less than the recycled resource quantity in the resource recycling request; If so, the used resource quantity of each of the third matching resources is recycled according to the recycled resource quantity.

5. The resource management method according to claim 4, characterized in that: If so, after the step of recycling the used resource quantity of each of the third matching resources according to the recycled resource quantity, the method further includes: Detecting whether the used resource quantity of the target resource associated with the user tag information in the third matching resource is zero; If so, the target resource is deleted.

6. The resource management method according to any one of claims 1 or 2, characterized in that: The method further comprises: A new resource is added according to the resource quantity information carried in the resource registration request, and each fourth tag information carried in the resource registration request is respectively associated with the newly added resource, wherein the resource registration request is initiated by a resource provider or generated according to resource usage restriction information set for a resource user, and when generated according to the resource usage restriction information, each fourth tag information is generated according to the restriction conditions in the resource usage restriction information and the resource quantity information is generated according to the restricted resource quantity in the resource usage restriction information.

7. The resource management method according to any one of claims 1 or 2, characterized in that: The method further comprises: Receive a resource offline request, and match each fifth tag information carried by the resource offline request with the tag information associated with each preset resource, so as to search for a resource to be offline in each preset resource, wherein the resource to be offline is a resource associated with the same tag information as that corresponding to each fifth tag information; If the resource to be taken offline is found, the resource quantity information of the resource to be taken offline and the association information between the resource to be taken offline and the tag information are deleted to take the resource offline.

8. A resource management device, characterized in that: The device comprises: A receiving module, used to receive a resource application request from a resource user, and match each piece of first tag information in a first tag group carried in the resource application request with each piece of preset tag information, to obtain a target tag number associated with the preset tag information that successfully matches the first tag information; A first detection module is used to take resources associated with the target tag number in various preset resources as candidate resources, and detect whether all matching preset tag information associated with the candidate resources have tag information that is successfully matched in the first tag group; A second detection module is used to, if all exist, use the candidate resource as a first matching resource that successfully matches the first tag group, and detect whether the available resource quantity of each of the first matching resources is not less than the requested resource quantity in the resource application request; a determination module, configured to, if yes, determine the resource application of the resource user, and lock the available resource quantity of each of the first matching resources according to the applied resource quantity; The receiving module is further used to: receive resource usage reporting information from the resource user, match each piece of second tag information in the second tag group carried by the resource usage reporting information with tag information associated with each preset resource, and obtain a second matching resource that successfully matches the second tag group, wherein the second tag group is the same as the first tag group; The second detection module is further used to: detect whether the locked resource quantity of each second matching resource is not less than the actual used resource quantity in the resource usage reporting information; The device further includes: a conversion module, configured to convert the locked resource quantity in each of the second matching resources into the used resource quantity according to the actually used resource quantity.

9. A resource management device, characterized in that: The resource management device comprises: a memory, a processor, and a resource management program stored in the memory and executable on the processor. When the resource management program is executed by the processor, the steps of the resource management method according to any one of claims 1 to 7 are implemented.

Images