Accelerator card resource monitoring method, device, storage medium and program product

Abstract

The embodiments of the present invention provide an accelerator card resource monitoring method, device, storage medium and program product, which relate to the field of computer technology. The method includes: obtaining the process identifier of the target service process in the target host and the accelerator card resource usage information corresponding to the target service process; determining the container group identifier of the target container group to which the target container corresponding to the target service process belongs according to the process identifier; obtaining metadata information corresponding to the container group identifier according to the container group identifier, wherein the metadata information includes multiple service label information; parsing the target service label information from the metadata information, wherein the target service label information is at least part of the multiple service label information; generating the first accelerator card resource monitoring information corresponding to the target service process according to the target service label information, the process identifier and the accelerator card resource usage information. The monitoring granularity of this scheme is finer, and can meet the needs of finer granularity accelerator card resource monitoring.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an accelerator card resource monitoring method, device, storage medium and program product. Background Art

[0002] In application scenarios involving large-scale computing, such as large models, containerized applications can significantly improve computing efficiency by using accelerator cards, which can include graphics processing units (GPUs), field programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs). In this process, the use of accelerator card resources can be understood by monitoring the accelerator card resources.

[0003] At present, the accelerator card resources are generally monitored at the hardware granularity (i.e., host level) (for example, monitoring the usage rate and temperature of the accelerator card). The monitoring granularity is coarse and cannot meet some finer-grained accelerator card resource monitoring needs. Summary of the invention

[0004] The embodiments of the present invention provide an accelerator card resource monitoring method, device, storage medium and program product, which have fine monitoring granularity and can meet the accelerator card resource monitoring requirements of finer granularity.

[0005] In a first aspect, an embodiment of the present invention provides an accelerator card resource monitoring method, the method comprising:

[0006] Obtaining a process identifier of a target service process in a target host and accelerator card resource usage information corresponding to the target service process, wherein the target service process is any one of a plurality of service processes running in different containers in the target host, the different containers belong to different container groups, and the target service process corresponds to a target application;

[0007] Determine, according to the process identifier, a container group identifier of a target container group to which a target container corresponding to the target service process belongs;

[0008] According to the container group identifier, acquiring metadata information corresponding to the container group identifier, wherein the metadata information includes a plurality of service tag information;

[0009] Parsing target service tag information from the metadata information, where the target service tag information is at least part of the multiple service tag information;

[0010] First accelerator card resource monitoring information corresponding to the target service process is generated according to the target service label information, the process identifier and the accelerator card resource usage information.

[0011] In a second aspect, an embodiment of the present invention provides an accelerator card resource monitoring device, the device comprising:

[0012] an acquisition module, configured to acquire a process identifier of a target service process in a target host and accelerator card resource usage information corresponding to the target service process, wherein the target service process is any one of a plurality of service processes running in different containers in the target host, the different containers belong to different container groups, and the target service process corresponds to a target application;

[0013] A determination module, configured to determine, according to the process identifier, a container group identifier of a target container group to which a target container corresponding to the target service process belongs;

[0014] A parsing module, configured to parse target service tag information from the metadata information, wherein the target service tag information is at least part of the plurality of service tag information;

[0015] A generating module is used to generate first accelerator card resource monitoring information corresponding to the target service process according to the target service label information, the process identifier and the accelerator card resource usage information.

[0016] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein an executable code is stored on the memory, and when the executable code is executed by the processor, the processor executes the acceleration card resource monitoring method as described in the first aspect.

[0017] In a fourth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the accelerator card resource monitoring method as described in the first aspect.

[0018] In a fifth aspect, an embodiment of the present invention provides a computer program product, which includes: a computer program, when the computer program is executed by a processor of an electronic device, the processor executes the acceleration card resource monitoring method as described in the first aspect.

[0019] The accelerator card resource monitoring method provided by the embodiment of the present invention realizes the acquisition of accelerator card resource usage information of finer granularity (service process granularity) by acquiring the process identifier of the target service process corresponding to the target application and the accelerator card resource usage information corresponding to the target service process. Afterwards, according to the process identifier, the container group identifier of the target container group to which the target container corresponding to the target service process belongs can be determined, and according to the container group identifier, the metadata information corresponding to the container group identifier is acquired. It can be understood that the metadata information includes multiple service tag information. In actual application, it is only necessary to parse at least part of the target service tag information in the multiple service tag information from the metadata information according to the actual service requirements, and then generate the first accelerator card resource monitoring information corresponding to the target service process according to the target service tag information, the process identifier and the accelerator card resource usage information. In this process, parsing the target service tag information in the multiple service tag information from the metadata information is equivalent to marking the target service process with the corresponding service tag, so that the resource monitoring information (i.e., the first accelerator card resource monitoring information) with more practical significance for the service dimension can be obtained in combination with the original accelerator card resource usage information acquired later, so as to meet the set accelerator card resource monitoring requirements of finer granularity.

[0020] It should be noted that, in actual application, the process identifiers of multiple service processes corresponding to the target application and the accelerator card resource usage information corresponding to the multiple service processes can be obtained at the same time, and the above-mentioned operation of generating the first accelerator card resource monitoring information can be performed on the multiple service processes respectively, and finally the first accelerator card resource monitoring information corresponding to the multiple service processes is obtained, that is, the resource usage monitoring of each service process under the target application is completed from the service process granularity. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flowchart of an accelerator card resource monitoring method provided by an embodiment of the present invention;

[0023] Figure 2 An application schematic diagram of an accelerator card resource monitoring system provided by an embodiment of the present invention;

[0024] Figure 3 Another flow chart of an accelerator card resource monitoring method provided by an embodiment of the present invention;

[0025] Figure 4 An application schematic diagram of an accelerator card resource monitoring system provided by an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of the structure of an accelerator card resource monitoring device provided by an embodiment of the present invention;

[0027] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0030] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features in the embodiments can be combined with each other. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0031] At present, when deploying applications through container clusters, in order to understand the usage of accelerator card resources of each host where the container is located, the accelerator card resources are generally monitored at the hardware granularity (i.e. host level). The monitoring granularity is coarse and cannot meet some finer granularity accelerator card resource monitoring needs. The so-called host-level monitoring only focuses on indicators such as the usage rate of the accelerator card in each host, regardless of who uses it.

[0032] In view of this, the embodiment of the present invention provides the following ideas to solve the above problems: obtain the process identifiers of multiple service processes corresponding to the target application and the accelerator card resource usage information corresponding to the multiple service processes, so as to achieve more fine-grained (service process granularity) acquisition of accelerator card resource usage information. Afterwards, the container group identifiers of the container group to which each container corresponding to the multiple service processes belongs are determined according to the process identifiers of the multiple service processes, and the metadata information corresponding to the container group identifier is obtained according to the container group identifier. In actual applications, the metadata information includes multiple service tag information. For each service process, the corresponding service tag information is parsed from the multiple service tag information of the metadata information, which is equivalent to marking the multiple service processes with their respective corresponding service tags, so that the original accelerator card resource usage information obtained can be combined with the obtained information to obtain more practical resource monitoring information for the service semantic dimension, so as to meet the set finer-grained accelerator card resource monitoring requirements. Among them, the service semantic dimension may include, for example, the user, the task to be executed, the location area, etc.

[0033] Figure 1 A flowchart of an accelerator card resource monitoring method provided by an embodiment of the present invention, the method can be executed by a custom monitoring component deployed in a container cluster, such as Figure 1 As shown, the method includes:

[0034] 101. Obtain the process identifier of the target service process in the target host and the accelerator card resource usage information corresponding to the target service process. The target service process is any one of multiple service processes running in different containers in the target host. Different containers belong to different container groups. The target service process corresponds to the target application.

[0035] 102. Determine, according to the process identifier, a container group identifier of a target container group to which a target container corresponding to the target service process belongs.

[0036] 103. Obtain metadata information corresponding to the container group identifier according to the container group identifier, where the metadata information includes multiple service tag information.

[0037] 104. Analyze target service tag information from the metadata information, where the target service tag information is at least a part of the multiple service tag information.

[0038] 105. Generate first accelerator card resource monitoring information corresponding to the target service process according to the target service label information, the process identifier and the accelerator card resource usage information.

[0039] It should be noted that the target application can be any type of application such as financial application, game application, artificial intelligence (such as large language model), etc., and there is no limitation here. The accelerator card resources can be graphics processing unit (GPU), field programmable gate array (FPGA), application specific integrated circuit (ASIC), etc., which are not listed here one by one.

[0040] When an application is deployed in a container cluster in a container manner, it is understandable that from the perspective of a physical machine, the cluster will contain multiple hosts. In order to assume the calculation and other processing during the use of the application, the host can be provided with accelerator card resources such as GPU, and one or more accelerator cards can be provided on a host. For an application (referred to as the target application), its deployment in a container cluster may occupy one or more hosts, that is, multiple container groups (pods) for deploying the target application are built in one or more hosts, and each container group contains at least one container. Different functional modules in the target application can be deployed in different container groups. The target application can be used by different users. For example, the application corresponding to a large language model can be used by different users to meet the usage needs of different users. For example, some users use it to complete question-and-answer tasks, and some users use it to complete translation tasks, and so on.

[0041] After the target application is started, multiple corresponding service processes will be started in the corresponding host as access requests are triggered by different users, or different access requests are triggered by the same user. These service processes run in the container environment of each container group corresponding to the target application.

[0042] The accelerator card resource monitoring method provided in the embodiment of the present invention can be applied to tasks that require monitoring the usage of the accelerator card resources on each host in a container cluster in the service semantic dimension, and can also be applied to tasks that require monitoring the usage of the accelerator card resources on the host deployed by a certain application in the container cluster in the service semantic dimension.

[0043] In actual applications, when it is necessary to monitor the usage of the accelerator card resources of each host on which the target application is deployed, there may be two situations: the host corresponding to the target application is known and the host corresponding to the target application is unknown. The implementation of the above step 101 is described below for these two situations respectively.

[0044] For the case where the host corresponding to the target application is known, the target host in step 101 can be the host where the target application is deployed. Since there may be more than one target host corresponding to the target application, and the processing procedures for each target host are similar, only one target host is used as an example for explanation. And for the convenience of description, it is assumed that only the target application is deployed in the target host. At this time, the service processes that use the accelerator card resources running in all containers in the target host can be directly obtained. Assuming that there are a total of N service processes, when only the target application is deployed in the container in the target host, these N service processes actually correspond to the target application. Afterwards, for any target service process among these N service processes, the process identifier of the target service process and the accelerator card resource usage information corresponding to the target service process are obtained.

[0045] It should be noted that if it is known that not only the target application but also other applications are deployed in the container of the target host, then the N service processes correspond to all the applications deployed in the target host, and at this time, the identification of the application can be regarded as one of the metadata information described in the above step 103. In other words, at this time, the N service processes obtained in step 101 cannot identify their respective corresponding applications, and at this time, for any target service process among the N service processes, the process identification of the target service process and the accelerator card resource usage information corresponding to the target service process are also obtained.

[0046] In the case where the host corresponding to the target application is unknown, the target host in step 101 is any one of the several hosts included in the container cluster. At this time, the service processes using the accelerator card resources running in all containers in the target host are directly obtained. Assuming that there are N service processes in total, these N service processes correspond to all applications deployed in the target host. Similarly, for any target service process among these N service processes, the process identifier of the target service process and the accelerator card resource usage information corresponding to the target service process are obtained. Subsequently, the application corresponding to the target service process can be determined based on the metadata information in step 103.

[0047] The above is an explanation of the need to monitor the usage of accelerator card resources of the target application. If the monitoring task is to monitor the usage of the accelerator card resources on each host in the container cluster in the service semantic dimension, then in essence, the execution of step 101 at this time is equivalent to the above-mentioned situation where the host corresponding to the target application is unknown, and will not be repeated.

[0048] In summary, for any target service process among the multiple service processes running in different containers in the target host, it is first necessary to obtain the process identifier of the target service process and the accelerator card resource usage information corresponding to the target service process. It can be understood that the accelerator card resource usage information refers to the usage information of the accelerator card resources provided by the target service process to the target host. Specifically, the process identifier of the target service process and the accelerator card resource usage information corresponding to the target service process can be obtained through the driver of each accelerator card in the target host corresponding to the target service process. It can be understood that since the target service process is located in the target host, the identifiers of the multiple service processes running in total in each container of the target host and the accelerator card resource usage information of each service process can be determined by accessing the driver of each accelerator card in the target host. The target service process is any one of these service processes. For the target service process, its accelerator card resource usage information describes which accelerator card resources the target service process uses, and the usage of these accelerator card resources (such as the utilization rate of the accelerator card, the size of the video memory used, the accelerator card temperature, etc.), which lays the foundation for the subsequent use of the accelerator card resource usage information to complete the accelerator card resource monitoring of the service semantic dimension of the target service process.

[0049] After the process identifiers of the plurality of service processes are obtained, the container identifier of the target container corresponding to each service process and the container group identifier of the target container group to which the target container belongs are determined according to the process identifier of each service process.

[0050] As an implementation method: for the target service process, the management tree path information of the resource management control group (also called Cgroups (Control Groups)) corresponding to the process ID can be obtained according to its corresponding process ID; the container ID of the target container corresponding to the target service process and the container group ID of the target container group to which the target container belongs are parsed from the management tree path information.

[0051] In specific implementation, the management tree path information of the Cgroups of the target service process can be obtained by viewing the setting configuration file. Specifically, the setting configuration file contains the management tree path information of the Cgroups to which multiple service processes belong. The "management tree" means that the management of resources can be implemented in a tree structure, and the management tree path information is actually the file path describing the name of the Cgroups. In the container cluster scenario, Cgroups can be used to implement resource management of service processes, such as limiting the accelerator card resources, memory resources, CPU cores, etc. that can be used by a certain service process. The operating system will expose the functions and configurations of Cgroups in the form of files, so each Cgroups can have a corresponding file path, and the file path will contain the service process's belonging relationship information: the service process's identifier, the identifier of the container to which it belongs, and the identifier of the container group to which it belongs. In actual application, when it is necessary to obtain the container identifier of the target container corresponding to the target service process and the container group identifier of the target container group to which the target container belongs, it is only necessary to use the command line interface provided by the container management tool such as Docker to query the above path information corresponding to the target service process to obtain it. By using the management tree path information of Cgroups, the target container corresponding to the target service process and the target container group to which the target container belongs can be quickly and accurately determined, laying the foundation for the subsequent monitoring of the accelerator resource usage information corresponding to the target service process. In addition, Cgroups can also provide resource isolation at the service process level. By parsing the management tree path information, it can ensure that the resource usage of the target service process and its target container is isolated from each other, which helps prevent resource abuse and protects other processes from being affected.

[0052] As another implementation method: all containers in all container groups can be traversed to determine whether there is a service process matching the target service process among the service processes running in each container (specifically, this can be determined by the process ID of the service process). If it is found that the service process running in the target container among all containers matches the target service process, the container ID of the target container and the container group ID of the target container group to which the target container belongs are queried using a command line tool (such as kubectl).

[0053] Afterwards, metadata information corresponding to the target container group is obtained according to the obtained container group identifier. For example, the specific process of obtaining the metadata information is as follows:

[0054] As an implementation method, the container status query interface (such as CRI (containerd runtime interface)) in the target host corresponding to the target service process is used to query the container identifier and container group identifier corresponding to the target service process to obtain metadata information corresponding to the container group identifier. It should be noted that during the creation of each container group in the target host, the metadata information corresponding to each container group will be configured to the corresponding container group. The metadata information corresponding to each container group is of the service semantic dimension, including but not limited to the following service tag information: application identifier, user identifier, task identifier, geographic area identifier, etc. These metadata information of the service semantic dimension are intended to describe the actual application scenarios of the applications running in the corresponding container group, such as who uses it, what tasks it is used to perform, and in what location area it is used.

[0055] It is understandable that in actual applications, a setting management component (such as a Kubelet component) is configured in the target host, and the setting management component can interact with the container through the container status query interface, and then obtain relevant information of the container through the container status query interface, such as the container identifier, the container group identifier to which the container belongs, the container group namespace (namespace), the metadata information of the container group, etc. In specific implementation, after querying the container identifier of the target container and the container group identifier of the corresponding target container group through the container status query interface, the metadata information corresponding to the container group identifier can be obtained from the above-mentioned relevant information according to the container group identifier. By using the container status query interface, the container identifier and the container group identifier corresponding to the above-mentioned target service process can be quickly obtained and queried, and then the metadata information corresponding to the container group identifier can be understood in real time, so that the first accelerator card resource monitoring information can be quickly obtained based on the metadata information later.

[0056] As another implementation method, the container identifier and container group identifier corresponding to the target service process can be obtained by using a command line tool to execute a set command (such as kubectldescribe pod <pod-name>), and then the metadata information corresponding to the container group identifier can be found in the setting configuration file based on the container group identifier, wherein the setting configuration file includes the container identifier, the container group identifier to which the container belongs, the metadata information of the container group, etc. After obtaining the above metadata information, the required target service label information is parsed from the metadata information to obtain the target service label information corresponding to the target service process. It can be understood that the metadata information contains multiple service label information, each service label information can be represented in the form of a key-value pair. In actual applications, the user can input the service label description information to be parsed, and the matching target service label information is obtained by matching the service label description information with each service label information in the metadata information. After that, the matching method is not specifically limited in the embodiment of the present invention, such as but not limited to the method of keyword recognition and matching, the vector conversion of the service label information in the metadata and the service label description information through a certain vector encoding model, the method of calculating the distance between vectors, and so on. Afterwards, the target service tag information, process identifier and accelerator card resource usage information corresponding to the target service process are integrated and processed to obtain the first accelerator card resource monitoring information corresponding to the target service process. For example, according to the field order of target service tag information-process identifier-accelerator card resource usage information, the first accelerator card resource monitoring information containing the contents of these fields is generated to be output to the user for viewing. In actual applications, the above-mentioned first accelerator card resource monitoring information may also include the container identifier of the above-mentioned target container and the container group identifier of the target container group, so that, for example, according to the field order of target service tag information-container group identifier-container identifier-process identifier-accelerator card resource usage information, the first accelerator card resource monitoring information containing the contents of these fields is generated.

[0057] In addition, it should be noted that in an embodiment of the present invention, the target service tag information includes at least one of the following: a user identifier and a task identifier, wherein the user identifier corresponds to a user who uses the target application through the target service process, and the task identifier corresponds to a task performed by the user using the target application.

[0058] For ease of understanding, the following Figure 2 The process of generating the first accelerator card resource monitoring information is specifically described by an example: Figure 2In the example, the target application is deployed on two hosts (assuming host A and host B). There are three container groups on host A, namely container group a1 (assuming it contains container 1, container 2, and container 3), container group a2 (assuming it contains container 1, container 2, container 3, and container 4), and container group a3 (assuming it contains container 1 and container 2). There are two container groups on host B, namely container group b1 (assuming it contains container 1, container 2, and container 3) and container group b2 (assuming it contains container 1 and container 2). Assume that the target application has six service processes, namely service process 1, service process 2, service process 3, service process 4, service process 5, and service process 6. Service process 1 runs in container 1 in container group a1, service process 2 runs in container 2 in container group a2, service process 3 runs in container 4 in container group a2, service process 4 runs in container 1 in container group a3, service process 5 runs in container 1 in container group b1, and service process 6 runs in container 2 in container group b2.

[0059] In specific implementation, by using the method described in the above embodiment, the process identifiers (PID) of service process 1, service process 2, service process 3, service process 4, service process 5, and service process 6, as well as the accelerator card resource usage information corresponding to each service process can be obtained respectively. For ease of description, the following takes service process 1 as an example for explanation:

[0060] According to the process identifier of service process 1, determine the container identifier of the target container (i.e., container 1) corresponding to service process 1 and the container group identifier of the target container group (i.e., container group a1) to which the target container belongs. According to the container group identifier, obtain the metadata information corresponding to the container group identifier (assuming that it contains 10 service label information in total), and parse the target service label information from the metadata information. Assume that the target service label information obtained is 2 of the 10 service label information, where the service label information can be expressed in the form of a key-value pair, for example, the 2 target service label information are: "User Name: abc", "Project Name: xxx". After obtaining the 2 target service label information, the first accelerator card resource monitoring information corresponding to service process 1 can be generated according to the 2 target service label information corresponding to service process 1, the process identifier and the accelerator card resource usage information.

[0061] Based on the same method as above, the first accelerator card resource monitoring information corresponding to service process 2, service process 3, service process 4, service process 5 and service process 6 is generated respectively. Finally, the accelerator card resource monitoring information of each service process of the target application can be summarized and output together, so that the user can centrally view the accelerator card resource usage of each service process in the target application in the service semantic dimension.

[0062] The accelerator card resource monitoring method provided by the embodiment of the present invention achieves the acquisition of accelerator card resource usage information at a finer granularity (service process granularity) by acquiring the process identifiers of multiple service processes in the host and the accelerator card resource usage information corresponding to the multiple service processes. Subsequently, metadata at the container group granularity can be obtained by tracing back the container and container group to which each service process belongs, and the target service label information corresponding to each service process is parsed from the metadata information, thereby achieving the labeling operation of the service semantic dimension of the target service process, so that the accelerator card resource monitoring information at the service semantic dimension of each process can be obtained in combination with the original accelerator card resource usage information at the process granularity obtained, thereby meeting the set accelerator card resource monitoring requirements at a finer granularity.

[0063] Figure 3 Another flow chart of an accelerator card resource monitoring method provided by an embodiment of the present invention is as follows: Figure 3 As shown, the method includes:

[0064] 301. Obtain the process identifier of the target service process in the target host and the accelerator card resource usage information corresponding to the target service process. The target service process is any one of the multiple service processes running in different containers in the target host. Different containers belong to different container groups. The target service process corresponds to the target application.

[0065] 302. Determine, according to the process identifier, a container identifier of a target container corresponding to the target service process.

[0066] 303. Obtain metadata information corresponding to the container group identifier according to the container group identifier, where the metadata information includes multiple service tag information.

[0067] 304. Analyze target service tag information from the metadata information, where the target service tag information is at least a part of the multiple service tag information.

[0068] 305. Obtain the accelerator card resource allocation information corresponding to the target container according to the container identifier.

[0069] 306. Generate second acceleration card resource monitoring information corresponding to the target service process according to the target service label information, the process identifier, and the acceleration card resource allocation information.

[0070] It should be noted that the specific execution process of step 301-step 304 can be found in the above embodiment and will not be repeated here. As for step 305 and step 306, it can be understood that in actual applications, as described above, a setting management component (such as a Kubelet component) is configured in the target host, and the setting management component can interact with the container through the container status query interface, and then obtain the relevant information of each container in the target host through the container status query interface, wherein, based on the above embodiment, the relevant information also includes the accelerator card resource allocation information of each container, such as the number of allocated accelerator cards, the size of available video memory, and the mode information of sharing or exclusive use of the accelerator card, etc. Specifically, different container environment variables can be queried through the query interface to obtain different allocation information through different container environment variables, such as the allocation information of the number of accelerator cards, the allocation information of the available video memory size, the usage mode information, etc., which can be obtained through different container environment variables respectively.

[0071] In specific implementation, after querying the relevant information of each container through the container status query interface, the relevant information of the target container can be obtained according to the container identifier of the target container corresponding to the target service process, as described above, including but not limited to the identifier of the container group to which it belongs, the metadata information of the container group, and the acceleration card resource allocation information of the container.

[0072] After obtaining the accelerator card resource allocation information corresponding to the target container through the above container identifier, the second accelerator card resource monitoring information corresponding to the target service process can be generated according to the target service tag information, the container identifier of the target container and the accelerator card resource allocation information parsed from the metadata information of the corresponding target container group, for example, according to the field order of target service tag information-container identifier-accelerator card resource allocation information, the second accelerator card resource monitoring information containing the contents of these fields is generated to be output to the user for viewing. In actual applications, the above second accelerator card resource monitoring information can also include the container group identifier of the above target container group, so that, for example, according to the field order of target service tag information-container group identifier-container identifier-accelerator card resource allocation information, the second accelerator card resource monitoring information containing the contents of these fields is generated.

[0073] The second accelerator card resource monitoring information allows users to intuitively view the allocation of accelerator card resources under each container of the target application under the set service semantic dimension. Combined with the first accelerator card resource monitoring information, users can see the use and allocation of accelerator card resources by the target application under the service semantic dimension, enriching the monitoring content and achieving more comprehensive monitoring.

[0074] It is understandable that in actual applications, the target application can be used by multiple users, and different users may use different tasks using the target application. For example, user A uses the target application to perform a problem finding task, while user B uses the target application to perform a translation task. In view of this, the target service tag information in the embodiment of the present invention at least includes a user identifier (such as user name: abc) and a task identifier (such as task name: xxx).

[0075] Based on the above, the embodiment of the present invention is equivalent to expanding the functions of the existing accelerator card resource monitoring tool, generating a new custom accelerator card resource monitoring component, through which finer-grained resource usage monitoring can be achieved to meet finer-grained accelerator card resource monitoring needs.

[0076] Figure 4 An application diagram of an accelerator card resource monitoring system provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, in actual applications, the resource monitoring component may include two functional components: a bottom layer collector and an upper layer marker. These two functional components may run in a container group in a container cluster, and the container group may be configured to monitor the usage / allocation of accelerator card resources for service processes in other container groups in the container cluster.

[0077] like Figure 4 As shown in , in the upper-layer labeler, the target service label of the service semantic dimension of interest is set in advance through a configuration file. The underlying collector can interact with the drivers of the accelerator cards deployed on each host in the container cluster to obtain the accelerator card resource usage information of the service process running in the container in each host through these drivers.

[0078] It is understandable that the types of acceleration cards configured in different hosts may be different, so different types of acceleration cards have different drivers.

[0079] Taking the monitoring task of accelerator card resource usage as an example, in actual applications, first, the bottom-level collector can obtain the ID of each service process in the container cluster and its accelerator card resource usage information through the above-mentioned accelerator card drivers. After that, the bottom-level collector determines the container ID and container group ID corresponding to the corresponding service process through the management tree path information of the Cgroup corresponding to each service process. After that, the bottom-level collector obtains the relevant information of each container in each host through the CRI of each host (including but not limited to the container ID, container group ID, and metadata information of the container group). Based on this, the bottom-level collector can summarize the process ID, container ID, container group ID and container group metadata corresponding to each service process, and upload this information to the upper-level labeler.

[0080] Based on the configuration of the above target service label information, the upper-level labeler parses the required target service label information from the metadata of the container group corresponding to each service process, generates and outputs the monitoring information of the accelerator card resource usage corresponding to each service process.

[0081] In this embodiment, the bottom layer collector only collects the metadata of the entire container group without parsing it, and hands it over to the upper layer labeler for parsing based on the configuration result of the target service label.

[0082] The following will describe in detail the accelerator card resource monitoring device of one or more embodiments of the present invention. Those skilled in the art will appreciate that these devices can be configured using commercially available hardware components through the steps taught in this solution.

[0083] Figure 5 A schematic diagram of the structure of an accelerator card resource monitoring device provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, the device includes: an acquisition module 11, a determination module 12, a parsing module 13 and a generation module 14.

[0084] The acquisition module 11 is used to obtain the process identifier of the target service process in the target host and the acceleration card resource usage information corresponding to the target service process, wherein the target service process is any one of the multiple service processes running in different containers in the target host, and the different containers belong to different container groups, and the target service process corresponds to the target application.

[0085] The determination module 12 is configured to determine, according to the process identifier, a container group identifier of a target container group to which a target container corresponding to the target service process belongs.

[0086] The parsing module 13 is configured to parse the metadata information to obtain target service tag information, where the target service tag information is at least part of the plurality of service tag information.

[0087] The generating module 14 is configured to generate first accelerator card resource monitoring information corresponding to the target service process according to the target service label information, the process identifier and the accelerator card resource usage information.

[0088] Among them, optionally, the generation module 14 is also used to: determine the container identifier of the target container corresponding to the target service process according to the process identifier, obtain the acceleration card resource allocation information corresponding to the target container according to the container identifier, and generate the second acceleration card resource monitoring information corresponding to the target service process according to the target service tag information, the container identifier and the acceleration card resource allocation information.

[0089] Optionally, the device further comprises: a configuration module, configured to configure the metadata information to the target container group during the creation process of the target container group.

[0090] Among them, optionally, the acquisition module 11 is specifically used to: obtain the process identifier of the target service process and the acceleration card resource usage information corresponding to the target service process through the driver of each acceleration card in the target host, and query the container group identifier through the container status query interface in the target host to obtain the metadata information corresponding to the container group identifier, and query the container identifier through the container status query interface in the target host to obtain the acceleration card resource allocation information corresponding to the container identifier.

[0091] Among them, optionally, the determination module 12 is specifically used to: obtain management tree path information of the resource management control group corresponding to the process identifier; and parse out the container identifier of the target container corresponding to the target service process and the container group identifier of the target container group to which the target container belongs from the management tree path information.

[0092] Figure 5 The device shown can execute the steps provided in the aforementioned embodiments. For detailed execution process and technical effects, please refer to the description in the aforementioned embodiments, which will not be repeated here.

[0093] In one possible design, the above Figure 5 The structure of the acceleration card resource monitoring device shown can be implemented as an electronic device. Figure 6 As shown, the electronic device may include: a processor 21, a memory 22, and a communication interface 23. The memory 22 stores executable code, and when the executable code is executed by the processor 21, the processor 21 can at least implement the accelerator card resource monitoring method provided in the above embodiment. The electronic device may be included in a container cluster, that is, exists as a host in the container cluster.

[0094] In addition, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the accelerator card resource monitoring method provided in the aforementioned embodiment.

[0095] In addition, an embodiment of the present invention provides a computer program product, which includes: a computer program, when the computer program is executed by a processor of an electronic device, the processor can at least implement the accelerator card resource monitoring method provided in the above embodiment.

[0096] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0097] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and application programs. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring accelerator card resources, characterized in that: The method comprises: Obtaining a process identifier of a target service process in a target host and accelerator card resource usage information corresponding to the target service process, wherein the target service process is any one of a plurality of service processes running in different containers in the target host, the different containers belong to different container groups, the target service process corresponds to any container group of a target application, and the target application corresponds to at least one container group; Determine, according to the process identifier, a container group identifier of a target container group to which a target container corresponding to the target service process belongs; According to the container group identifier, acquiring metadata information corresponding to the container group identifier, wherein the metadata information includes a plurality of service tag information; According to the setting result of the target service tag under the target application, parsing the target service tag information from the metadata information corresponding to the different service processes corresponding to the target application, the target service tag information is at least part of the multiple service tag information; First accelerator card resource monitoring information corresponding to the target service process is generated according to the target service tag information corresponding to the target service process, the process identifier, and the accelerator card resource usage information.

2. The method according to claim 1, characterized in that The method further comprises: Determine, according to the process identifier, a container identifier of a target container corresponding to the target service process; According to the container identifier, acquiring the accelerator card resource allocation information corresponding to the target container; Second accelerator card resource monitoring information corresponding to the target service process is generated according to the target service tag information corresponding to the target service process, the container identifier, and the accelerator card resource allocation information.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: During the process of creating the target container group, the metadata information is configured to the target container group.

4. The method according to claim 1 or 2, characterized in that: The obtaining of the process identifier of the target service process in the target host and the accelerator card resource usage information corresponding to the target service process includes: The process identifier of the target service process and the accelerator card resource usage information corresponding to the target service process are obtained through the driver programs of each accelerator card in the target host.

5. The method according to claim 1 or 2, characterized in that: The determining, according to the process identifier, a container identifier of a target container corresponding to the target service process and a container group identifier of a target container group to which the target container belongs includes: Obtaining management tree path information of the resource management control group corresponding to the process identifier; The container identifier of the target container corresponding to the target service process and the container group identifier of the target container group to which the target container belongs are parsed from the management tree path information.

6. The method according to claim 1 or 2, characterized in that: The acquiring metadata information corresponding to the container group identifier according to the container group identifier includes: The container group identifier is queried through a container status query interface in the target host to obtain metadata information corresponding to the container group identifier.

7. The method according to claim 2, characterized in that: The acquiring, according to the container identifier, the accelerator card resource allocation information corresponding to the target container includes: The container identifier is queried through the container status query interface in the target host to obtain the accelerator card resource allocation information corresponding to the container identifier.

8. An electronic device, characterized in that: include: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the acceleration card resource monitoring method as described in any one of claims 1 to 7.

9. A non-transitory machine-readable storage medium, characterized in that: The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes the acceleration card resource monitoring method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that include: A computer program, when the computer program is executed by a processor of an electronic device, causes the processor to execute the acceleration card resource monitoring method as described in any one of claims 1 to 7.

Images