Promisius-based AI platform resource monitoring method and system

Abstract

The invention provides a Promise-based AI platform resource monitoring method and system, and relates to the technical field of resource monitoring, and the Promise-based AI platform resource monitoring method comprises the following steps: creating resource objects of a container management platform, and configuring a resource label for each resource object; a pre-configured label index set is used for screening resource labels meeting the monitoring requirement, list connection association processing is carried out on the screened resource labels and resource monitoring indexes provided by the open source container monitoring tool, and a wide list is generated; aggregation processing is carried out on the resource monitoring indexes in the wide table, and the total resource use amount under the preset query dimension is calculated; and a global index mapping rule base is constructed, and standardized access of different monitoring devices is adapted by configuring an adaptation layer. According to the method, the complexity caused by different equipment monitoring plug-ins is shielded through the unified query adaptation layer, the code quantity and complexity of a query interface are reduced, and the maintainability and readability of codes are improved.

Description

Technical Field

[0001] The present invention relates to the field of resource monitoring technology, and in particular, to a Prometheus-based AI platform resource monitoring method and system. Background Art

[0002] With the popularization of cloud native concepts and multi-tenant solutions, advanced computing resources such as GPU (Graphics Processing Unit) are relatively scarce. The refined management and monitoring of computing resources by AI (Artificial Intelligence) platforms are becoming increasingly important. More and more AI platforms use resource pooling management to achieve refined management of computing resources at the tenant, organization, project, and user levels. However, the monitoring of the actual utilization of these allocated computing resources is still relatively lacking. It generally stays at the default monitoring granularity based on open source resource monitoring tools, monitoring the resource utilization of tasks and machines, while the monitoring of tenants, organizations, projects, and users is generally lacking, and it is impossible to effectively locate the phenomenon of resource waste, resulting in the inability to improve the overall resource utilization. In addition, in addition to the conventional CPU (Central Processing Unit), memory, etc., the AI ​​platform also needs to monitor the GPU for computing power, and there are differences in the naming of GPUs of different brands for the same label, which also increases the complexity of resource monitoring.

[0003] The resource indicators of Kubemetes containers are collected through components such as cadvisor. Cadvisor provides indicators of container CPU, memory, network, disk, and file system resource usage. GPU resource usage indicators are often collected by components provided by various hardware vendors. Each vendor's components are different, and the names of common indicators are also different. Both cadvisor and the GPU indicator collection tools provided by various vendors do not record user-defined labels in the indicator set due to concerns that too many labels will lead to an explosive growth in indicator storage. Only a few fixed labels such as the container group name, namespace, and container name are retained. This leads to two problems:

[0004] (1) We cannot use labels to identify the tenant, organization, project, or user to which a container belongs, which makes it difficult for us to calculate resource usage statistics for tenants, organizations, projects, and users.

[0005] (2) Since the GPU indicator collection component is provided by hardware manufacturers, the names of indicators of different hardware manufacturers vary greatly. It is necessary to modify the indicator query statement according to different GPU manufacturers. In the context of information innovation, there are currently many domestic GPU manufacturers, and the manufacturers actually used by customers are also different. Even the same customer may use GPUs from multiple manufacturers. This undoubtedly increases the workload of adaptation.

[0006] Regarding problem (1), there are currently few AI platforms that can simultaneously implement resource monitoring solutions for tenants, organizations, projects, and users. Similar solutions include:

[0007] 1. Use the container group name, add the required labels to the name according to certain rules, and then use regular expressions to perform pattern matching to extract the required label values.

[0008] 2. Use namespace tags. Namespace is also one of the tags for classifying and managing resources. Add the tags we need to the namespace according to certain rules, and then use regular expressions and other methods to perform pattern matching to extract the required tag values.

[0009] 3. The tenant, organization, project, and user affiliation of the container group is recorded in a database table within the AI ​​platform. By querying the table, all container groups of the relevant tenants, organizations, projects, and users are obtained. The resource usage of the container groups is then queried separately and accumulated and summarized.

[0010] The solutions described above can solve certain problems at any time, but there are problems such as many restrictions or poor performance. Method 1 is similar to method 2. Since the label of Prometheus monitoring indicators is limited to 64 characters, if it exceeds the limit, some labels will be truncated, so a single label cannot contain too much label information. Whether method 2 can be implemented also depends on whether the AI ​​platform can obtain the permission to create a custom namespace. Generally, enterprise-level cloud native platforms have clear plans for the naming and number of namespaces, and AI platforms may not be able to obtain the permission to create custom namespaces.

[0011] The main problem with method 3 is that it requires multiple calls to the query interface, which results in very slow query efficiency and misaligned time dimensions. If the number of container groups is large, the waiting time will be very long.

[0012] For problem (2), the common solutions are:

[0013] 1. Use the code branch management tool to manage different code branches and adapt them to different GPU hardware devices.

[0014] 2. Use logical branches such as IF in the code, and follow different code logic branches according to the actual hardware device of the GPU.

[0015] Method 1: Different code branches can be merged through code management tools such as git. Although this can reduce the workload of maintaining multiple versions of code to a certain extent, with the increase in the number of adapter manufacturers, the workload of code version maintenance is still relatively large, and it does not support the mixed use of GPUs from multiple manufacturers very well.

[0016] Method 2 is compatible with the mixed use of GPUs from multiple manufacturers, but it has high code redundancy and low code maintainability. Each query interface must have a large number of logical branches, which can easily lead to under-modification, missed modifications, and incorrect modifications.

[0017] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0018] In view of this, the present invention provides an AI platform resource monitoring method and system based on Prometheus to solve the above-mentioned problems.

[0019] In order to solve the above problems, the specific technical solutions adopted by the present invention are as follows:

[0020] According to one aspect of the present invention, a method for monitoring AI platform resources based on Prometheus is provided, and the method for monitoring AI platform resources based on Prometheus comprises the following steps:

[0021] S1. Based on the cloud-native AI platform, create resource objects of the container management platform and configure resource tags for each resource object;

[0022] S2. Based on the regular expression matching algorithm, use the pre-configured label indicator set to filter resource labels that meet the monitoring requirements, associate the filtered resource labels with the resource monitoring indicators provided by the open source container monitoring tool, and generate a wide table;

[0023] S3: Aggregate the resource monitoring indicators in the wide table and calculate the total resource usage under the preset query dimension;

[0024] S4. Build a global indicator mapping rule base and adapt the standardized access of different monitoring devices by configuring the adaptation layer.

[0025] Preferably, the step of creating resource objects of the container management platform based on the cloud native AI platform and configuring resource tags for each resource object includes the following steps:

[0026] S11. During the execution of tasks on the cloud native AI platform, resource objects for managing and deploying containers are created according to business needs;

[0027] S12. Define and assign a name and namespace to each created resource object;

[0028] S13. Use the labeling function of the container management platform to add resource labels to each object;

[0029] S14. Use the command line tool of the container management platform to query the resource tag of the resource object and verify whether the resource tag is correctly added to the resource object. If it is not correctly added, use the update command to reconfigure the correct tag for the resource object.

[0030] Preferably, the method of using a pre-configured label indicator set to screen resource labels that meet monitoring requirements based on a regular expression matching algorithm, and performing a joint table association process on the screened resource labels and the resource monitoring indicators provided by the open source container monitoring tool to generate a wide table includes the following steps:

[0031] S21. Based on the resource tags configured for each resource object, the Prometheus monitoring platform is used to collect the indicator information of the resource tags and build a tag indicator set.

[0032] S22. According to monitoring requirements, extract the regular expression and use it as a filtering condition, and obtain the prefix factor and the necessary factor set by constructing an abstract syntax tree corresponding to the regular expression;

[0033] S23, using the extracted prefix factors and necessary factors, matching is performed on the generalized suffix tree of the constructed tag index set to screen candidate resource tags;

[0034] S24. Associate the index of the filtered resource tag with the resource monitoring index provided by the open source container monitoring tool to generate a wide table including the resource tag index and the resource monitoring index.

[0035] Preferably, extracting a regular expression according to monitoring requirements and using it as a filtering condition, and obtaining a prefix factor and a necessary factor set by constructing an abstract syntax tree corresponding to the regular expression comprises the following steps:

[0036] S221, determining the resource tags to be monitored according to the monitoring requirements, establishing a regular expression and parsing the regular expression using grammar rules, and constructing a corresponding abstract syntax tree, where the nodes of the abstract syntax tree represent components of the regular expression;

[0037] S222, extract all leaf nodes on the path from the root node to the left subtree in the abstract syntax tree and construct a prefix factor set;

[0038] S223. Traverse all paths of the abstract syntax tree, extract the nodes that must appear and construct a set of necessary factors.

[0039] Preferably, the steps of determining the resource tags to be monitored according to the monitoring requirements, establishing a regular expression and parsing the regular expression using grammar rules, and constructing a corresponding abstract syntax tree, wherein the nodes of the abstract syntax tree represent components of the regular expression, include the following steps:

[0040] S2221. Based on the determined resource tag, convert the resource tag into a regular expression using a context-free grammar;

[0041] S2222, using a lexical analyzer to decompose the regular expression into a string to obtain a token sequence;

[0042] S2223. Based on the syntax analyzer, convert the token sequence into an abstract syntax tree according to the grammar rules.

[0043] Preferably, the method of using the extracted prefix factors and necessary factors to match on the generalized suffix tree of the constructed tag index set to screen candidate resource tags includes the following steps:

[0044] S231, traversing the label indicator set, and adding the resource label in the label indicator set as a suffix to the generalized suffix tree;

[0045] S232, for each prefix factor and each necessary factor, use depth-first search to find all paths matching the prefix factor and the necessary factor in the generalized suffix tree, record the leaf nodes of all paths and establish a prefix candidate interval and a necessary candidate interval respectively;

[0046] S233: Cross-compare the prefix candidate interval with the necessary candidate interval, filter out leaf nodes that exist in the intersection area of ​​the prefix candidate interval and the necessary candidate interval, and use the filtered leaf nodes as the final candidate resource labels.

[0047] Preferably, the step of associating the index of the filtered resource tag with the resource monitoring index provided by the open source container monitoring tool to generate a wide table including the resource tag index and the resource monitoring index comprises the following steps:

[0048] S241. Obtain resource monitoring indicators from open source container monitoring tools;

[0049] S242, based on the joint table association rule, the resource monitoring indicator is associated with the indicator of the filtered resource tag according to the container group name and the namespace to obtain an association result;

[0050] S243. According to the association result, a wide table including resource monitoring indicators and resource label indicators is obtained.

[0051] Preferably, the aggregating the resource monitoring indicators in the wide table and calculating the total resource usage under the preset query dimension includes the following steps:

[0052] S31. Determine the resource monitoring indicator to be queried according to the preset query dimension;

[0053] S32. Create an aggregate query statement using SQL language according to the query dimension and the resource monitoring indicator to be queried;

[0054] S33. Execute the aggregate query statement, extract the indicator data from the wide table and perform aggregation processing to obtain the total resource usage under each preset dimension.

[0055] Preferably, the construction of a global indicator mapping rule base and adapting standardized access of different monitoring devices by configuring an adaptation layer comprises the following steps:

[0056] S41. Use resource monitoring indicators provided by open source container monitoring tools as standard indicator names;

[0057] S42, collecting and analyzing the original monitoring indicator names provided by various monitoring devices, establishing a mapping relationship between the original monitoring indicator name of each monitoring device and the standard indicator name, and creating a global indicator mapping rule base;

[0058] S43, establishing an adaptation layer for receiving a monitoring query request containing a standard indicator name input by a user according to a global indicator mapping rule base;

[0059] S44, the adaptation layer converts the indicator name through the global indicator mapping rule base according to the standard indicator name in the monitoring query request input by the user;

[0060] S45. Obtain resource monitoring indicators based on the converted indicator names.

[0061] According to another aspect of the present invention, there is provided an AI platform resource monitoring system based on Prometheus, the AI ​​platform resource monitoring system based on Prometheus comprising: a resource label configuration module, a wide table generation module, a resource usage total amount calculation module and a standardized access module;

[0062] The resource tag configuration module is used to create resource objects of the container management platform based on the cloud native AI platform and configure resource tags for each resource object;

[0063] The wide table generation module is used to use a pre-configured label indicator set to filter resource labels that meet monitoring requirements based on a regular expression matching algorithm, perform joint table association processing on the filtered resource labels and resource monitoring indicators provided by the open source container monitoring tool, and generate a wide table;

[0064] The resource usage total amount calculation module is used to aggregate the resource monitoring indicators in the wide table and calculate the total resource usage under the preset query dimension;

[0065] The standardized access module is used to construct a global indicator mapping rule base and adapt the standardized access of different monitoring devices by configuring the adaptation layer.

[0066] Preferably, the creating resource objects of the container management platform based on the cloud native AI platform and configuring resource tags for each resource object includes:

[0067] When executing tasks on the cloud-native AI platform, resource objects for managing and deploying containers are created according to business needs;

[0068] Define and assign a name and namespace to each resource object created;

[0069] Use the labeling function of the container management platform to add resource labels to each object;

[0070] Use the command line tool of the container management platform to query the resource tags of the resource object and verify whether the resource tags are correctly added to the resource object. If they are not correctly added, use the update command to reconfigure the correct tags for the resource object.

[0071] Preferably, the regular expression matching algorithm is used to filter resource labels that meet monitoring requirements using a pre-configured label indicator set, and the filtered resource labels are associated with resource monitoring indicators provided by an open source container monitoring tool to generate a wide table, including:

[0072] Based on the resource tags configured for each resource object, the Prometheus monitoring platform is used to collect the indicator information of the resource tags and build a tag indicator set;

[0073] According to monitoring requirements, extract the regular expression and use it as the filtering condition. By building the abstract syntax tree corresponding to the regular expression, obtain the prefix factor and necessary factor set;

[0074] Using the extracted prefix factors and necessary factors, match them on the generalized suffix tree of the constructed tag index set to screen candidate resource tags;

[0075] The indicators of the filtered resource tags are associated with the resource monitoring indicators provided by the open source container monitoring tool to generate a wide table containing resource tag indicators and resource monitoring indicators.

[0076] Preferably, the aggregating the resource monitoring indicators in the wide table and calculating the total resource usage under the preset query dimension includes:

[0077] Determine the resource monitoring indicators that need to be queried based on the preset query dimensions;

[0078] Use SQL language to create aggregate query statements based on query dimensions and resource monitoring indicators that need to be queried;

[0079] Execute the aggregation query statement, extract the indicator data from the wide table and perform aggregation processing to obtain the total resource usage under each preset dimension.

[0080] Preferably, the construction of a global indicator mapping rule base and adapting standardized access of different monitoring devices by configuring an adaptation layer includes:

[0081] Use resource monitoring indicators provided by open source container monitoring tools as standard indicator names;

[0082] Collect and analyze the original monitoring indicator names provided by various monitoring devices, establish a mapping relationship between the original monitoring indicator name of each monitoring device and the standard indicator name, and create a global indicator mapping rule base;

[0083] According to the global indicator mapping rule base, an adaptation layer is established for receiving a monitoring query request containing a standard indicator name input by a user;

[0084] The adaptation layer converts the indicator name according to the standard indicator name in the monitoring query request input by the user through the global indicator mapping rule base;

[0085] Get resource monitoring indicators based on the converted indicator names.

[0086] The beneficial effects of the present invention are:

[0087] 1. The present invention can use PormQL very conveniently to realize resource monitoring at the tenant, organization, project, and user levels by means of aggregate query with kube_pod_labels joint table, thereby improving development and query efficiency. The realization of resource monitoring at the tenant, organization, project, and user levels enables the AI ​​platform to better control resource utilization, better perform resource recovery, resource planning, cost division, and other needs. Through a unified query adaptation layer, the complexity brought by different device monitoring plug-ins is shielded, the amount and complexity of the query interface code are reduced, and the maintainability and readability of the code are improved.

[0088] 2. The present invention can define standard indicator names for GPU devices from different manufacturers, and manage the mapping relationship between standard indicator names and indicator names of different device GPUs through a rule library. Developers use standard indicator names to develop codes, and during actual product deployment and operation, indicator name mapping is automatically performed according to the actual GPU manufacturer. When new devices are added, only new mapping rules need to be added to the rule library, without the need to add or modify code.

[0089] 3. The present invention can flexibly define complex matching rules through regular expression matching algorithm to ensure that only resource tags that meet specific conditions are screened out. The prefix factor and necessary factor are used to optimize the matching process, which can greatly reduce unnecessary searches and comparisons, thereby improving matching efficiency. By constructing data structures such as abstract syntax trees and generalized suffix trees, complex matching logic and label structures are presented in a visual and structured manner, making management and maintenance easier. By associating the screened resource tags with resource monitoring indicators in a joint table, a wide table containing rich information can be generated. This wide table not only includes resource monitoring data, but also label information associated with these resources, making data integration and comprehensive analysis possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0091] Figure 1 is a flowchart of a Prometheus-based AI platform resource monitoring method according to an embodiment of the present invention;

[0092] Figure 2 It is a schematic diagram of a process of performing joint table association on indicators in a method for monitoring AI platform resources based on Prometheus according to an embodiment of the present invention;

[0093] Figure 3 It is a schematic diagram of resource usage details in a Prometheus-based AI platform resource monitoring method according to an embodiment of the present invention;

[0094] Figure 4 It is a schematic diagram of a standardized monitoring process in a Prometheus-based AI platform resource monitoring method according to an embodiment of the present invention;

[0095] Figure 5 It is a flow chart of an AI platform resource monitoring system based on Prometheus according to an embodiment of the present invention.

[0096] In the figure:

[0097] 1. Resource tag configuration module; 2. Wide table generation module; 3. Total resource usage calculation module; 4. Standardized access module. DETAILED DESCRIPTION

[0098] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0099] According to an embodiment of the present invention, a method and system for monitoring AI platform resources based on Prometheus are provided.

[0100] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, according to one embodiment of the present invention, a method for monitoring AI platform resources based on Prometheus is provided, and the method for monitoring AI platform resources based on Prometheus includes the following steps:

[0101] S1. Based on the cloud-native AI platform, create resource objects of the container management platform and configure resource tags for each resource object;

[0102] As a preferred implementation, the creation of resource objects of the container management platform based on the cloud native AI platform and configuring resource tags for each resource object include the following steps:

[0103] S11. During the execution of tasks on the cloud native AI platform, resource objects for managing and deploying containers are created according to business needs;

[0104] It should be noted that when executing tasks on the cloud native AI platform, resource objects are created according to specific business needs. These objects may be Pods, Deployments, Services, etc., which are used to host and manage containerized applications.

[0105] S12. Define and assign a name and namespace to each created resource object;

[0106] It should be noted that each resource object is assigned a unique name and namespace to uniquely identify and isolate resources in the cluster. Namespaces help to logically group resources and improve the manageability of the cluster.

[0107] S13. Use the labeling function of the container management platform to add resource labels to each object;

[0108] It should be noted that the label function of the container management platform can be used to add custom labels to resource objects. These labels can be business-related metadata, such as tenant name, organization name, project name, and user ID, to facilitate subsequent resource management and monitoring.

[0109] S14. Use the command line tool of the container management platform to query the resource tag of the resource object and verify whether the resource tag is correctly added to the resource object. If it is not correctly added, use the update command to reconfigure the correct tag for the resource object.

[0110] Specifically, when the cloud-native AI platform performs tasks such as training and prediction, it adds custom resource labels such as tenants, organizations, projects, and users to the created pods and deployments.

[0111] YAML files are used in Kubernetes to describe the configuration and status of Kubemetes resources. YAML files define the objects of Kubemetes resources in an easy-to-read format. The definitions of labels such as tenants, organizations, projects, and users are also implemented through YAML.

[0112] For example, the following YAML creates a container group named example-pod and adds four labels: tenant, organization, project, and user:

[0113]

[0114] S2. Based on the regular expression matching algorithm, use the pre-configured label indicator set (kube_pod_labels) to filter resource labels that meet the monitoring requirements, associate the filtered resource labels with the resource monitoring indicators provided by the open source container monitoring tool, and generate a wide table;

[0115] It should be noted that when viewing resource monitoring in the tenant, organization, project, and user dimensions, use the tenant, organization, project, user and other custom labels in the kube_pod_labels indicator set to associate with the CPU, memory, network, disk, and file system indicators of cadvisor. By using the tenant, organization, project, user and other custom labels, you can filter out the resource usage of a specific tenant, organization, project, or user.

[0116] As a preferred implementation, the method based on the regular expression matching algorithm uses a pre-configured label indicator set to filter resource labels that meet the monitoring requirements, associates the filtered resource labels with the resource monitoring indicators provided by the open source container monitoring tool, and generates a wide table, including the following steps:

[0117] S21. Based on the resource tags configured for each resource object, the Prometheus monitoring platform is used to collect the indicator information of the resource tags and build a tag indicator set.

[0118] Specifically, on the cloud-native AI platform, custom resource labels are configured for resource objects (such as Pods, Deployments, etc.) on the container management platform. These labels usually include dimensions such as tenants, organizations, projects, and users to facilitate subsequent resource management and monitoring. These labels are defined through YAML files or other configuration methods when creating resource objects and stored in Kubernetes metadata.

[0119] Prometheus is an open source monitoring and alerting system that is widely used for monitoring large-scale distributed systems. In order to collect indicator information of resource tags, it is necessary to integrate the Prometheus monitoring platform with the Kubernetes cluster, which involves the following steps:

[0120] Deploy Prometheus Server: Deploy the Prometheus Server component in the Kubernetes cluster to collect, store, and query monitoring data.

[0121] Configure Exporter: Deploy corresponding Exporter (such as NodeExporter, Kube-State-Metrics, etc.) for different types of resources to expose resource monitoring indicators. These Exporters will regularly obtain monitoring data from Kubernetes API or other data sources and expose it to Prometheus Server through HTTP interface.

[0122] Configure Prometheus scraping rules: In the configuration file of Prometheus Server, define the scraping rules (scrape configs), specify which Exporters Prometheus Server should scrape data from, as well as parameters such as the frequency of scraping.

[0123] Once the Prometheus monitoring platform is integrated with the Kubernetes cluster, the Prometheus Server will periodically fetch monitoring data from the corresponding Exporter according to the configured fetching rules. These monitoring data contain the indicator information of resource objects and the resource tags associated with them.

[0124] S22. According to monitoring requirements, extract the regular expression and use it as a filtering condition, and obtain the prefix factor and the necessary factor set by constructing an abstract syntax tree corresponding to the regular expression;

[0125] As a preferred implementation, the method of extracting a regular expression according to monitoring requirements and using it as a filtering condition, and obtaining a prefix factor and a necessary factor set by constructing an abstract syntax tree corresponding to the regular expression includes the following steps:

[0126] S221, determining the resource tags to be monitored according to the monitoring requirements, establishing a regular expression and parsing the regular expression using grammar rules, and constructing a corresponding abstract syntax tree, where the nodes of the abstract syntax tree represent components of the regular expression;

[0127] As a preferred implementation, the steps of determining the resource tags to be monitored according to the monitoring requirements, establishing a regular expression and parsing the regular expression using grammar rules, and constructing a corresponding abstract syntax tree, wherein the nodes of the abstract syntax tree represent the components of the regular expression, include the following steps:

[0128] S2221. Based on the determined resource tag, convert the resource tag into a regular expression using a context-free grammar;

[0129] Specifically, the following are examples of steps to design a regular expression based on resource tag naming rules:

[0130] (1) Analyze the naming rules of resource tags:

[0131] Determines whether resource tags contain a specific prefix, suffix, or separator.

[0132] Understand the limitations of the character set in the label (for example, whether it contains only alphanumeric characters, or also underscores, hyphens, etc.).

[0133] Determines the maximum length of the label.

[0134] (2) Design a regular expression pattern:

[0135] Based on the naming rules obtained from the analysis, start designing regular expressions.

[0136] These rules are expressed using regular expression special characters and constructs. For example:

[0137] Use ^ and $ to anchor the start and end of a string.

[0138] Use [a-zA-Z0-9_] to match letters, numbers, and underscores (if your tags contain these characters).

[0139] Use + or * to indicate repetition of a character or character class.

[0140] Use ( and ) to group expressions.

[0141] Use | to indicate selection (if you need to match multiple different patterns).

[0142] (3) Test regular expression:

[0143] Test your regular expression with a set of known resource tags to ensure that it correctly matches all expected tags.

[0144] (4) Adjust the regular expression until it meets all requirements.

[0145] S2222, using a lexical analyzer to decompose the regular expression into a string to obtain a token sequence;

[0146] The application understands that the task of the lexer is to decompose the string input of the regular expression into a series of tokens. These tokens represent different components of the regular expression, such as common characters, metacharacters, quantifiers, etc.

[0147] When processing a regular expression, the lexical analyzer breaks the input string into a series of tokens according to the grammatical rules of the regular expression. These tokens include:

[0148] Ordinary characters: These are the most basic tokens, which directly represent single characters in the regular expression. For example, in the regular expression abc, a, b, and c are all ordinary character tokens.

[0149] Metacharacters: These are characters with special meanings that change the behavior of regular expressions. For example, . represents any single character, * represents that the preceding element can appear zero or more times, + represents that the preceding element can appear once or more times, ? represents that the preceding element can appear zero or once, | represents selection (i.e., matching the expression on the left or the expression on the right), and so on. In the regular expression a.*b, . and * are both metacharacter tokens.

[0150] Quantifiers: Although quantifiers (such as *, +, ?) are themselves metacharacters, they are listed separately here to emphasize their special role in regular expressions: controlling the number of repetitions of the preceding element.

[0151] Parentheses: ( and ) are used for grouping, they play a very important role in regular expressions as they change the precedence of operations and allow references to sub-expressions (in backreferences).

[0152] Square brackets: [ and ] are used to define character classes, allowing you to match any single character within the square brackets. For example, [abc] can match any one of a, b, or c.

[0153] Escape character: \ is used to escape special characters in regular expressions, making them lose their special meaning. For example, \. matches a literal dot character (.), not any single character.

[0154] The lexer identifies these different components when breaking down the regular expression string and outputs them as tokens. These tokens are then used by the parser to build an abstract syntax tree (AST) or perform other operations.

[0155] S2223. Based on the syntax analyzer, convert the token sequence into an abstract syntax tree according to the grammar rules.

[0156] The application understands that the task of the parser is to combine the token sequence generated by the lexical analyzer into an abstract syntax tree (AST). AST is a tree structure that represents the grammatical structure of the regular expression, where each node represents a component of the regular expression.

[0157] Convert the regular expression to AST, then the nodes in AST will include:

[0158] Root node: represents the entire regular expression.

[0159] Character nodes: represent common characters in regular expressions.

[0160] Metacharacter nodes: represent special characters in regular expressions, such as ., *, +, ?, etc. These nodes require additional information to indicate their scope (for example, which character or subexpression is modified by the quantifier).

[0161] Grouping nodes: represent sub-expressions surrounded by parentheses. These nodes usually contain one or more child nodes, representing the regular expression structure within the grouping.

[0162] Selection nodes: In regular expression engines that support selection operators (such as |), selection nodes are used to represent multiple possible matching paths.

[0163] Assertion nodes: In regular expression engines that support lookaheads and lookbehinds, these nodes are used to represent patterns that match conditions but do not consume characters.

[0164] S222, extract all leaf nodes on the path from the root node to the left subtree in the abstract syntax tree and construct a prefix factor set;

[0165] It should be noted that the prefix factor usually refers to the part at the beginning of the regular expression, which does not contain quantifiers (such as *, +,?) or selection operators (|), because the prefix factor is a definite string that will not repeat or branch. In order to extract the prefix factor, start from the "root" node and traverse downward along the path that does not contain quantifiers or selection operators until such an operator is encountered or a leaf node is reached.

[0166] S223. Traverse all paths of the abstract syntax tree, extract the nodes that must appear and construct a set of necessary factors.

[0167] It should be noted that necessary factors refer to nodes that must appear on all matching paths.

[0168] Traverse all paths of the abstract syntax tree, find nodes that exist on each path, and add them to the necessary factor set. For example, for the regular expression a(b|c)d, the necessary factor set is {'a', 'd'}, because no matter whether b or c is selected, a and d must appear.

[0169] S23, using the extracted prefix factors and necessary factors, matching is performed on the generalized suffix tree of the constructed tag index set to screen candidate resource tags;

[0170] As a preferred implementation, the method of using the extracted prefix factors and necessary factors to match on the generalized suffix tree of the constructed tag index set to screen candidate resource tags includes the following steps:

[0171] S231, traversing the label indicator set, and adding the resource label in the label indicator set as a suffix to the generalized suffix tree;

[0172] It should be noted that this step is the basis for constructing a generalized suffix tree (GST). First, there needs to be a set containing multiple resource tags, and this set is called a "tag indicator set". Then, each resource tag in this set is traversed and added to the GST one by one.

[0173] S232, for each prefix factor and each necessary factor, use depth-first search to find all paths matching the prefix factor and the necessary factor in the generalized suffix tree, record the leaf nodes of all paths and establish a prefix candidate interval and a necessary candidate interval respectively;

[0174] It should be noted that for each prefix factor and necessary factor, a depth-first search (DFS) algorithm is executed to find all paths matching the factor in the GST. Since each path in the GST represents a suffix of a label, the end point of the found path (i.e., the leaf node) corresponds to a resource label containing the factor.

[0175] By recording all such leaf nodes, we can build prefix candidate intervals and necessary candidate intervals based on them. The prefix candidate interval contains all resource labels that match the prefix factor, while the necessary candidate interval contains all resource labels that match the necessary factor.

[0176] S233: Cross-compare the prefix candidate interval with the necessary candidate interval, filter out leaf nodes that exist in the intersection area of ​​the prefix candidate interval and the necessary candidate interval, and use the filtered leaf nodes as the final candidate resource labels.

[0177] It should be noted that the purpose of cross-matching is to find resource labels that meet both the prefix factor and the necessary factor conditions. These labels start with a specific prefix and contain necessary substrings, so they are potential objects that meet specific monitoring requirements. By comparing two candidate intervals, the program can find their intersection area, and the leaf nodes in this area are the final candidate resource labels. These labels will be used for subsequent monitoring and analysis.

[0178] S24. Associate the index of the filtered resource tag with the resource monitoring index provided by the open source container monitoring tool to generate a wide table including the resource tag index and the resource monitoring index.

[0179] As a preferred implementation, the method of associating the index of the filtered resource tag with the resource monitoring index provided by the open source container monitoring tool to generate a wide table including the resource tag index and the resource monitoring index includes the following steps:

[0180] S241. Obtain resource monitoring indicators from open source container monitoring tools;

[0181] It should be noted that resource monitoring indicators are obtained from open source container monitoring tools (such as cadvisor). These indicators usually include CPU usage, memory usage, disk I / O, etc.

[0182] S242, based on the joint table association rule, the resource monitoring indicator is associated with the indicator of the filtered resource tag according to the container group name and the namespace to obtain an association result;

[0183] S243. According to the association result, a wide table including resource monitoring indicators and resource label indicators is obtained.

[0184] Specifically, if Figure 2-3 As shown, the filtered resource label indicator set (such as kube_pod_labels filtered by tenant, organization, project, and user) is associated with the resource monitoring indicator set.

[0185] The association is performed based on container group name and namespace, ensuring that only (container group name, namespace) combinations that exist in both datasets are included in the final wide table.

[0186] After association, the wide table will contain custom tags such as container group name, namespace, tenant, organization, project, user, and corresponding resource monitoring indicators.

[0187] Among them, the joint table association rule is: the combination of (container group name, namespace) that must exist in both the kube_pod_labels indicator set and the resource monitoring indicator set will appear in the final wide table.

[0188] The fields included in the wide table are: container group name + namespace + custom labels such as tenants, organizations, projects, and users in kube_pod_labels + all labels and indicators in the resource monitoring indicator set (all labels and indicators except container group name and namespace).

[0189] For example, if you want to view the memory usage details of all container groups of project "projectA", the corresponding steps are as follows:

[0190] (1) Get the kube_pod_labels indicator set project, which is all the container groups of projectA, and get the indicator set A1;

[0191] (2) The indicator set A1 is linked with the indicator set container_memory_usage_bytes under cadvisor by container group name and namespace to obtain indicator set B. Indicator set B is the memory usage details of all container groups in project A.

[0192] The Prometheus reference code for steps (1) and (2) is:

[0193]

[0194] S3: Aggregate the resource monitoring indicators in the wide table and calculate the total resource usage under the preset query dimension;

[0195] As a preferred implementation, the aggregating of resource monitoring indicators in the wide table and calculating the total resource usage under a preset query dimension includes the following steps:

[0196] S31. Determine the resource monitoring indicator to be queried according to the preset query dimension;

[0197] It should be noted that the resource monitoring indicators to be queried are determined. For example, memory usage, CPU usage, etc. need to be queried.

[0198] Determine the preset query dimensions. For example, summarize by project, tenant, organization, user, etc.

[0199] S32. Create an aggregate query statement using SQL language according to the query dimension and the resource monitoring indicator to be queried;

[0200] Based on the determined resource monitoring indicators and query dimensions, you can create an aggregate query statement by following the steps below:

[0201] a. Select table and field

[0202] Select the table that contains resource monitoring metrics (for example, container_memory_usage_bytes).

[0203] Select a table that contains label information (e.g. kube_pod_labels), which contains the container group name, namespace, and custom labels (such as tenant, organization, project, user, etc.).

[0204] b. Perform joint table operations

[0205] Use the join operation to join the resource monitoring indicator table with the label information table. The conditions of the join are usually the container group name and namespace.

[0206] c. Add filter conditions

[0207] Add filter conditions according to specific query requirements. For example, if you are only interested in a specific project (such as projectA), you can add the corresponding filter conditions.

[0208] d. Group by query dimension

[0209] Use the GROUP BY clause to group by the preset query dimensions. For example, if you want to summarize by project, tenant, and organization, list these fields in the GROUP BY clause.

[0210] e. Applying aggregate functions

[0211] Apply aggregate functions (such as SUM, AVG, etc.) in the SELECT clause to calculate the total or average resource usage under each group. For example, use the SUM function to calculate the total memory usage and the AVG function to calculate the average CPU usage.

[0212] f. Execute query

[0213] Finally, execute the constructed aggregation query statement to extract and process data from the wide table to obtain the total resource usage under each preset dimension.

[0214] S33. Execute the aggregate query statement, extract the indicator data from the wide table and perform aggregation processing to obtain the total resource usage under each preset dimension.

[0215] It should be noted that the previously constructed SQL aggregation query statement is run. This can be done through database management tools (such as pgAdmin, MySQL Workbench), command line tools (such as psql, mysql), or database libraries in programming languages ​​(such as Python's psycopg2, sqlalchemy, Java's JDBC, etc.). After executing the query, the database will return a result set. This result set contains the total resource usage grouped by the preset dimensions.

[0216] For each preset dimension, the query results already include the total resource usage. If further aggregation processing is required (for example, calculating the sum of multiple items), additional calculations can be performed on the result set.

[0217] Calculate the average: If you need to calculate the average resource usage under a certain dimension, you can calculate the average of the total amount in the result set.

[0218] Other statistics: Based on specific needs, other statistical indicators can also be calculated, such as maximum value, minimum value, standard deviation, etc.

[0219] For example, after viewing the memory usage details of all container groups of project "projectA" as described above, the indicator set B is summarized according to the project field to obtain the total memory usage of projectA.

[0220] The total Prometheus reference code is:

[0221]

[0222] S4. Build a global indicator mapping rule base and adapt the standardized access of different monitoring devices by configuring the adaptation layer.

[0223] It should be noted that the label names with the same meaning may be inconsistent between different GPU device manufacturers or different monitoring components. By configuring the label name mapping rules of different node devices in the rule base, an adaptation layer is designed under the AI ​​platform monitoring query interface. Developers use unified indicator names to develop query interfaces, and the adaptation layer automatically maps them to actual label names based on the brand or model of the node or device.

[0224] As a preferred implementation, the construction of a global indicator mapping rule base and adapting standardized access of different monitoring devices by configuring an adaptation layer includes the following steps:

[0225] S41. Use resource monitoring indicators provided by open source container monitoring tools as standard indicator names;

[0226] It should be noted that a set of standard indicator names will be defined, and these names will be used to uniformly represent the monitoring indicators of various monitoring devices.

[0227] For example, for GPU monitoring, the following standard metric names can be defined:

[0228] gpu_utilization: GPU utilization;

[0229] gpu_memory_used: GPU video memory usage;

[0230] gpu_temperature: GPU temperature;

[0231] S42, collecting and analyzing the original monitoring indicator names provided by various monitoring devices, establishing a mapping relationship between the original monitoring indicator name of each monitoring device and the standard indicator name, and creating a global indicator mapping rule base;

[0232] S43, establishing an adaptation layer for receiving a monitoring query request containing a standard indicator name input by a user according to a global indicator mapping rule base;

[0233] S44, the adaptation layer converts the indicator name through the global indicator mapping rule base according to the standard indicator name in the monitoring query request input by the user;

[0234] S45. Obtain resource monitoring indicators based on the converted indicator names.

[0235] For example, building a global indicator mapping rule base and adapting standardized access to different monitoring devices by configuring an adaptation layer can also be performed in the following ways:

[0236] 1. Confirm the GPU indicators to be monitored and the names of the platform standard indicators:

[0237] Assume that the platform GPU monitoring indicators and platform standard indicator names are as follows:

[0238] GPU usage: gpu_utilization;

[0239] GPU memory usage: gpu_memory_used;

[0240] GPU temperature: gpu_temperature;

[0241] 2. For GPUs from different manufacturers, the indicator names of GPU utilization, GPU memory usage, and GPU temperature are different. Configure the mapping of platform standard indicator names to indicator names of each manufacturer in the platform mapping rule library.

[0242] The mapping rule base can be composed of different rule groups. Each device corresponds to a rule group, and the rule group project maintains the indicator mapping rules of the device name.

[0243] Take Nvidia and Cambricon GPUs as examples:

[0244] For Nvidia, the names of these indicators are:

[0245] GPU usage: DCGM_FI_DEV_GPU_UTIL;

[0246] GPU video memory usage: DCGM_FI_DEV_FB_USED;

[0247] GPU temperature: DCGM_FI_DEV_GPU_TEMP;

[0248] Then configure the mapping rules in the rule base as follows:

[0249] [NVIDIA];

[0250] gpu_utilization: DCGM_FI_DEV_GPU_UTIL;

[0251] gpu_memory_used: DCGM_FI_DEV_FB_USED;

[0252] gpu_temperature: DCGM_Fl_DEV_GPU_TEMP;

[0253] For Cambricon, the names of these indicators are:

[0254] GPU usage: mlu_utilization;

[0255] GPU video memory usage: mlu_memory_used;

[0256] GPU temperature: mlu_temperature;

[0257] The configuration mapping rules are as follows:

[0258] [CAMBRICON];

[0259] gpu_utilization: mlu_utilization;

[0260] gpu_memory_used: mlu_memory_used;

[0261] gpu_temperature: mlu_temperature;

[0262] 3. The platform monitoring module uses standard indicator names to write PromQL. PromQL pre-sets the mapping rule library in the adaptation layer, matches the mapping rules based on the names, and then performs name mapping according to the matching mapping rules. The indicator names in PromQL are converted to the indicator names of the actual manufacturer's GPU, and then the converted PromQL is submitted to Prometheus for execution. A set of code + mapping rule library is used to adapt to the monitoring of multiple manufacturers' GPUs.

[0263] like Figure 4 As shown, take querying the GPU memory usage of the project "your_project_name" as an example:

[0264] (1) Call the resource query API for the project's GPU memory usage, passing the project name "projectA" as a parameter;

[0265] (2) The GPU memory usage query API generates a PromQL query statement with a standard name:

[0266]

[0267] (3) Configure the Exporter adaptation layer and obtain the GPU Exporter configuration, assuming that it includes nvidia, cambricon, etc.;

[0268] (4) Get the indicator mapping rules of the nvidia rule group:

[0269] gpu_utilization: DCGM_FI_DEV_GPU_UTIL

[0270] gpu_memory_used: DCGM_FI_DEV_FB_USED

[0271] gpu_temperature: DCGM_FI_DEV_GPU_TEMP

[0272] (5) According to the indicator mapping rules, convert the gpu_memory_used indicator name in the query statement, generate a new PromQL, and submit it to Prometheus for execution:

[0273]

[0274]

[0275] (6) Get the indicator value and return the result.

[0276] In addition, it is necessary to explain:

[0277] Prometheus is an open source real-time monitoring and alarm system. Its main features include multi-dimensional data model, flexible query statement PromQL and data visualization display;

[0278] Kubernetes is an open source platform for managing containerized applications on multiple hosts in a cloud platform.

[0279] cadvisor is a tool that monitors Kubernetes containers through resource usage and performance characteristics of containers;

[0280] Kubelet is an important component in the Kubernetes cluster. It runs on each node and is responsible for managing containers and container groups on the node.

[0281] kubeGenericRuntimeManager is the interface class responsible for container runtime management in kubelet;

[0282] Exporter is a component that collects monitoring data and provides data to the outside world through Prometheus monitoring specifications;

[0283] kube-state-metrics is a component of Kubernetes. It generates data about the status of different resources by listening to the Kubernetes API server. Prometheus can capture and store this data. kube_pod_labels is a metric provided by kube-state-metrics. This metric contains the labels of all pods.

[0284] Labels are a key mechanism in Kubernetes for classifying and identifying resources. Labels are key-value pairs attached to Kubernetes resources (such as Pod, Service, Deployment, etc.). These key-value pairs are used to manage and select resources. Labels enable users to efficiently manage and query resources based on business needs.

[0285] Pod is the smallest deployable computing unit and a core concept in Kubernetes. Each container group represents a collection of one or more containers running in a cluster. All model training, prediction and other tasks of the AI ​​platform will create one or more container groups. By counting the resource usage of these container groups, the resource usage of the AI ​​platform can be monitored.

[0286] A namespace is a logical separation mechanism used to divide resources in a cluster into multiple isolated groups. Each namespace provides an independent environment to ensure that resources between different projects, teams, or applications do not interfere with each other.

[0287] like Figure 5 As shown, according to another aspect of the present invention, there is provided an AI platform resource monitoring system based on Prometheus, the AI ​​platform resource monitoring system based on Prometheus comprising: a resource label configuration module 1, a wide table generation module 2, a resource usage total amount calculation module 3 and a standardized access module 4;

[0288] The resource tag configuration module 1 is used to create resource objects of the container management platform based on the cloud native AI platform and configure resource tags for each resource object;

[0289] The wide table generation module 2 is used to filter resource labels that meet monitoring requirements based on a regular expression matching algorithm using a pre-configured label indicator set, perform joint table association processing on the filtered resource labels and the resource monitoring indicators provided by the open source container monitoring tool, and generate a wide table;

[0290] The resource usage total amount calculation module 3 is used to aggregate the resource monitoring indicators in the wide table and calculate the total resource usage under the preset query dimension;

[0291] The standardized access module 4 is used to construct a global indicator mapping rule base and adapt the standardized access of different monitoring devices by configuring an adaptation layer.

[0292] As a preferred implementation, the creation of resource objects of the container management platform based on the cloud native AI platform and configuring resource tags for each resource object include:

[0293] When executing tasks on the cloud-native AI platform, resource objects for managing and deploying containers are created according to business needs;

[0294] Define and assign a name and namespace to each resource object created;

[0295] Use the labeling function of the container management platform to add resource labels to each object;

[0296] Use the command line tool of the container management platform to query the resource tags of the resource object and verify whether the resource tags are correctly added to the resource object. If they are not correctly added, use the update command to reconfigure the correct tags for the resource object.

[0297] As a preferred implementation, the regular expression matching algorithm is used to filter resource labels that meet monitoring requirements using a pre-configured label indicator set, and the filtered resource labels are associated with resource monitoring indicators provided by an open source container monitoring tool to generate a wide table, including:

[0298] Based on the resource tags configured for each resource object, the Prometheus monitoring platform is used to collect the indicator information of the resource tags and build a tag indicator set;

[0299] According to monitoring requirements, extract the regular expression and use it as the filtering condition. By building the abstract syntax tree corresponding to the regular expression, obtain the prefix factor and necessary factor set;

[0300] Using the extracted prefix factors and necessary factors, match them on the generalized suffix tree of the constructed tag index set to screen candidate resource tags;

[0301] The indicators of the filtered resource tags are associated with the resource monitoring indicators provided by the open source container monitoring tool to generate a wide table containing resource tag indicators and resource monitoring indicators.

[0302] As a preferred implementation, the aggregating of resource monitoring indicators in the wide table and calculating the total resource usage under a preset query dimension includes:

[0303] Determine the resource monitoring indicators that need to be queried based on the preset query dimensions;

[0304] Use SQL language to create aggregate query statements based on query dimensions and resource monitoring indicators that need to be queried;

[0305] Execute the aggregation query statement, extract the indicator data from the wide table and perform aggregation processing to obtain the total resource usage under each preset dimension.

[0306] As a preferred implementation, the construction of a global indicator mapping rule base and adapting standardized access of different monitoring devices by configuring an adaptation layer include:

[0307] Use resource monitoring indicators provided by open source container monitoring tools as standard indicator names;

[0308] Collect and analyze the original monitoring indicator names provided by various monitoring devices, establish a mapping relationship between the original monitoring indicator name of each monitoring device and the standard indicator name, and create a global indicator mapping rule base;

[0309] According to the global indicator mapping rule base, an adaptation layer is established for receiving a monitoring query request containing a standard indicator name input by a user;

[0310] The adaptation layer converts the indicator name according to the standard indicator name in the monitoring query request input by the user through the global indicator mapping rule base;

[0311] Get resource monitoring indicators based on the converted indicator names.

[0312] In summary, with the help of the above technical solution of the present invention, the present invention can use PormQL very conveniently to realize resource monitoring at the tenant, organization, project, and user level by means of aggregate query with kube_pod_labels joint table, improve development and query efficiency, and realize resource monitoring at the tenant, organization, project, and user level, so that the AI ​​platform can better control the utilization of resources, better carry out resource recovery, resource planning, cost division, etc., through a unified query adaptation layer, shield the complexity brought by different device monitoring plug-ins, reduce the query interface code volume and complexity, and improve the maintainability and readability of the code. The present invention can be for GPU devices of different manufacturers, by defining standard indicator names, and managing the mapping relationship between standard indicator names and GPU indicator names of different devices through the rule library. Developers use standard indicator names for code development. In the actual product deployment and operation process, indicator name mapping is automatically performed according to the actual GPU manufacturer. When a new device is added, it is only necessary to add new mapping rules to the rule library without adding or modifying the code. The present invention uses a regular expression matching algorithm to flexibly define complex matching rules to ensure that only resource tags that meet specific conditions are screened out. Prefix factors and necessary factors are used to optimize the matching process, which can greatly reduce unnecessary searches and comparisons, thereby improving matching efficiency. By constructing data structures such as abstract syntax trees and generalized suffix trees, complex matching logic and label structures are presented in a visual and structured manner, making management and maintenance easier. By associating the screened resource tags with resource monitoring indicators in a joint table, a wide table containing rich information can be generated. This wide table not only contains resource monitoring data, but also contains label information associated with these resources, making data integration and comprehensive analysis possible.

[0313] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program code.

[0314] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A Prometheus-based AI platform resource monitoring method, characterized in that: The Prometheus-based AI platform resource monitoring method includes the following steps: S1. Based on the cloud-native AI platform, create resource objects of the container management platform and configure resource tags for each resource object; S2. Based on the regular expression matching algorithm, use the pre-configured label indicator set to filter resource labels that meet the monitoring requirements, associate the filtered resource labels with the resource monitoring indicators provided by the open source container monitoring tool, and generate a wide table; S3: Aggregate the resource monitoring indicators in the wide table and calculate the total resource usage under the preset query dimension; S4. Build a global indicator mapping rule base and adapt the standardized access of different monitoring devices by configuring the adaptation layer.

2. According to the Prometheus-based AI platform resource monitoring method according to claim 1, it is characterized in that: The steps of creating resource objects of the container management platform based on the cloud native AI platform and configuring resource tags for each resource object include: S11. During the execution of tasks on the cloud native AI platform, resource objects for managing and deploying containers are created according to business needs; S12. Define and assign a name and namespace to each created resource object; S13. Use the labeling function of the container management platform to add resource labels to each object; S14. Use the command line tool of the container management platform to query the resource tag of the resource object and verify whether the resource tag is correctly added to the resource object. If it is not correctly added, use the update command to reconfigure the correct tag for the resource object.

3. According to the Prometheus-based AI platform resource monitoring method of claim 1, it is characterized in that: The method of using a pre-configured label indicator set based on a regular expression matching algorithm to filter resource labels that meet monitoring requirements, and performing a joint table association process on the filtered resource labels and the resource monitoring indicators provided by the open source container monitoring tool to generate a wide table includes the following steps: S21. Based on the resource tags configured for each resource object, the Prometheus monitoring platform is used to collect the indicator information of the resource tags and build a tag indicator set. S22. According to monitoring requirements, extract the regular expression and use it as a filtering condition, and obtain the prefix factor and the necessary factor set by constructing an abstract syntax tree corresponding to the regular expression; S23, using the extracted prefix factors and necessary factors, matching is performed on the generalized suffix tree of the constructed tag index set to screen candidate resource tags; S24. Associate the index of the filtered resource tag with the resource monitoring index provided by the open source container monitoring tool to generate a wide table including the resource tag index and the resource monitoring index.

4. According to the Prometheus-based AI platform resource monitoring method of claim 3, it is characterized in that: The method of extracting a regular expression according to monitoring requirements and using it as a filtering condition, and obtaining a prefix factor and a necessary factor set by constructing an abstract syntax tree corresponding to the regular expression includes the following steps: S221, determining the resource tags to be monitored according to the monitoring requirements, establishing a regular expression and parsing the regular expression using grammar rules, and constructing a corresponding abstract syntax tree, where the nodes of the abstract syntax tree represent components of the regular expression; S222, extract all leaf nodes on the path from the root node to the left subtree in the abstract syntax tree and construct a prefix factor set; S223. Traverse all paths of the abstract syntax tree, extract the nodes that must appear and construct a set of necessary factors.

5. The AI ​​platform resource monitoring method based on Prometheus according to claim 4 is characterized in that: The method of determining the resource tags to be monitored according to the monitoring requirements, establishing a regular expression and parsing the regular expression using grammar rules, and constructing a corresponding abstract syntax tree, wherein the nodes of the abstract syntax tree represent the components of the regular expression, comprises the following steps: S2221. Based on the determined resource tag, convert the resource tag into a regular expression using a context-free grammar; S2222, using a lexical analyzer to decompose the regular expression into a string to obtain a token sequence; S2223. Based on the syntax analyzer, convert the token sequence into an abstract syntax tree according to the grammar rules.

6. The AI ​​platform resource monitoring method based on Prometheus according to claim 3 is characterized in that: The method of using the extracted prefix factors and necessary factors to match on the generalized suffix tree of the constructed tag index set and screening candidate resource tags includes the following steps: S231, traversing the label indicator set, and adding the resource label in the label indicator set as a suffix to the generalized suffix tree; S232, for each prefix factor and each necessary factor, use depth-first search to find all paths matching the prefix factor and the necessary factor in the generalized suffix tree, record the leaf nodes of all paths and establish a prefix candidate interval and a necessary candidate interval respectively; S233 , cross-comparing the prefix candidate interval with the necessary candidate interval, screening out leaf nodes that exist in the intersection area of ​​the prefix candidate interval and the necessary candidate interval, and using the screened out leaf nodes as final candidate resource labels.

7. The AI ​​platform resource monitoring method based on Prometheus according to claim 6 is characterized in that: The step of associating the index of the filtered resource tag with the resource monitoring index provided by the open source container monitoring tool to generate a wide table including the resource tag index and the resource monitoring index includes the following steps: S241. Obtain resource monitoring indicators from open source container monitoring tools; S242, based on the joint table association rule, the resource monitoring indicator is associated with the indicator of the filtered resource tag according to the container group name and the namespace to obtain an association result; S243. According to the association result, a wide table including resource monitoring indicators and resource label indicators is obtained.

8. The AI ​​platform resource monitoring method based on Prometheus according to claim 1, characterized in that: The aggregating of resource monitoring indicators in the wide table and calculating the total resource usage under the preset query dimension includes the following steps: S31. Determine the resource monitoring indicator to be queried according to the preset query dimension; S32. Create an aggregate query statement using SQL language according to the query dimension and the resource monitoring indicator to be queried; S33. Execute the aggregate query statement, extract the indicator data from the wide table and perform aggregation processing to obtain the total resource usage under each preset dimension.

9. The AI ​​platform resource monitoring method based on Prometheus according to claim 1, characterized in that: The construction of a global indicator mapping rule base and adapting standardized access of different monitoring devices by configuring an adaptation layer include the following steps: S41. Use resource monitoring indicators provided by open source container monitoring tools as standard indicator names; S42, collecting and analyzing the original monitoring indicator names provided by various monitoring devices, establishing a mapping relationship between the original monitoring indicator name of each monitoring device and the standard indicator name, and creating a global indicator mapping rule base; S43, establishing an adaptation layer for receiving a monitoring query request containing a standard indicator name input by a user according to a global indicator mapping rule base; S44, the adaptation layer converts the indicator name through the global indicator mapping rule base according to the standard indicator name in the monitoring query request input by the user; S45. Obtain resource monitoring indicators based on the converted indicator names.

10. A Prometheus-based AI platform resource monitoring system, used to implement the Prometheus-based AI platform resource monitoring method described in any one of claims 1 to 9, characterized in that: The AI ​​platform resource monitoring system based on Prometheus includes: resource label configuration module, wide table generation module, total resource usage calculation module and standardized access module; The resource tag configuration module is used to create resource objects of the container management platform based on the cloud native AI platform and configure resource tags for each resource object; The wide table generation module is used to use a pre-configured label indicator set to filter resource labels that meet monitoring requirements based on a regular expression matching algorithm, perform joint table association processing on the filtered resource labels and resource monitoring indicators provided by the open source container monitoring tool, and generate a wide table; The resource usage total amount calculation module is used to aggregate the resource monitoring indicators in the wide table and calculate the total resource usage under the preset query dimension; The standardized access module is used to construct a global indicator mapping rule base and adapt the standardized access of different monitoring devices by configuring the adaptation layer.

11. The AI ​​platform resource monitoring system based on Prometheus according to claim 10, characterized in that: The method of creating resource objects of the container management platform based on the cloud native AI platform and configuring resource tags for each resource object includes: When executing tasks on the cloud-native AI platform, resource objects for managing and deploying containers are created according to business needs; Define and assign a name and namespace to each resource object created; Use the labeling function of the container management platform to add resource labels to each object; Use the command line tool of the container management platform to query the resource tags of the resource object and verify whether the resource tags are correctly added to the resource object. If they are not correctly added, use the update command to reconfigure the correct tags for the resource object.

12. The AI ​​platform resource monitoring system based on Prometheus according to claim 10, characterized in that: Based on the regular expression matching algorithm, the pre-configured label indicator set is used to filter resource labels that meet the monitoring requirements, and the filtered resource labels are associated with the resource monitoring indicators provided by the open source container monitoring tool to generate a wide table, including: Based on the resource tags configured for each resource object, the Prometheus monitoring platform is used to collect the indicator information of the resource tags and build a tag indicator set; According to monitoring requirements, extract the regular expression and use it as the filtering condition. By building the abstract syntax tree corresponding to the regular expression, obtain the prefix factor and necessary factor set; Using the extracted prefix factors and necessary factors, match them on the generalized suffix tree of the constructed tag index set to screen candidate resource tags; The indicators of the filtered resource tags are associated with the resource monitoring indicators provided by the open source container monitoring tool to generate a wide table containing resource tag indicators and resource monitoring indicators.

13. The AI ​​platform resource monitoring system based on Prometheus according to claim 10, characterized in that: The aggregating of resource monitoring indicators in the wide table and calculating the total resource usage under the preset query dimension includes: Determine the resource monitoring indicators that need to be queried based on the preset query dimensions; Use SQL language to create aggregate query statements based on query dimensions and resource monitoring indicators that need to be queried; Execute the aggregation query statement, extract the indicator data from the wide table and perform aggregation processing to obtain the total resource usage under each preset dimension.

14. The AI ​​platform resource monitoring system based on Prometheus according to claim 10, characterized in that: The construction of a global indicator mapping rule base and the adaptation of standardized access of different monitoring devices by configuring an adaptation layer include: Use resource monitoring indicators provided by open source container monitoring tools as standard indicator names; Collect and analyze the original monitoring indicator names provided by various monitoring devices, establish a mapping relationship between the original monitoring indicator names of each monitoring device and the standard indicator names, and create a global indicator mapping rule base; According to the global indicator mapping rule base, an adaptation layer is established for receiving a monitoring query request containing a standard indicator name input by a user; The adaptation layer converts the indicator name according to the standard indicator name in the monitoring query request input by the user through the global indicator mapping rule base; Get resource monitoring indicators based on the converted indicator names.

Images