Inference service monitoring method and device, computer equipment and storage medium

Abstract

The invention relates to an inference service monitoring method and device, computer equipment and a storage medium. According to the method, the plurality of data acquisition scripts are arranged in the nodes in the cluster and are used for acquiring the monitoring data of the service monitoring indexes, and the plug-in is arranged to drive the server proxy component to regularly execute the execution tasks of the data acquisition scripts so as to collect the monitoring data, so that the target monitoring data expected to be monitored can be directly acquired; the target inference service is created when the monitoring data collection command of the target index is received, the corresponding relation between the virtual line of the target inference service and the container set created by the target inference service is obtained, the target monitoring data can be quickly retrieved and obtained based on the corresponding relation, multi-layer calling and searching are not needed, and the efficiency is improved. The corresponding data can be directly called and checked, and the searching efficiency of the target monitoring data is improved.

Description

Technical Field

[0001] The present application relates to the field of server technology, and in particular to a method, apparatus, computer device and storage medium for monitoring an inference service. Background Art

[0002] Inference service is the process of deploying trained models online for real-time prediction. Therefore, it is necessary to monitor the inference service in real time to ensure the normal operation of the service. The monitoring indicators of inference service include: request success rate, request latency, number of service replicas, service central processing unit (CPU) utilization, memory utilization, network IO, and graphics processing unit (GPU) utilization and video memory utilization.

[0003] Most current monitoring and reasoning services use prometheus, which is an open source service monitoring system and time series database. However, it is not flexible enough for clustered and highly available cloud platforms, and its configuration is complex, introducing more technology stacks. The plug-in-driven server agent (telegraf) used to collect and report indicators is more lightweight, and using custom scripts to obtain data is more flexible and controllable. The time series database (Influxdb) to store data is also more targeted. For the monitoring indicators of the reasoning service deployed by HELM, the implementation problems are: performance-related indicators are separated from the encapsulation statements of prometheus and need to be calculated by the plug-in-driven server agent (telegraf) script; graphics processor-related indicators need to find the corresponding relationship between the container set (pod) and the container, get the used graphics processor from the container's environment variables, and then get the graphics processor data; service request-related indicators are customized due to deployment, and each service will create its own istio service grid to receive external requests. Therefore, the indicator data of the existing data collection script needs to be retrieved at multiple layers, which is inconvenient, and it is not possible to directly retrieve and view the corresponding data. Summary of the invention

[0004] Based on this, a method, device, computer equipment and storage medium for monitoring a reasoning service are provided to solve the technical problem that the indicator data retrieval of the current data collection script for monitoring reasoning services requires multiple layers of retrieval and is inconvenient to search, and cannot directly retrieve and view the corresponding data.

[0005] In one aspect, a method for monitoring an inference service is provided, the method comprising:

[0006] Multiple data collection scripts are set in the nodes of the cluster, and the data collection scripts are used to collect monitoring data of service monitoring indicators;

[0007] Installing a plug-in driven server agent component on the node, setting the execution task of the data collection script in the configuration file of the plug-in driven server agent component, and the plug-in driven server agent component periodically executing the execution task of the data collection script to collect monitoring data;

[0008] In response to receiving a monitoring data collection command for a target indicator, a target reasoning service is created, a correspondence between a virtual line of the target reasoning service and a container set created by the target reasoning service is obtained, and based on the correspondence, a data collection script corresponding to the target reasoning service is executed by the plug-in driving server agent component to obtain target monitoring data;

[0009] The target monitoring data is analyzed and calculated to obtain performance data corresponding to the target indicator, and the performance data is displayed in time series.

[0010] In one embodiment, the method further comprises:

[0011] Configure a time series database to store the collected monitoring data;

[0012] In response to a command for collecting monitoring data of a target indicator, the target monitoring data corresponding to the target indicator is queried and acquired from the time series database.

[0013] In one embodiment, obtaining a correspondence between a virtual circuit of the target reasoning service and a container set created by the target reasoning service, and obtaining target monitoring data by executing a data collection script corresponding to the target reasoning service through the plug-in driven server agent component based on the correspondence includes:

[0014] Setting a first association tag between the name of the inference service and a virtual circuit of the inference service, and setting a second association tag between the name of the inference service and a container set created by the inference service;

[0015] When creating a target reasoning service, associating a virtual circuit of the target reasoning service with a container set created by the target reasoning service using the first association tag and the second association tag;

[0016] The plug-in driven server agent component controls the execution of the data collection script corresponding to the target reasoning service, and obtains the target monitoring data according to the correspondence between the virtual line of the target reasoning service and the container set created by the target reasoning service.

[0017] In one embodiment, the step of setting multiple data collection scripts on nodes in the cluster includes:

[0018] Setting the data collection script includes collecting a virtual line description script, collecting a container performance script, collecting a deployment copy script, collecting a service request duration script, collecting a service request quantity script, collecting a CPU performance script, collecting a memory performance script, and collecting a network performance script;

[0019] The virtual line description acquisition script is used to obtain the container set, container set label, and virtual line under the namespace created by the user in the cluster, and traverse and query the label of the inference service virtual line to make the virtual line correspond to the container set, and store the virtual line name, inference service name, the node where the inference service is located, and the Internet protocol of the corresponding container set in the time series database;

[0020] The collection container performance script is used to obtain the container corresponding to the container set under the namespace created by the user in the cluster, and traverse the container to query the internal environment variables to obtain the graphics processor performance data;

[0021] The deployment replica collection script is used to collect the expected number of replicas and the number of ready replicas of deployment, replica set, ready replica, background support service set, and container set;

[0022] The collection service request duration script is used to record the duration of the data called by the external request after the inference service is deployed;

[0023] The script for collecting the number of service requests is used to collect Internet protocol records of the container set corresponding to the inference service deployment node, and call the Prometheus lightweight interface to query the number of requests for the inference service and the incremental value of the number of requests;

[0024] The CPU performance acquisition script is used to obtain CPU performance indicator data;

[0025] The memory performance acquisition script is used to obtain memory performance indicator data;

[0026] The network performance acquisition script is used to obtain network performance indicator data.

[0027] In one embodiment, analyzing and calculating the target monitoring data to obtain the performance data corresponding to the target indicator includes:

[0028] Analyze the CPU performance indicator data to obtain the CPU usage value corresponding to each moment within the CPU core usage duration, and obtain the CPU usage rate (container_CPU_usage_seconds_total) of the first duration by dividing the difference of the CPU usage values ​​within a first duration range within the CPU core usage duration by the first duration;

[0029] The upper limit of the number of cores allocated to the CPU of the current container is calculated through container_spec_CPU_quota / container_spec_CPU_period, where container_spec_CPU_quota is the container CPU quota and container_spec_CPU_period is the container CPU period.

[0030] Calculate CPU usage by sum(rate(container_CPU_usage_seconds_total)) / sum(container_spec_CPU_quota / container_spec_CPU_period);

[0031] The total occupancy of all CPU cores is divided by the total number of occupied cores to obtain the CPU utilization of each core.

[0032] In one embodiment, analyzing and calculating the target monitoring data to obtain the performance data corresponding to the target indicator includes:

[0033] Get the number of used bytes of container memory in the container set container_memory_working_set_bytes, and sum the number of used bytes of containers in the container set sum(container_memory_working_set_bytes);

[0034] Get the maximum byte limit of container memory in the container set, determine whether the maximum byte limit of container memory has a zero value, if not, sum the maximum byte limit of container memory in the container set, if there is a zero value, get the total container memory machine_memory_bytes;

[0035] In response to the container memory maximum byte limit not having a zero value, calculating the memory utilization by sum(container_memory_working_set_bytes) / sum(container_spec_memory_limit_bytes);

[0036] In response to the container memory maximum byte limit having a zero value, the memory utilization is calculated by sum(container_memory_working_set_bytes) / machine_memory_bytes.

[0037] In one embodiment, analyzing and calculating the target monitoring data to obtain the performance data corresponding to the target indicator includes:

[0038] Obtain the number of network input bytes and the number of network output bytes to form the number of network input and output bytes corresponding to each moment;

[0039] The network input and output rate is obtained by dividing the difference in the number of network input and output bytes within the second time range by the second time and then taking the time derivative.

[0040] In another aspect, a reasoning service monitoring device is provided, the device comprising:

[0041] A collection module, used to set multiple data collection scripts in the nodes in the cluster, and the data collection scripts are used to collect monitoring data of service monitoring indicators;

[0042] A monitoring service module, used for installing a plug-in driven server agent component on the node, setting the execution task of the data acquisition script in the configuration file of the plug-in driven server agent component, and the plug-in driven server agent component periodically executing the execution task of the data acquisition script to collect monitoring data;

[0043] The platform page display module is used to create a target reasoning service in response to receiving a monitoring data collection command for a target indicator, obtain the correspondence between the virtual line of the target reasoning service and the container set created by the target reasoning service, and based on the correspondence, execute the data collection script corresponding to the target reasoning service through the plug-in driving server agent component to obtain the target monitoring data.

[0044] In another aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0045] Multiple data collection scripts are set in the nodes of the cluster, and the data collection scripts are used to collect monitoring data of service monitoring indicators;

[0046] Installing a plug-in driven server agent component on the node, setting the execution task of the data collection script in the configuration file of the plug-in driven server agent component, and the plug-in driven server agent component periodically executing the execution task of the data collection script to collect monitoring data;

[0047] In response to receiving a monitoring data collection command for a target indicator, a target reasoning service is created, a correspondence between a virtual line of the target reasoning service and a container set created by the target reasoning service is obtained, and based on the correspondence, a data collection script corresponding to the target reasoning service is executed by the plug-in driving server agent component to obtain target monitoring data;

[0048] The target monitoring data is analyzed and calculated to obtain performance data corresponding to the target indicator, and the performance data is displayed in time series.

[0049] In another aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0050] Multiple data collection scripts are set in the nodes of the cluster, and the data collection scripts are used to collect monitoring data of service monitoring indicators;

[0051] Installing a plug-in driven server agent component on the node, setting the execution task of the data collection script in the configuration file of the plug-in driven server agent component, and the plug-in driven server agent component periodically executing the execution task of the data collection script to collect monitoring data;

[0052] In response to receiving a monitoring data collection command for a target indicator, a target reasoning service is created, a correspondence between a virtual line of the target reasoning service and a container set created by the target reasoning service is obtained, and based on the correspondence, a data collection script corresponding to the target reasoning service is executed by the plug-in driving server agent component to obtain target monitoring data;

[0053] The target monitoring data is analyzed and calculated to obtain performance data corresponding to the target indicator, and the performance data is displayed in time series.

[0054] The above-mentioned reasoning service monitoring method, device, computer equipment and storage medium can directly obtain the target monitoring data expected to be monitored by setting multiple data collection scripts in the nodes in the cluster for collecting monitoring data of service monitoring indicators, and setting a plug-in to drive the server agent component to periodically execute the execution task of the data collection script to collect monitoring data, and set to create a target reasoning service when receiving the monitoring data collection command of the target indicator, obtain the correspondence between the virtual line of the target reasoning service and the container set created by the target reasoning service, and can quickly retrieve and obtain the target monitoring data based on the correspondence, without the need for multi-layer retrieval and search, and can directly retrieve and view the corresponding data, thereby improving the search efficiency of the target monitoring data. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 This is an application environment diagram of the reasoning service monitoring method in one embodiment of the present application;

[0057] Figure 2 This is a flowchart of an inference service monitoring method in one embodiment of the present application;

[0058] Figure 3 A flowchart of the steps of obtaining the correspondence between the virtual circuit of the target reasoning service and the container set created by the target reasoning service in one embodiment of the present application, and obtaining the target monitoring data by executing the data collection script corresponding to the target reasoning service through the plug-in driven server agent component based on the correspondence;

[0059] Figure 4 This is a structural block diagram of an inference service monitoring device in one embodiment of the present application;

[0060] Figure 5 This is a diagram of the internal structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] As described in the background technology, the inference service is the process of deploying the trained model online for real-time prediction. Therefore, the inference service needs to be monitored in real time to ensure that the service can run normally. The monitoring indicators of the inference service include: request success rate, request latency, number of service replicas, service central processing unit (CPU) utilization, memory utilization, network IO, and graphics processing unit (GPU) utilization and video memory utilization.

[0063] Currently, most monitoring and reasoning services use Prometheus, which is an open source service monitoring system and time series database. However, it is not flexible enough for clustered high-availability cloud platforms, and its configuration is complex, which introduces more technology stacks. Telegraf is a plug-in-driven server agent for collecting and reporting indicators. Telegraf is more lightweight, and using custom scripts to obtain data is more flexible and controllable. Influxdb is a time series database, and influxdb stores data more specifically.

[0064] In the k8s cloud platform, in most cases, the corresponding inference service and container set (pod) will be directly created according to the configuration. Highly integrated use and implementation are more convenient, but a more flexible HELM deployment method is also required, providing deployment configuration templates to create more customized inference services through custom YAML. HELM is the official package management tool of Kubernetes, which deploys applications in k8s through templates and configurations. For the inference service deployed by HELM, the created service and container set cannot be associated simply by name. Labels need to be introduced to associate the created virtual circuit (svc) with the container set through labels, so as to obtain the monitoring and collection indicators of the corresponding target.

[0065] For the monitoring indicators of the reasoning service deployed by HELM, the implementation problems are: the performance-related indicators are separated from the encapsulation statements of prometheus and need to be calculated by the telegraf script; the graphics processor-related indicators need to find the correspondence between the container set and the container, get the used graphics processor from the environment variables of the container, and then get the graphics processor data; the service request-related indicators are customized deployments, and each service will create its own istio service grid to receive external requests. Istio is an open source project platform that solves problems such as service discovery, load balancing, traffic control, and fault recovery in microservice architectures. Therefore, the indicator data retrieval of the existing data collection script requires multiple layers of retrieval, which is inconvenient, and it is not possible to directly retrieve and view the corresponding data.

[0066] The reasoning service monitoring method provided in this application can be applied to Figure 1In the application environment shown. Among them, k8s is the abbreviation of Kubernetes, an open source containerized application for managing multiple hosts in the cloud platform. Based on the underlying services and containers of k8s, the collection module, the time series database (Influxdb) storage module, the monitoring service module and the platform page display module are set. Multiple data collection scripts are set in the collection module. The data collection scripts include the collection virtual line description script (collect_svc_describe.py), the collection container performance script (collect_container_performance.py), the collection deployment replica script (collect_deployment_replica.py), the collection service request duration script (service_istio_requests_duration.py), the collection service request quantity script (service_istio_requests_total.py), the collection central processing unit performance script (service_cpu_cadvisor.py), the collection memory performance script (service_memory_cadvisor.py), and the collection network performance script (service_network_cadvisor.py). In the collection module, a plug-in driven server agent (telegraf) component is set to manage the data collection script. The time series database storage module stores the collected data in a formal manner. The monitoring service module can directly respond to SQL commands, query and calculate monitoring data from the time series database, and the platform page display module displays the queried monitoring data on the page.

[0067] In one embodiment, Figure 2 As shown, a method for monitoring an inference service is provided, comprising the following steps:

[0068] Step S1, multiple data collection scripts are set in the nodes in the cluster, and the data collection scripts are used to collect monitoring data of service monitoring indicators;

[0069] Step S2, installing a plug-in driven server agent component on the node, setting an execution task of a data collection script in a configuration file of the plug-in driven server agent component, and causing the plug-in driven server agent component to periodically execute the execution task of the data collection script to collect monitoring data;

[0070] Step S3, in response to receiving a monitoring data collection command for a target indicator, creating a target reasoning service, obtaining a correspondence between a virtual line of the target reasoning service and a container set created by the target reasoning service, and executing a data collection script corresponding to the target reasoning service through a plug-in-driven server agent component based on the correspondence to obtain target monitoring data;

[0071] Step S4, analyze and calculate the target monitoring data to obtain performance data corresponding to the target indicator, and display the performance data in time series.

[0072] Among them, by setting multiple data collection scripts in the nodes in the cluster for monitoring data collection of service monitoring indicators, and setting the plug-in to drive the server agent component to periodically execute the execution task of the data collection script to collect monitoring data, the target monitoring data that is expected to be monitored can be directly obtained, and when the monitoring data collection command of the target indicator is received, a target inference service is created, and the correspondence between the virtual line of the target inference service and the container set created by the target inference service is obtained. The target monitoring data can be quickly retrieved based on the corresponding relationship without the need for multi-layer retrieval and search, and the corresponding data can be directly retrieved and viewed, thereby improving the search efficiency of the target monitoring data.

[0073] In this embodiment, the method further includes:

[0074] Configure a time series database to store the collected monitoring data;

[0075] In response to a command to collect monitoring data for executing a target indicator, target monitoring data corresponding to the target indicator is queried and acquired from a time series database.

[0076] Among them, the collected monitoring data is stored in a time series database, which can realize long-term data preservation and facilitate subsequent query acquisition and data calculation.

[0077] like Figure 3 As shown, in this embodiment, obtaining the correspondence between the virtual circuit of the target reasoning service and the container set created by the target reasoning service, and obtaining the target monitoring data by executing the data collection script corresponding to the target reasoning service through the plug-in driving server agent component based on the correspondence includes:

[0078] Step S31, setting a first association tag for the name of the inference service and the virtual circuit of the inference service, and setting a second association tag for the name of the inference service and the container set created by the inference service;

[0079] Step S32, when creating a target reasoning service, associating the virtual circuit of the target reasoning service with the container set created by the target reasoning service using the first association tag and the second association tag;

[0080] Step S33, controlling the execution of the data collection script corresponding to the target reasoning service through the plug-in driven server agent component, and acquiring the target monitoring data according to the correspondence between the virtual circuit of the target reasoning service and the container set created by the target reasoning service.

[0081] Specifically: First, when creating an inference service, the services created by the platform with one click and those created by a custom yaml template are distinguished by labeling. The label content is the inference service name for easy matching and querying. The general label is that the service name created by the platform corresponds to the first-layer label name of the inference service virtual circuit (svc) to filter user-deployed services. After filtering, the specific difference is that the service name created by the platform can be fully matched in the second-layer label name; the service created by the user-defined yaml template obtains the index through a fixed key value in the second-layer label, and the index corresponds to the index label value in the specific created pod, which can adapt to the index matching situation of one service creating multiple pods. Then, through the custom collection script, the service correspondence relationship of the above two situations is collected, and then the specific data of the service monitoring indicators is collected through the corresponding results.

[0082] When creating an inference service, the service name is fixedly stored with the Selector label as the first-level label matching; for platform-created services, the service name is stored with the meta.helm.sh / release-name label in the second-level label Annotations for full matching; for user-defined yaml template-created services, the pod index name is stored with the app= label in the second-level label Annotations; the app= label is also used to store the index name in multiple pods created by the corresponding user-defined services for matching between svc and pod.

[0083] For the inference service created by HELM custom YAML, the present invention associates the two by defining the label of the virtual circuit (svc) and the label of the pod. When the page is passed in the user's creation of the inference service name, the underlying svc, pod, and container can be obtained, and then the GPU used can be obtained through the environment variables defined in the container, which can completely connect the query method of the indicator data.

[0084] In this embodiment, setting multiple data collection scripts on nodes in the cluster includes:

[0085] Set up data collection scripts including collecting virtual line description script, collecting container performance script, collecting deployment copy script, collecting service request duration script, collecting service request quantity script, collecting CPU performance script, collecting memory performance script, and collecting network performance script;

[0086] The collection virtual circuit description script is used to obtain the container set, container set label, and virtual circuit under the user-created namespace in the cluster, and traverse the inference service virtual circuit to query its label, so that the virtual circuit corresponds to the container set, and store the virtual circuit name, inference service name, the node where the inference service is located, and the Internet protocol of the corresponding container set in the time series database;

[0087] The container performance collection script is used to obtain the container corresponding to the container set under the user-created namespace in the cluster, and traverse the container to query the internal environment variables to obtain the GPU performance data;

[0088] The deployment replica collection script is used to collect the expected number of replicas and the number of ready replicas of deployments, replica sets, ready replicas, background support service sets, and container sets;

[0089] The service request duration collection script is used to record the duration of external request data after the inference service is deployed;

[0090] The script for collecting the number of service requests is used to collect the Internet protocol records of the container set corresponding to the inference service deployment node, and call the Prometheus lightweight interface to query the number of inference service requests and the incremental value of the number of requests;

[0091] The CPU performance collection script is used to obtain CPU performance indicator data;

[0092] The memory performance collection script is used to obtain memory performance indicator data;

[0093] The network performance collection script is used to obtain network performance indicator data.

[0094] The execution task of the data collection script is set in the configuration file of the plug-in-driven server agent component, and the plug-in-driven server agent component executes the data collection script regularly. The tasks with a collection interval of 15 seconds include: collect_svc_describe.py collection task, collect_container_performance.py collection task, collect_deployment_replica.py collection task, and service_istio_requests_duration.py. Since the above tasks involve traversal or large amounts of data, the collection interval is appropriately extended. Other tasks with a collection interval of 10 seconds include: service_istio_requests_total.py collection task, service_cpu_cadvisor.py collection task, service_memory_cadvisor.py collection task, and service_network_cadvisor.py collection task. The above tasks are real-time monitoring indicator data.

[0095] The above collect_svc_describe.py script obtains the pod, pod label, and svc under the user-created namespace in the cluster, and traverses and queries the labels of the inference service svc to make the svc correspond to the pod, and stores the underlying svc name, the user-created HELM inference service specified name, the node where the service is located after successful creation, and the corresponding pod IP in Influxdb. This step mainly solves the problem that when the user customizes YAML to create a service, the created service name cannot correspond to the svc and pod created at the bottom layer. It maps by traversing and using labels, and records the pod IP for subsequent istio query request records.

[0096] The collect_container_performance.py script above obtains the container corresponding to the pod in the namespace created by the user in the cluster, traverses and queries the internal environment variables of the container, and extracts the UUID of the GPU actually occupied, so that the DCGM component can be used to query the GPU performance data and display it on the page.

[0097] The collect_deployment_replica.py script above uses the k8s command to collect the expected number of replicas and the number of ready replicas for five types of tasks: deployment, replica set, ready replica (statefulset), background support service set (daemonset), and pod, and displays the running status of multiple types of inference services created by users.

[0098] In the above service_istio_requests_duration.py and service_istio_requests_total.py scripts, after the inference service is deployed, the built-in istio service component will record the data of external request calls and provide a query interface. The script combines the recorded pod IP and calls the Prometheus lightweight interface to directly query the request status of the inference service, including the incremental value of all requests, the response_code value returned by the request, and the histogram bucket distribution of the request delay.

[0099] The service_cpu_cadvisor.py, service_memory_cadvisor.py, and service_network_cadvisor.py scripts above can obtain performance indicators of different nodes in a unified manner through the cadvisor interface provided by kubelet.

[0100] Preferably, cadvisor and istio related collection scripts are provided on the master node where the virtual IP in the k8s cluster is located, and container collection scripts are provided on all nodes. The telegraf component is installed on each node, and the configuration of script execution command, collection interval, timeout, etc. is completed, and the telegraf service is started to collect monitoring data of the content printed by the script execution command.

[0101] In this embodiment, analyzing and calculating the target monitoring data to obtain performance data corresponding to the target indicator includes:

[0102] Analyze the CPU performance indicator data to obtain the CPU usage value corresponding to each moment in the CPU core usage time, and obtain the CPU usage rate (container_CPU_usage_seconds_total) of the first time period by dividing the difference of the CPU usage values ​​in the first time period range within the CPU core usage time by the first time period;

[0103] The upper limit of the number of cores allocated to the CPU of the current container is calculated through container_spec_CPU_quota / container_spec_CPU_period, where container_spec_CPU_quota is the container CPU quota and container_spec_CPU_period is the container CPU period.

[0104] Calculate CPU usage by sum(rate(container_CPU_usage_seconds_total)) / sum(container_spec_CPU_quota / container_spec_CPU_period);

[0105] The total occupancy of all CPU cores is divided by the total number of occupied cores to obtain the CPU utilization of each core.

[0106] Specifically, the CPU core occupancy time is a cumulative value. For example, to calculate the CPU occupancy rate for one minute: (current value - value one minute ago) / one minute. container_spec_cpu_quota and container_spec_cpu_period are the core numbers for calculating the upper limit of the CPU allocated to the current container. If the upper limit is not set, these two numbers are empty or zero. If the upper limit is set, container_spec_cpu_period is generally 100000. rate is used to calculate the occupancy rate during this period. In Influxdb, the derivative function is used to implement it, and group by time (duration) is used to group by time and calculate the occupancy rate by derivative.

[0107] In this embodiment, analyzing and calculating the target monitoring data to obtain performance data corresponding to the target indicator includes:

[0108] Get the number of used bytes of container memory in the container set container_memory_working_set_bytes, and sum the number of used bytes of containers in the container set sum(container_memory_working_set_bytes);

[0109] Get the maximum byte limit of container memory in the container set, and determine whether the maximum byte limit of container memory has a zero value. If not, sum the maximum byte limits of container memory in the container set. If there is a zero value, get the total container memory machine_memory_bytes.

[0110] In response to the container memory maximum byte limit not having a zero value, the memory utilization is calculated by sum(container_memory_working_set_bytes) / sum(container_spec_memory_limit_bytes);

[0111] In response to the container memory maximum byte limit having a zero value, the memory utilization is calculated by sum(container_memory_working_set_bytes) / machine_memory_bytes.

[0112] For container_spec_memory_limit_bytes, if the maximum value is set when creating the container, this is the total memory size of the container in bytes. If it is not set, this is 0 and the total memory size of the container machine_memory_bytes needs to be used.

[0113] For machine_memory_bytes, which is the maximum memory value of the node, you need to filter the indicator items and query again. Only one item will be found for one node. For example, to query the total host memory of node1: kubectl get--raw= / api / v1 / nodes / node1 / proxy / metrics / cadvisor|grep machine_memory_bytes|grep machine_id.

[0114] The sum of the used amount of container_memory_working_set_bytes in the pod is used as the numerator.

[0115] If the container_spec_memory_limit_bytes value in the pod is not set and is equal to 0, this value is not used. Instead, it is divided by the total memory of the node, machine_memory_bytes. If container_spec_memory_limit_bytes is set, the total amount is summed and used as the denominator.

[0116] Specifically, when container_spec_memory_limit_bytes is non-zero: sum(container_memory_working_set_bytes) / sum(container_spec_memory_limit_bytes); when container_spec_memory_limit_bytes is zero: sum(container_memory_working_set_bytes) / machine_memory_bytes.

[0117] In this embodiment, analyzing and calculating the target monitoring data to obtain performance data corresponding to the target indicator includes:

[0118] Obtain the number of network input bytes and the number of network output bytes to form the number of network input and output bytes corresponding to each moment;

[0119] The network input and output rate is obtained by dividing the difference in the number of network input and output bytes within the second time range by the second time and then taking the time derivative.

[0120] For example, if the rate is one minute, the network input and output rate is calculated as: (current value - value one minute ago) / one minute, and the derivative can be directly calculated by time.

[0121] This application can use telegraf custom scripts and influxdb storage to replace the acquisition and calculation of monitoring data in a non-prometheus technology stack. The platform is more lightweight, does not introduce more components, and the custom script is more flexible in processing data. The calculation method of CPU, memory, number of requests and success rate replaces the aggregation function provided by prometheus, making the inference service indicator data displayed on the page more complete and accurate. Make the monitoring function of the k8s cloud platform for multi-type and multi-copy inference services more complete.

[0122] For monitoring methods other than the Prometheus technology stack, the present invention uses telegraf to periodically execute custom collection scripts, which can improve the collection of data such as performance, requests, and number of copies of the inference service.

[0123] For data display methods other than the Prometheus technology stack, it is necessary to replace the aggregation function provided by Prometheus. The present invention uses cadvisor and istio services, only relies on the basic component functions of k8s, and does not need to introduce new component services. The corresponding time series data display results are obtained through the basic aggregation functions of influxdb and java calculations.

[0124] In this embodiment, displaying the performance data in time series includes:

[0125] The type of performance data to obtain;

[0126] According to the type of performance data, the performance data is formed into a performance curve graph or a histogram according to the time series;

[0127] Display performance curves or histograms.

[0128] The present invention sums up the CPU occupancy and memory usage of all pods in the service, improves the situation where the CPU has an upper limit on the number of cores and the memory has no upper limit, analyzes the histogram data of the number of successful requests queried by istio and obtains the corresponding bucket range as the basis for calculating the request success rate, making the performance of the entire inference service and the request data display more accurate.

[0129] In the above-mentioned reasoning service monitoring method, by setting multiple data collection scripts in the nodes in the cluster for collecting monitoring data of service monitoring indicators, and setting the plug-in-driven server agent component to periodically execute the execution task of the data collection script to collect monitoring data, the target monitoring data expected to be monitored can be directly obtained, and it is set to create a target reasoning service when the monitoring data collection command of the target indicator is received, and the correspondence between the virtual line of the target reasoning service and the container set created by the target reasoning service is obtained. The target monitoring data can be quickly retrieved based on the corresponding relationship without the need for multi-layer retrieval and search, and the corresponding data can be directly retrieved and viewed, thereby improving the search efficiency of the target monitoring data.

[0130] In one embodiment, Figure 4 As shown, a reasoning service monitoring device 10 is provided, including: a collection module 1, a monitoring service module 2, and a platform page display module 3.

[0131] The collection module 1 is used to set multiple data collection scripts in the nodes in the cluster, and the data collection scripts are used to collect monitoring data of service monitoring indicators.

[0132] The monitoring service module 2 is used to install a plug-in driven server agent component on the node, set the execution task of the data collection script in the configuration file of the plug-in driven server agent component, and the plug-in driven server agent component periodically executes the execution task of the data collection script to collect monitoring data; in response to receiving a monitoring data collection command for a target indicator, create a target reasoning service, obtain the correspondence between the virtual line of the target reasoning service and the container set created by the target reasoning service, and based on the correspondence, execute the data collection script corresponding to the target reasoning service through the plug-in driven server agent component to obtain the target monitoring data.

[0133] The platform page display module 3 is used to analyze and calculate the target monitoring data to obtain the performance data corresponding to the target indicator, and display the performance data in time series.

[0134] like Figure 4 As shown, the inference service monitoring device 10 also includes: a time series database storage module 4.

[0135] The time series database storage module 4 is used to configure the time series database to store the collected monitoring data; in response to a monitoring data collection command for executing a target indicator, query and obtain the target monitoring data corresponding to the target indicator from the time series database.

[0136] In this embodiment, obtaining a correspondence between a virtual circuit of a target reasoning service and a container set created by the target reasoning service, and obtaining target monitoring data by executing a data collection script corresponding to the target reasoning service through a plug-in driven server agent component based on the correspondence includes:

[0137] Setting a first association tag for the name of the inference service and the virtual circuit of the inference service, and setting a second association tag for the name of the inference service and the container set created by the inference service;

[0138] When creating a target reasoning service, associating a virtual circuit of the target reasoning service with a container set created by the target reasoning service using the first association tag and the second association tag;

[0139] The plug-in driven server agent component controls the running of the data collection script corresponding to the target reasoning service, and obtains the target monitoring data according to the correspondence between the virtual line of the target reasoning service and the container set created by the target reasoning service.

[0140] In this embodiment, setting multiple data collection scripts on nodes in the cluster includes:

[0141] Set up data collection scripts including collecting virtual line description script, collecting container performance script, collecting deployment copy script, collecting service request duration script, collecting service request quantity script, collecting CPU performance script, collecting memory performance script, and collecting network performance script;

[0142] The collection virtual circuit description script is used to obtain the container set, container set label, and virtual circuit under the user-created namespace in the cluster, and traverse the inference service virtual circuit to query its label, so that the virtual circuit corresponds to the container set, and store the virtual circuit name, inference service name, the node where the inference service is located, and the Internet protocol of the corresponding container set in the time series database;

[0143] The container performance collection script is used to obtain the container corresponding to the container set under the user-created namespace in the cluster, and traverse the container to query the internal environment variables to obtain the GPU performance data;

[0144] The deployment replica collection script is used to collect the expected number of replicas and the number of ready replicas of deployments, replica sets, ready replicas, background support service sets, and container sets;

[0145] The service request duration collection script is used to record the duration of external request data after the inference service is deployed;

[0146] The script for collecting the number of service requests is used to collect the Internet protocol records of the container set corresponding to the inference service deployment node, and call the Prometheus lightweight interface to query the number of inference service requests and the incremental value of the number of requests;

[0147] The CPU performance collection script is used to obtain CPU performance indicator data;

[0148] The memory performance collection script is used to obtain memory performance indicator data;

[0149] The network performance collection script is used to obtain network performance indicator data.

[0150] In this embodiment, analyzing and calculating the target monitoring data to obtain performance data corresponding to the target indicator includes:

[0151] Analyze the CPU performance indicator data to obtain the CPU usage value corresponding to each moment in the CPU core usage time, and obtain the CPU usage rate (container_CPU_usage_seconds_total) of the first time period by dividing the difference of the CPU usage values ​​in the first time period range within the CPU core usage time by the first time period;

[0152] The upper limit of the number of cores allocated to the CPU of the current container is calculated through container_spec_CPU_quota / container_spec_CPU_period, where container_spec_CPU_quota is the container CPU quota and container_spec_CPU_period is the container CPU period.

[0153] Calculate CPU usage by sum(rate(container_CPU_usage_seconds_total)) / sum(container_spec_CPU_quota / container_spec_CPU_period);

[0154] The total occupancy of all CPU cores is divided by the total number of occupied cores to obtain the CPU utilization of each core.

[0155] In this embodiment, analyzing and calculating the target monitoring data to obtain performance data corresponding to the target indicator includes:

[0156] Get the number of used bytes of container memory in the container set container_memory_working_set_bytes, and sum the number of used bytes of containers in the container set sum(container_memory_working_set_bytes);

[0157] Get the maximum byte limit of container memory in the container set, and determine whether the maximum byte limit of container memory has a zero value. If not, sum the maximum byte limits of container memory in the container set. If there is a zero value, get the total container memory machine_memory_bytes.

[0158] In response to the container memory maximum byte limit not having a zero value, the memory utilization is calculated by sum(container_memory_working_set_bytes) / sum(container_spec_memory_limit_bytes);

[0159] In response to the container memory maximum byte limit having a zero value, the memory utilization is calculated by sum(container_memory_working_set_bytes) / machine_memory_bytes.

[0160] In this embodiment, analyzing and calculating the target monitoring data to obtain performance data corresponding to the target indicator includes:

[0161] Obtain the number of network input bytes and the number of network output bytes to form the number of network input and output bytes corresponding to each moment;

[0162] The network input and output rate is obtained by dividing the difference in the number of network input and output bytes within the second time range by the second time and then taking the time derivative.

[0163] In the above-mentioned inference service monitoring device, by setting multiple data collection scripts in the nodes in the cluster for collecting monitoring data of service monitoring indicators, and setting the plug-in-driven server agent component to periodically execute the execution task of the data collection script to collect monitoring data, the target monitoring data expected to be monitored can be directly obtained, and it is set to create a target inference service when the monitoring data collection command of the target indicator is received, and the correspondence between the virtual line of the target inference service and the container set created by the target inference service is obtained. The target monitoring data can be quickly retrieved based on the corresponding relationship without the need for multi-layer retrieval and search, and the corresponding data can be directly retrieved and viewed, thereby improving the search efficiency of the target monitoring data.

[0164] For the specific definition of the reasoning service monitoring device, please refer to the definition of the reasoning service monitoring method above, which will not be repeated here. Each module in the above-mentioned reasoning service monitoring device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0165] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0166] Multiple data collection scripts are set up on the nodes in the cluster. The data collection scripts are used to collect monitoring data of service monitoring indicators.

[0167] Install the plug-in driven server agent component on the node, set the execution task of the data collection script in the configuration file of the plug-in driven server agent component, and the plug-in driven server agent component periodically executes the execution task of the data collection script to collect monitoring data;

[0168] In response to receiving a monitoring data collection command for a target indicator, a target reasoning service is created, a correspondence between a virtual line of the target reasoning service and a container set created by the target reasoning service is obtained, and based on the correspondence, a plug-in driven server agent component executes a data collection script corresponding to the target reasoning service to obtain target monitoring data;

[0169] Analyze and calculate the target monitoring data to obtain the performance data corresponding to the target indicator, and display the performance data in time series.

[0170] In one embodiment, the computer program further implements the following steps when executed by a processor:

[0171] Configure a time series database to store the collected monitoring data;

[0172] In response to a command to collect monitoring data for executing a target indicator, target monitoring data corresponding to the target indicator is queried and acquired from a time series database.

[0173] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0174] Obtaining the correspondence between the virtual circuit of the target reasoning service and the container set created by the target reasoning service, and executing the data collection script corresponding to the target reasoning service through the plug-in driving server agent component based on the correspondence to obtain the target monitoring data includes:

[0175] Setting a first association tag for the name of the inference service and the virtual circuit of the inference service, and setting a second association tag for the name of the inference service and the container set created by the inference service;

[0176] When creating a target reasoning service, associating a virtual circuit of the target reasoning service with a container set created by the target reasoning service using the first association tag and the second association tag;

[0177] The plug-in driven server agent component controls the running of the data collection script corresponding to the target reasoning service, and obtains the target monitoring data according to the correspondence between the virtual line of the target reasoning service and the container set created by the target reasoning service.

[0178] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0179] Setting up multiple data collection scripts on nodes in a cluster includes:

[0180] Set up data collection scripts including collecting virtual line description script, collecting container performance script, collecting deployment copy script, collecting service request duration script, collecting service request quantity script, collecting CPU performance script, collecting memory performance script, and collecting network performance script;

[0181] The collection virtual circuit description script is used to obtain the container set, container set label, and virtual circuit under the user-created namespace in the cluster, and traverse the inference service virtual circuit to query its label, so that the virtual circuit corresponds to the container set, and store the virtual circuit name, inference service name, the node where the inference service is located, and the Internet protocol of the corresponding container set in the time series database;

[0182] The container performance collection script is used to obtain the container corresponding to the container set under the user-created namespace in the cluster, and traverse the container to query the internal environment variables to obtain the GPU performance data;

[0183] The deployment replica collection script is used to collect the expected number of replicas and the number of ready replicas of deployments, replica sets, ready replicas, background support service sets, and container sets;

[0184] The service request duration collection script is used to record the duration of external request data after the inference service is deployed;

[0185] The script for collecting the number of service requests is used to collect the Internet protocol records of the container set corresponding to the inference service deployment node, and call the Prometheus lightweight interface to query the number of inference service requests and the incremental value of the number of requests;

[0186] The CPU performance collection script is used to obtain CPU performance indicator data;

[0187] The memory performance collection script is used to obtain memory performance indicator data;

[0188] The network performance collection script is used to obtain network performance indicator data.

[0189] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0190] Analyze and calculate the target monitoring data to obtain the performance data corresponding to the target indicators, including:

[0191] Analyze the CPU performance indicator data to obtain the CPU usage value corresponding to each moment in the CPU core usage time, and obtain the CPU usage rate (container_CPU_usage_seconds_total) of the first time period by dividing the difference of the CPU usage values ​​in the first time period range within the CPU core usage time by the first time period;

[0192] The upper limit of the number of cores allocated to the CPU of the current container is calculated through container_spec_CPU_quota / container_spec_CPU_period, where container_spec_CPU_quota is the container CPU quota and container_spec_CPU_period is the container CPU period.

[0193] Calculate CPU usage by sum(rate(container_CPU_usage_seconds_total)) / sum(container_spec_CPU_quota / container_spec_CPU_period);

[0194] The total occupancy of all CPU cores is divided by the total number of occupied cores to obtain the CPU utilization of each core.

[0195] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0196] Analyze and calculate the target monitoring data to obtain the performance data corresponding to the target indicators, including:

[0197] Get the number of used bytes of container memory in the container set container_memory_working_set_bytes, and sum the number of used bytes of containers in the container set sum(container_memory_working_set_bytes);

[0198] Get the maximum byte limit of container memory in the container set, and determine whether the maximum byte limit of container memory has a zero value. If not, sum the maximum byte limits of container memory in the container set. If there is a zero value, get the total container memory machine_memory_bytes.

[0199] In response to the container memory maximum byte limit not having a zero value, the memory utilization is calculated by sum(container_memory_working_set_bytes) / sum(container_spec_memory_limit_bytes);

[0200] In response to the container memory maximum byte limit having a zero value, the memory utilization is calculated by sum(container_memory_working_set_bytes) / machine_memory_bytes.

[0201] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0202] Analyze and calculate the target monitoring data to obtain the performance data corresponding to the target indicators, including:

[0203] Obtain the number of network input bytes and the number of network output bytes to form the number of network input and output bytes corresponding to each moment;

[0204] The network input and output rate is obtained by dividing the difference in the number of network input and output bytes within the second time range by the second time and then taking the time derivative.

[0205] For the specific limitations on the steps implemented when the computer program is executed by the processor, please refer to the limitations on the method of reasoning service monitoring above, which will not be repeated here.

[0206] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store reasoning service monitoring data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a reasoning service monitoring method is implemented.

[0207] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0208] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0209] Multiple data collection scripts are set up on the nodes in the cluster. The data collection scripts are used to collect monitoring data of service monitoring indicators.

[0210] Install the plug-in driven server agent component on the node, set the execution task of the data collection script in the configuration file of the plug-in driven server agent component, and the plug-in driven server agent component periodically executes the execution task of the data collection script to collect monitoring data;

[0211] In response to receiving a monitoring data collection command for a target indicator, a target reasoning service is created, a correspondence between a virtual line of the target reasoning service and a container set created by the target reasoning service is obtained, and based on the correspondence, a plug-in driven server agent component executes a data collection script corresponding to the target reasoning service to obtain target monitoring data;

[0212] Analyze and calculate the target monitoring data to obtain the performance data corresponding to the target indicator, and display the performance data in time series.

[0213] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0214] Configure a time series database to store the collected monitoring data;

[0215] In response to a command to collect monitoring data for executing a target indicator, target monitoring data corresponding to the target indicator is queried and acquired from a time series database.

[0216] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0217] Obtaining the correspondence between the virtual circuit of the target reasoning service and the container set created by the target reasoning service, and executing the data collection script corresponding to the target reasoning service through the plug-in driving server agent component based on the correspondence to obtain the target monitoring data includes:

[0218] Setting a first association tag for the name of the inference service and the virtual circuit of the inference service, and setting a second association tag for the name of the inference service and the container set created by the inference service;

[0219] When creating a target reasoning service, associating a virtual circuit of the target reasoning service with a container set created by the target reasoning service using the first association tag and the second association tag;

[0220] The plug-in driven server agent component controls the running of the data collection script corresponding to the target reasoning service, and obtains the target monitoring data according to the correspondence between the virtual line of the target reasoning service and the container set created by the target reasoning service.

[0221] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0222] Setting up multiple data collection scripts on nodes in a cluster includes:

[0223] Set up data collection scripts including collecting virtual line description script, collecting container performance script, collecting deployment copy script, collecting service request duration script, collecting service request quantity script, collecting CPU performance script, collecting memory performance script, and collecting network performance script;

[0224] The collection virtual circuit description script is used to obtain the container set, container set label, and virtual circuit under the user-created namespace in the cluster, and traverse the inference service virtual circuit to query its label, so that the virtual circuit corresponds to the container set, and store the virtual circuit name, inference service name, the node where the inference service is located, and the Internet protocol of the corresponding container set in the time series database;

[0225] The container performance collection script is used to obtain the container corresponding to the container set under the user-created namespace in the cluster, and traverse the container to query the internal environment variables to obtain the GPU performance data;

[0226] The deployment replica collection script is used to collect the expected number of replicas and the number of ready replicas of deployments, replica sets, ready replicas, background support service sets, and container sets;

[0227] The service request duration collection script is used to record the duration of external request data after the inference service is deployed;

[0228] The script for collecting the number of service requests is used to collect the Internet protocol records of the container set corresponding to the inference service deployment node, and call the Prometheus lightweight interface to query the number of inference service requests and the incremental value of the number of requests;

[0229] The CPU performance collection script is used to obtain CPU performance indicator data;

[0230] The memory performance collection script is used to obtain memory performance indicator data;

[0231] The network performance collection script is used to obtain network performance indicator data.

[0232] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0233] Analyze and calculate the target monitoring data to obtain the performance data corresponding to the target indicators, including:

[0234] Analyze the CPU performance indicator data to obtain the CPU usage value corresponding to each moment in the CPU core usage time, and obtain the CPU usage rate (container_CPU_usage_seconds_total) of the first time period by dividing the difference of the CPU usage values ​​in the first time period range within the CPU core usage time by the first time period;

[0235] The upper limit of the number of cores allocated to the CPU of the current container is calculated through container_spec_CPU_quota / container_spec_CPU_period, where container_spec_CPU_quota is the container CPU quota and container_spec_CPU_period is the container CPU period.

[0236] Calculate CPU usage by sum(rate(container_CPU_usage_seconds_total)) / sum(container_spec_CPU_quota / container_spec_CPU_period);

[0237] The total occupancy of all CPU cores is divided by the total number of occupied cores to obtain the CPU utilization of each core.

[0238] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0239] Analyze and calculate the target monitoring data to obtain the performance data corresponding to the target indicators, including:

[0240] Get the number of used bytes of container memory in the container set container_memory_working_set_bytes, and sum the number of used bytes of containers in the container set sum(container_memory_working_set_bytes);

[0241] Get the maximum byte limit of container memory in the container set, and determine whether the maximum byte limit of container memory has a zero value. If not, sum the maximum byte limits of container memory in the container set. If there is a zero value, get the total container memory machine_memory_bytes.

[0242] In response to the container memory maximum byte limit not having a zero value, the memory utilization is calculated by sum(container_memory_working_set_bytes) / sum(container_spec_memory_limit_bytes);

[0243] In response to the container memory maximum byte limit having a zero value, the memory utilization is calculated by sum(container_memory_working_set_bytes) / machine_memory_bytes.

[0244] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0245] Analyze and calculate the target monitoring data to obtain the performance data corresponding to the target indicators, including:

[0246] Obtain the number of network input bytes and the number of network output bytes to form the number of network input and output bytes corresponding to each moment;

[0247] The network input and output rate is obtained by dividing the difference in the number of network input and output bytes within the second time range by the second time and then taking the time derivative.

[0248] For the specific limitations on the steps implemented when the processor executes the computer program, please refer to the limitations on the method of reasoning service monitoring above, which will not be repeated here.

[0249] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0250] Multiple data collection scripts are set up on the nodes in the cluster. The data collection scripts are used to collect monitoring data of service monitoring indicators.

[0251] Install the plug-in driven server agent component on the node, set the execution task of the data collection script in the configuration file of the plug-in driven server agent component, and the plug-in driven server agent component periodically executes the execution task of the data collection script to collect monitoring data;

[0252] In response to receiving a monitoring data collection command for a target indicator, a target reasoning service is created, a correspondence between a virtual line of the target reasoning service and a container set created by the target reasoning service is obtained, and based on the correspondence, a plug-in driven server agent component executes a data collection script corresponding to the target reasoning service to obtain target monitoring data;

[0253] Analyze and calculate the target monitoring data to obtain the performance data corresponding to the target indicator, and display the performance data in time series.

[0254] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0255] Configure a time series database to store the collected monitoring data;

[0256] In response to a command to collect monitoring data for executing a target indicator, target monitoring data corresponding to the target indicator is queried and acquired from a time series database.

[0257] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0258] Obtaining the correspondence between the virtual circuit of the target reasoning service and the container set created by the target reasoning service, and executing the data collection script corresponding to the target reasoning service through the plug-in driving server agent component based on the correspondence to obtain the target monitoring data includes:

[0259] Setting a first association tag for the name of the inference service and the virtual circuit of the inference service, and setting a second association tag for the name of the inference service and the container set created by the inference service;

[0260] When creating a target reasoning service, associating a virtual circuit of the target reasoning service with a container set created by the target reasoning service using the first association tag and the second association tag;

[0261] The plug-in driven server agent component controls the running of the data collection script corresponding to the target reasoning service, and obtains the target monitoring data according to the correspondence between the virtual line of the target reasoning service and the container set created by the target reasoning service.

[0262] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0263] Setting up multiple data collection scripts on nodes in a cluster includes:

[0264] Set up data collection scripts including collecting virtual line description script, collecting container performance script, collecting deployment copy script, collecting service request duration script, collecting service request quantity script, collecting CPU performance script, collecting memory performance script, and collecting network performance script;

[0265] The collection virtual circuit description script is used to obtain the container set, container set label, and virtual circuit under the user-created namespace in the cluster, and traverse the inference service virtual circuit to query its label, so that the virtual circuit corresponds to the container set, and store the virtual circuit name, inference service name, the node where the inference service is located, and the Internet protocol of the corresponding container set in the time series database;

[0266] The container performance collection script is used to obtain the container corresponding to the container set under the user-created namespace in the cluster, and traverse the container to query the internal environment variables to obtain the GPU performance data;

[0267] The deployment replica collection script is used to collect the expected number of replicas and the number of ready replicas of deployments, replica sets, ready replicas, background support service sets, and container sets;

[0268] The service request duration collection script is used to record the duration of external request data after the inference service is deployed;

[0269] The script for collecting the number of service requests is used to collect the Internet protocol records of the container set corresponding to the inference service deployment node, and call the Prometheus lightweight interface to query the number of inference service requests and the incremental value of the number of requests;

[0270] The CPU performance collection script is used to obtain CPU performance indicator data;

[0271] The memory performance collection script is used to obtain memory performance indicator data;

[0272] The network performance collection script is used to obtain network performance indicator data.

[0273] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0274] Analyze and calculate the target monitoring data to obtain the performance data corresponding to the target indicators, including:

[0275] Analyze the CPU performance indicator data to obtain the CPU usage value corresponding to each moment in the CPU core usage time, and obtain the CPU usage rate (container_CPU_usage_seconds_total) of the first time period by dividing the difference of the CPU usage values ​​in the first time period range within the CPU core usage time by the first time period;

[0276] The upper limit of the number of cores allocated to the CPU of the current container is calculated through container_spec_CPU_quota / container_spec_CPU_period, where container_spec_CPU_quota is the container CPU quota and container_spec_CPU_period is the container CPU period.

[0277] Calculate CPU usage by sum(rate(container_CPU_usage_seconds_total)) / sum(container_spec_CPU_quota / container_spec_CPU_period);

[0278] The total occupancy of all CPU cores is divided by the total number of occupied cores to obtain the CPU utilization of each core.

[0279] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0280] Analyze and calculate the target monitoring data to obtain the performance data corresponding to the target indicators, including:

[0281] Get the number of used bytes of container memory in the container set container_memory_working_set_bytes, and sum the number of used bytes of containers in the container set sum(container_memory_working_set_bytes);

[0282] Get the maximum byte limit of container memory in the container set, and determine whether the maximum byte limit of container memory has a zero value. If not, sum the maximum byte limits of container memory in the container set. If there is a zero value, get the total container memory machine_memory_bytes.

[0283] In response to the container memory maximum byte limit not having a zero value, the memory utilization is calculated by sum(container_memory_working_set_bytes) / sum(container_spec_memory_limit_bytes);

[0284] In response to the container memory maximum byte limit having a zero value, the memory utilization is calculated by sum(container_memory_working_set_bytes) / machine_memory_bytes.

[0285] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0286] Analyze and calculate the target monitoring data to obtain the performance data corresponding to the target indicators, including:

[0287] Obtain the number of network input bytes and the number of network output bytes to form the number of network input and output bytes corresponding to each moment;

[0288] The network input and output rate is obtained by dividing the difference in the number of network input and output bytes within the second time range by the second time and then taking the time derivative.

[0289] For the specific limitations on the steps implemented when the computer program is executed by the processor, please refer to the limitations on the method of reasoning service monitoring above, which will not be repeated here.

[0290] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0291] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0292] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for monitoring an inference service, characterized in that: include: Multiple data collection scripts are set in the nodes of the cluster, and the data collection scripts are used to collect monitoring data of service monitoring indicators; Installing a plug-in driven server agent component on the node, setting the execution task of the data collection script in the configuration file of the plug-in driven server agent component, and the plug-in driven server agent component periodically executing the execution task of the data collection script to collect monitoring data; In response to receiving a monitoring data collection command for a target indicator, a target reasoning service is created, a correspondence between a virtual line of the target reasoning service and a container set created by the target reasoning service is obtained, and based on the correspondence, a data collection script corresponding to the target reasoning service is executed by the plug-in driving server agent component to obtain target monitoring data; The target monitoring data is analyzed and calculated to obtain performance data corresponding to the target indicator, and the performance data is displayed in time series.

2. The inference service monitoring method according to claim 1, characterized in that: The method further comprises: Configure a time series database to store the collected monitoring data; In response to a command for collecting monitoring data of a target indicator, the target monitoring data corresponding to the target indicator is queried and acquired from the time series database.

3. The inference service monitoring method according to claim 1, characterized in that: The step of obtaining a correspondence between a virtual circuit of the target reasoning service and a container set created by the target reasoning service, and obtaining target monitoring data by executing a data collection script corresponding to the target reasoning service through the plug-in driven server agent component based on the correspondence comprises: Setting a first association tag between the name of the inference service and a virtual circuit of the inference service, and setting a second association tag between the name of the inference service and a container set created by the inference service; When creating a target reasoning service, associating a virtual circuit of the target reasoning service with a container set created by the target reasoning service using the first association tag and the second association tag; The plug-in driven server agent component controls the execution of the data collection script corresponding to the target reasoning service, and obtains the target monitoring data according to the correspondence between the virtual line of the target reasoning service and the container set created by the target reasoning service.

4. The inference service monitoring method according to claim 3, characterized in that: The step of setting multiple data collection scripts on nodes in the cluster includes: Setting the data collection script includes collecting a virtual line description script, collecting a container performance script, collecting a deployment copy script, collecting a service request duration script, collecting a service request quantity script, collecting a CPU performance script, collecting a memory performance script, and collecting a network performance script; The virtual line description acquisition script is used to obtain the container set, container set label, and virtual line under the namespace created by the user in the cluster, and traverse and query the label of the inference service virtual line to make the virtual line correspond to the container set, and store the virtual line name, inference service name, the node where the inference service is located, and the Internet protocol of the corresponding container set in the time series database; The collection container performance script is used to obtain the container corresponding to the container set under the namespace created by the user in the cluster, and traverse the container to query the internal environment variables to obtain the graphics processor performance data; The deployment replica collection script is used to collect the expected number of replicas and the number of ready replicas of deployment, replica set, ready replica, background support service set, and container set; The collection service request duration script is used to record the duration of the data called by the external request after the inference service is deployed; The script for collecting the number of service requests is used to collect Internet protocol records of the container set corresponding to the inference service deployment node, and call the Prometheus lightweight interface to query the number of requests for the inference service and the incremental value of the number of requests; The CPU performance acquisition script is used to obtain CPU performance indicator data; The memory performance acquisition script is used to obtain memory performance indicator data; The network performance acquisition script is used to obtain network performance indicator data.

5. The inference service monitoring method according to claim 4, characterized in that: The analyzing and calculating the target monitoring data to obtain the performance data corresponding to the target indicator includes: Analyze the CPU performance indicator data to obtain the CPU usage value corresponding to each moment within the CPU core usage duration, and obtain the CPU usage rate (container_CPU_usage_seconds_total) of the first duration by dividing the difference of the CPU usage values ​​within a first duration range within the CPU core usage duration by the first duration; The upper limit of the number of cores allocated to the CPU of the current container is calculated through container_spec_CPU_quota / container_spec_CPU_period, where container_spec_CPU_quota is the container CPU quota and container_spec_CPU_period is the container CPU period. The CPU utilization is calculated by sum(rate(container_CPU_usage_seconds_total)) / sum(container_spec_CPU_quota / container_spec_CPU_period), where sum is the sum of the corresponding values ​​of all containers in the container set when there are multiple containers in the container set; The total occupancy of all CPU cores is divided by the total number of occupied cores to obtain the CPU utilization of each core.

6. The inference service monitoring method according to claim 4, characterized in that: The analyzing and calculating the target monitoring data to obtain the performance data corresponding to the target indicator includes: Get the number of used bytes of container memory in the container set container_memory_working_set_bytes, and sum the number of used bytes of containers in the container set sum(container_memory_working_set_bytes); Get the maximum byte limit of container memory in the container set, determine whether the maximum byte limit of container memory has a zero value, if not, sum the maximum byte limit of container memory in the container set, if there is a zero value, get the total container memory machine_memory_bytes; In response to the container memory maximum byte limit not having a zero value, calculating the memory utilization by sum(container_memory_working_set_bytes) / sum(container_spec_memory_limit_bytes); In response to the container memory maximum byte limit having a zero value, the memory utilization is calculated by sum(container_memory_working_set_bytes) / machine_memory_bytes.

7. The inference service monitoring method according to claim 4, characterized in that: The analyzing and calculating the target monitoring data to obtain the performance data corresponding to the target indicator includes: Obtain the number of network input bytes and the number of network output bytes to form the number of network input and output bytes corresponding to each moment; The network input and output rate is obtained by dividing the difference in the number of network input and output bytes within the second time range by the second time and then taking the time derivative.

8. A reasoning service monitoring device, characterized in that: The device comprises: A collection module, used to set multiple data collection scripts in the nodes in the cluster, and the data collection scripts are used to collect monitoring data of service monitoring indicators; A monitoring service module is used to install a plug-in driven server agent component on the node, set the execution task of the data collection script in the configuration file of the plug-in driven server agent component, and the plug-in driven server agent component regularly executes the execution task of the data collection script to collect monitoring data; in response to receiving a monitoring data collection command of a target indicator, create a target reasoning service, obtain a correspondence between a virtual line of the target reasoning service and a container set created by the target reasoning service, and execute the data collection script corresponding to the target reasoning service through the plug-in driven server agent component based on the correspondence to obtain target monitoring data; The platform page display module is used to analyze and calculate the target monitoring data to obtain the performance data corresponding to the target indicator, and display the performance data in time series.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images