Idle service management method and device for GPU inference service
By monitoring and automatically counting the number of calls of GPU inference services, identifying and managing idle services, the problem of low GPU resource utilization is solved, and efficient and stable GPU inference service management is achieved.
Patent Information
- Application Number
- CN202510296264.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-12
- Publication Date
- 2025-06-27
AI Technical Summary
In the prior art, when managing GPU inference services, GPU resources cannot be effectively utilized, resulting in low resource utilization and low efficiency and risk of business interruption by manually managing idle services.
By monitoring the call status of GPU inference services in the K8S cluster, automatically count the service call volume, identify idle services, and automatically deactivate or enable GPU inference services after detecting identification information in the target list, accurate idle service management is achieved.
It improves the utilization rate of GPU resources, reduces the risk of business interruption, realizes automated idle service management, and improves management efficiency.
Smart Images
Figure CN120216131A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and particularly to a method and device for managing idle services for GPU inference services. Background Art
[0002] With the rapid development of artificial intelligence technology, enterprises' demand for AI models (i.e., GPU inference services) is increasing day by day. Especially in the inference environment (i.e., the test environment), the effective management of GPU (Graphics Processing Unit) resources has become a key issue. For example, when an enterprise's inference environment has nearly 100 GPU resources, due to resource limitations, when managing resources through a k8s cluster, the GPU resources cannot support the deployment and operation of nearly 200 GPU inference services simultaneously, resulting in low utilization rate of GPU resources and inability to meet the growing model test requirements.
[0003] To solve the above problems, traditional management methods usually identify idle GPU inference services (i.e., GPU inference services with no or very few call volumes within a specific time period. This service does not perform any computing tasks but occupies a large amount of GPU resources, resulting in a lack of sufficient resource support for the deployment of new services) manually, and manually take offline (i.e., deactivate) the idle GPU inference services, and manually bring online (i.e., activate) the GPU inference services when needed to improve the utilization rate of GPU resources. However, managing idle GPU inference services manually not only has low management efficiency but also increases the risk of business interruption. Especially when an idle GPU inference service needs to be brought online again, business interruption may occur due to manual operation delays or errors. Summary of the Invention
[0004] Based on the above problems, this application provides a method and device for managing idle services for GPU inference services, aiming to achieve precise idle service management in an automated manner, improve the efficiency of managing idle services for GPU inference services, and reduce the risk of business interruption.
[0005] The embodiments of this application disclose the following technical solutions:
[0006] In the first aspect of this application, a method for managing idle services for GPU inference services is provided, characterized in that the method includes:
[0007] Monitoring the call status of GPU inference services deployed on each node in the K8S cluster;
[0008] Based on the call status, counting the service call volume of each of the GPU inference services within a first preset period;
[0009] If it is determined that the GPU inference service is an idle service based on the service call volume, then use this GPU inference service as the target GPU inference service;
[0010] Detect whether the identification information of the target GPU inference service exists in the target list; the target list includes the identification information corresponding to each of multiple GPU inference services that are prohibited from being deactivated;
[0011] If the identification information of the target GPU inference service does not exist in the target list, then deactivate the target GPU inference service.
[0012] In an alternative embodiment, the method for managing idle services for GPU inference services further includes:
[0013] Read the service deactivation records within a second preset period; the service deactivation records are used to record the information of all deactivated GPU inference services in the K8S cluster;
[0014] If the information of the deactivated GPU inference service exists in the service deactivation records, then obtain the abnormal call logs from the first preset storage area; the abnormal call logs include the information of the GPU inference services with call failures;
[0015] If it is detected that the identification information of the target GPU inference service exists in the abnormal call logs, then activate the target GPU inference service.
[0016] In an alternative embodiment, the method for managing idle services for GPU inference services further includes:
[0017] Obtain the historical running status information of the target GPU inference service; the historical running status information includes the deactivation information and activation information of the target GPU inference service within a historical period;
[0018] Input the historical running status information into a prediction model to obtain the prediction result output by the prediction model; the prediction result includes the future activation time of the target GPU inference service, and the prediction model is a machine learning model obtained through model training based on the historical running status information of multiple idle GPU inference services and their future actual activation times, and the identification information of each of the idle GPU inference services is not in the target list;
[0019] Activate the target GPU inference service according to the activation time.
[0020] In an alternative embodiment, the step of counting the service call volume of each of the GPU inference services within a first preset period based on the call status includes:
[0021] Check whether there is a service call volume of the GPU inference service in the second preset storage area;
[0022] If there is a service call volume of the GPU inference service in the second preset storage area, directly count the service call volumes of each GPU inference service within the first preset period;
[0023] If there is no service call volume of the GPU inference service in the second preset storage area, obtain the gateway logs of each GPU inference service within the first preset period, and parse the gateway logs to obtain the service call volumes of each GPU inference service.
[0024] In an alternative embodiment, the step of, if it is determined that the GPU inference service is an idle service based on the service call volume, taking the GPU inference service as the target GPU inference service, includes:
[0025] Judge whether the service call volume of each GPU inference service is less than a preset threshold;
[0026] If the service call volume of the GPU inference service is greater than or equal to the preset threshold, determine that the GPU inference service is a non-idle service;
[0027] If the service call volume of the GPU inference service is less than the preset threshold, determine that the GPU inference service is the idle service, and take the GPU inference service as the target GPU inference service.
[0028] In an alternative embodiment, after deactivating the target GPU inference service, the method for managing idle services for GPU inference services further includes:
[0029] Generate an alarm prompt message according to the identification information of the target GPU inference service; the alarm prompt message is used to prompt the user that the target GPU inference service has been deactivated;
[0030] Send the alarm prompt message to the user according to a preset alarm notification method.
[0031] In a second aspect of the present application, there is provided a device for managing idle services for GPU inference services, the device includes:
[0032] A monitoring module, configured to monitor the call status of GPU inference services deployed on each node in the K8S cluster;
[0033] A statistics module, configured to count the service call volumes of each GPU inference service within the first preset period based on the call status;
[0034] A determination module, configured to, if it is determined based on the service call volume that the GPU inference service is an idle service, use the GPU inference service as the target GPU inference service;
[0035] A detection module, configured to detect whether the identification information of the target GPU inference service exists in the target list; the target list includes the identification information corresponding to each of multiple GPU inference services that are prohibited from being deactivated;
[0036] A deactivation module, configured to deactivate the target GPU inference service if the identification information of the target GPU inference service does not exist in the target list.
[0037] Optionally, the idle service management device for the GPU inference service further includes:
[0038] A reading module, configured to read the service deactivation record within a second preset time period; the service deactivation record is used to record the information of all deactivated GPU inference services in the K8S cluster;
[0039] A log acquisition module, configured to, if the information of the deactivated GPU inference service exists in the service deactivation record, acquire the abnormal call log from the first preset storage area; the abnormal call log includes the information of the GPU inference service with a call failure;
[0040] An enabling module, configured to enable the target GPU inference service if it is detected that the identification information of the target GPU inference service exists in the abnormal call log.
[0041] In a third aspect of the present application, there is provided a computer-readable storage medium storing a computer program, which when run by a processor, implements the idle service management method for the GPU inference service described in the first aspect above.
[0042] In a fourth aspect of the present application, there is provided a processor for running a computer program, and when the computer program runs, it executes the idle service management method for the GPU inference service described in the first aspect above.
[0043] Compared with the prior art, the present application has the following beneficial effects:
[0044] In the technical solution of this application, first, the call status of the GPU inference services deployed on each node in the K8S cluster is monitored; then, based on the call status, the service call volume of each GPU inference service within a first preset period is counted; if a GPU inference service is determined to be an idle service based on the service call volume, then this GPU inference service is used as the target GPU inference service; then, it is detected whether the identification information of the target GPU inference service exists in the target list; where the target list includes the identification information corresponding to each of multiple GPU inference services that are prohibited from being deactivated; if the identification information of the target GPU inference service does not exist in the target list, then the target GPU inference service is deactivated.
[0045] In the technical solution of this application, by monitoring in real time the call status of the GPU inference services deployed on each node in the K8S cluster, it is possible to achieve the automatic statistics of the service call volume of each GPU inference service within a first preset period, and accurately identify the idle GPU inference services in the K8S cluster; then, when it is detected that the identification information of the target GPU inference service (i.e., the idle GPU inference service) does not exist in the target list, the target GPU inference service can be automatically deactivated, thus realizing precise idle service management in an automated manner, avoiding the problem of low management efficiency caused by manually managing idle GPU inference services, and further improving the efficiency of idle service management for GPU inference services.
[0046] In addition, through the target list mechanism, it is ensured that the GPU inference services that are prohibited from being deactivated will not be deactivated by mistake, avoiding the problem of service interruption caused by manually deactivating the GPU inference services in the target list by mistake, thereby reducing the risk of service interruption and ensuring the stable operation of the service. Description of the Drawings
[0047] In order to more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.
[0048] Figure 1 It is a flowchart of a method for managing idle services for GPU inference services provided by an embodiment of this application;
[0049] Figure 2 It is a flowchart of another method for managing idle services for GPU inference services provided by an embodiment of this application;
[0050] Figure 3Flow chart of the statistical process of the service call volume of a GPU inference service provided by an embodiment of the present application;
[0051] Figure 4 Flow chart of the automatic enabling process of a GPU inference service that has been taken offline provided by an embodiment of the present application;
[0052] Figure 5 Flow chart of another automatic enabling process of a GPU inference service that has been taken offline provided by an embodiment of the present application;
[0053] Figure 6 Schematic structural diagram of an idle service management device for a GPU inference service provided by an embodiment of the present application. Detailed implementation manners
[0054] As described above, currently, the traditional management method usually identifies idle GPU inference services (that is, GPU inference services with no or very few call volumes within a specific time period. This service does not perform any computing tasks but occupies a large amount of GPU resources, resulting in a lack of sufficient resources for the deployment of new services) manually, and manually takes offline (that is, deactivates) the idle GPU inference services, and manually brings them online (that is, enables) the GPU inference services when needed to improve the utilization rate of GPU resources. However, managing idle GPU inference services manually not only has low management efficiency but also increases the risk of service interruption. Especially when an idle GPU inference service needs to be brought back online, service interruption may occur due to manual operation delays or errors.
[0055] After research, the inventors proposed an idle service management method for GPU inference services. In this solution, first, the call status of the GPU inference services deployed on each node in the K8S cluster is monitored; then, based on the call status, the service call volume of each GPU inference service within a first preset time period is statistically calculated; if a GPU inference service is determined to be an idle service based on the service call volume, then this GPU inference service is used as the target GPU inference service; then, it is detected whether the identification information of the target GPU inference service exists in the target list; where the target list includes the identification information corresponding to each of multiple GPU inference services that are prohibited from being deactivated; if the identification information of the target GPU inference service does not exist in the target list, then the target GPU inference service is deactivated.
[0056] In the technical solution of this application, by monitoring the invocation status of GPU inference services deployed on each node in the K8S cluster in real time, it is possible to achieve the service invocation volume of each GPU inference service within the first preset period based on automatic statistics, and accurately identify the idle GPU inference services in the K8S cluster; then when the identification information of the target GPU inference service (i.e., the idle GPU inference service) is not present in the detected target list, the target GPU inference service can be automatically deactivated, thus achieving precise idle service management in an automated manner, avoiding the problem of low management efficiency caused by manually managing idle GPU inference services, and further improving the efficiency of idle service management for GPU inference services.
[0057] In addition, through the target list mechanism, it is ensured that the GPU inference services that are prohibited from being deactivated will not be mis-deactivated, avoiding the problem of service interruption caused by manually mis-deactivating the GPU inference services in the target list, thereby reducing the risk of service interruption and ensuring the stable operation of the service.
[0058] To enable those skilled in the art to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.
[0059] Keyword definitions:
[0060] Prometheus: Prometheus is an open-source system monitoring and alerting toolkit, which is specifically designed for reliability and simplicity, and is suitable for monitoring applications serving dynamic environments, especially applications based on the microservices architecture, with functions such as a flexible query language, an efficient storage mechanism, and powerful visualization capabilities.
[0061] GPU inference service: It refers to using a hardware device equipped with a graphics processing unit (GPU) to execute the inference process of a machine learning model. Different from the training stage, the inference stage mainly focuses on how to efficiently and quickly use the already trained model to predict or classify new data.
[0062] Pop method: Usually refers to an operation of removing elements from a data structure, especially in a Last In First Out (LIFO) data structure like a stack. The term "Pop" comes from the operation terminology of a stack, meaning to pop or remove the top element of the stack. When an element is "popped", it is removed from the stack and usually returned to the caller for further processing.
[0063] Method embodiment
[0064] This application provides an embodiment of a method for managing idle services for GPU inference services. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, it is only provided as an example order. In some cases, the steps shown or described can be executed in a different order than here.
[0065] Figure 1 Is a flowchart of a method for managing idle services for GPU inference services provided by an embodiment of this application, as Figure 1 shown, the method includes the following steps:
[0066] Step S101, monitor the call status of the GPU inference services deployed on each node in the K8S cluster.
[0067] In an optional embodiment, an idle service management system for GPU inference services can be the execution entity of the method for managing idle services for GPU inference services in an embodiment of this application. For the convenience of description, the idle service management system for GPU inference services will be simply referred to as the system hereinafter.
[0068] In an embodiment of this application, as Figure 2 shown, the K8S cluster (i.e., the GPU inference service resource pool) includes multiple nodes: node1, node2, node3....nodeN, and GPU inference services 1 to X are deployed on multiple nodes. The system can monitor the call status of the GPU inference services deployed on each node in the K8S cluster in real time and store the call status data of the GPU inference services in a second preset storage area (i.e., Figure 2 prometheus shown in).
[0069] Optionally, in order to achieve automatic collection of the call status data of the GPU inference services, in an embodiment of this application, as Figure 2 shown, the system can periodically collect and summarize the monitored call status data of the GPU inference services into the prometheus storage. Among them, the call status data of the GPU inference services can be as follows:
[0070] aimg_request_count{pod="pod_name",name="image-reg"} 1000
[0071] Among them, aimg_request_count is the data name; 1000 is the service call volume of the GPU inference service; {pod="pod_name",name="image-reg"} is the data description information, pod is the k8s container name corresponding to the GPU inference service, and name is the service name of the GPU inference service (i.e., the identification information of the GPU inference service).
[0072] In an alternative embodiment, if the system cannot directly collect the call status data of the GPU inference service, as Figure 2 shown, the system can use the log monitoring tool filebeat to read the gateway logs containing the call status data of the GPU inference service on the node in real time, and temporarily store the gateway logs in Redis (i.e., the first storage area) to monitor the call status of the GPU inference service, so as to provide a data basis for the subsequent automatic statistics of the service call volume.
[0073] In the embodiments of the present application, the call status of the GPU inference service includes: information such as the service call volume, response time, latency, and service call result (call failure or call success) of the GPU inference service.
[0074] It should be noted that by monitoring the call status of the GPU inference services deployed on each node in the K8S cluster in real time, it can provide a data basis for accurately identifying the idle GPU inference services in the K8S cluster based on the automatically counted service call volume of each GPU inference service within the first preset period.
[0075] Step S102, based on the call status, count the service call volume of each GPU inference service within the first preset period.
[0076] In the embodiments of the present application, the first preset period is a pre-set statistical period for the service call volume. For example, the service call volume from 00:00 on the 01st of xx month to 24:00 on the 01st of xx month (i.e., statistical within 24H) is counted.
[0077] In order to achieve precise idle service management in an automated manner and improve the efficiency of idle service management for the GPU inference service, in the embodiments of the present application, the system can automatically count the service call volume of each GPU inference service within the first preset period based on the call status of the GPU inference service. Specifically, refer to Figure 3 , Figure 3The figure is a flowchart of the statistical process of the service call volume of a GPU inference service provided by an embodiment of the present application. The process includes the following steps:
[0078] Step S1021: Detect whether there is a service call volume of the GPU inference service in the second preset storage area.
[0079] In the embodiment of the present application, the system can determine the automatic statistical method of the service call volume of each GPU inference service within the first preset time period by detecting whether there is a service call volume of the GPU inference service in the second preset storage area (i.e., Figure 2 prometheus therein), avoiding the problem of repeated statistics of the service call volume of the GPU inference service and improving the service management efficiency.
[0080] Step S1022: If there is a service call volume of the GPU inference service in the second preset storage area, directly count the service call volume of each GPU inference service within the first preset time period.
[0081] Step S1023: If there is no service call volume of the GPU inference service in the second preset storage area, obtain the gateway logs of each GPU inference service within the first preset time period, and parse the gateway logs to obtain the service call volume of each GPU inference service.
[0082] In the embodiment of the present application, when there is no service call volume of the GPU inference service in the second preset storage area, the system can obtain the gateway logs of each GPU inference service within the first preset time period from Figure 2 Redis (i.e., the first storage area) shown therein. The gateway logs include the service call volume data of the GPU inference service, so that by parsing the gateway logs, the service call volume of each GPU inference service can be obtained.
[0083] It should be noted that by automatically counting the service call volume of each GPU inference service within the first preset time period, the problem of low data statistical efficiency caused by manually counting the service call volume is avoided, the time consumption of service call volume statistics is reduced, and the management efficiency is improved.
[0084] Step S103: If it is determined that the GPU inference service is an idle service based on the service call volume, use the GPU inference service as the target GPU inference service.
[0085] In the embodiment of the present application, an idle service refers to a GPU inference service that is not currently processing any call requests or has a small number of call requests being processed.
[0086] In order to accurately identify the idle service, in the embodiment of the present application, the system can use a prometheus query statement (such asFigure 2 In the promql), based on the service call volume of the GPU inference service and the call volume threshold of the idle service (i.e., the preset threshold), it is determined whether the GPU inference service is an idle service. Among them, the promql can be as follows:
[0087] sumby(pod,name)(sum_over_time(aimg_request_count[{promPeriod}]) <= {alarmValue}.
[0088] Specifically, the system can determine whether the service call volume of each GPU inference service is less than the preset threshold; if the service call volume of the GPU inference service is greater than or equal to the preset threshold, the system can determine that the GPU inference service is a non-idle service; if the service call volume of the GPU inference service is less than the preset threshold, the system can determine that the GPU inference service is an idle service and use this GPU inference service as the target GPU inference service.
[0089] It should be noted that by determining whether the service call volume of each GPU inference service is less than the preset threshold, it is possible to accurately identify whether the GPU inference service is an idle service, avoiding the problem of low idle service management caused by manually judging idle services, and improving the efficiency of idle service management.
[0090] Step S104: Detect whether the identification information of the target GPU inference service exists in the target list.
[0091] In step S104, the target list includes the identification information corresponding to each of the multiple GPU inference services that are prohibited from being deactivated. Among them, the identification information can be information such as the name of the GPU inference service and the service ID.
[0092] To avoid the problem of service interruption caused by manually deactivating the GPU inference service that is prohibited from being deactivated, and to reduce the risk of service interruption, in the embodiment of this application, the system can determine whether to deactivate the target GPU inference service by detecting whether the identification information of the target GPU inference service exists in the target list, ensuring that the GPU inference service that is prohibited from being deactivated will not be misdeactivated and guaranteeing the stable operation of the service.
[0093] Step S105: If the identification information of the target GPU inference service does not exist in the target list, deactivate the target GPU inference service.
[0094] In the embodiment of the present application, when the system detects that the identification information of the target GPU inference service does not exist in the target list, it can automatically deactivate the target GPU inference service by calling the k8s API, thereby achieving accurate idle service management in an automated manner, avoiding the problem of low management efficiency caused by manually managing idle GPU inference services, and improving the efficiency of idle service management for GPU inference services.
[0095] To improve the user experience, after deactivating the target GPU inference service, the system can generate an alarm prompt message based on the identification information of the target GPU inference service, where the alarm prompt message is used to prompt the user that the target GPU inference service has been deactivated; then send the alarm prompt message to the user according to the alarm notification method.
[0096] In the embodiment of the present application, the alarm notification method can be Figure 2 the work group and email shown in, and the system can send the alarm prompt message to the user according to the address information of the work group and email to prompt the user that the target GPU inference service has been deactivated.
[0097] To further improve the efficiency of idle service management for GPU inference services, in the embodiment of the present application, the system can perform an idle service online monitoring task to automatically online process the offline idle services. Among them, the higher the task scheduling frequency of the idle service online monitoring task (for example, the idle service online monitoring task is executed once every 10 seconds to 60 seconds), the faster the reaction speed of the service to go online automatically. The idle service online monitoring task can specifically include Figure 4 the steps in, and the process includes the following steps:
[0098] Step S401, read the service deactivation records within the second preset time period.
[0099] In step S401, the service deactivation records include information on all deactivated GPU inference services in the K8S cluster.
[0100] In this embodiment, the service deactivation records record the detailed information on the deactivation of the target GPU inference service. By reading the service deactivation records within the second preset time period, it is possible to accurately determine whether there are GPU inference services in the K8S cluster that are in the deactivated state, avoiding waste of GPU resources caused by unnecessary enable commands when there are no GPU inference services in the deactivated state in the K8S cluster, thereby improving the efficiency of idle service management.
[0101] Step S402, if there is information on deactivated GPU inference services in the service deactivation records, obtain the abnormal call logs from the first preset storage area.
[0102] In step S402, the exception call log includes information about the GPU inference service whose call fails.
[0103] In the embodiment of the present application, since the offline GPU inference service cannot normally respond to the service call request, after the K8S cluster receives the service call request of the offline GPU inference service, an exception call log including information about the GPU inference service whose call fails will be generated. Thus, when there is information about the deactivated GPU inference service in the service deactivation record, the exception call log can be obtained from Redis (i.e., the first preset storage area) in the pop manner, and it can be determined whether the offline GPU inference service is redeployed.
[0104] Optionally, if there is no information about the deactivated GPU inference service in the service deactivation record, the system can determine that the exception call log in the first preset storage area is an unnecessary log file, and automatically clear the exception call log in the first preset storage area, avoiding the problem that it is difficult to accurately identify the deactivated GPU inference service due to the large number of exception call logs, reducing the identification time of the deactivated GPU inference service, and thus improving the efficiency of idle service management.
[0105] Step S403, if it is detected that the identification information of the target GPU inference service exists in the exception call log, enable the target GPU inference service.
[0106] In order to accurately identify whether the target GPU inference service is a deactivated idle service to improve the efficiency of idle service management, in the embodiment of the present application, the system can determine whether the target GPU inference service is redeployed by detecting whether the identification information of the target GPU inference service exists in the exception call log; if the identification information of the target GPU inference service exists in the exception call log, the system can determine that the target GPU inference service has been redeployed, and enable the target GPU inference service by calling the k8s API.
[0107] Optionally, if the identification information of the target GPU inference service does not exist in the exception call log, the system can determine that the target GPU inference service has not been redeployed, and prohibit enabling the target GPU inference service.
[0108] It should be noted that automatically starting the deactivated GPU inference service according to the exception call log eliminates the need for manual determination of whether the deactivated GPU inference service is redeployed again, thus avoiding the problem of low efficiency caused by enabling the deactivated GPU inference service manually, improving the efficiency of idle service management, and reducing the management cost.
[0109] To further improve the efficiency of idle service management, in the embodiments of this application, the system can predict the future enabling time of a deactivated GPU inference service (i.e., the target GPU inference service) through a pre-trained prediction model, and enable the target GPU inference service according to the enabling time. Specifically, referring to Figure 5 , this figure is a flowchart of another automatic enabling process of the offline GPU inference service provided by the embodiments of this application, and this process includes the following steps:
[0110] Step S501, obtain the historical running status information of the target GPU inference service.
[0111] In step S501, the historical running status information includes the deactivation information and enabling information of the target GPU inference service during the historical period.
[0112] Step S502, input the historical running status information into the prediction model to obtain the prediction result output by the prediction model; the prediction result includes the future enabling time of the target GPU inference service.
[0113] In step S502, the prediction model is a machine learning model obtained through model training based on the historical running status information and future actual enabling time of multiple idle GPU inference services, and the identification information of each idle GPU inference service is not in the target list.
[0114] For example, the system can obtain the deactivation information and enabling information of the target GPU inference service from 00:00 on March 1 to 24:00 on March 1 (i.e., the historical period), and then input this information into the prediction model. Through the prediction model, the enabling times of the target GPU inference service from 00:00 on March 2 to 24:00 on March 2 are predicted to be 8:00 on March 2 and 18:00 on March 2.
[0115] Step S503, enable the target GPU inference service according to the enabling time.
[0116] In the embodiments of this application, after the system obtains the future enabling time of the target GPU inference service, when the time reaches the enabling time, the system can automatically enable the target GPU inference service.
[0117] In an alternative embodiment, if the system determines that the target GPU inference service is called again before the enabling time, at this time, the system can execute the steps in the above steps S401-403 to automatically start the target GPU inference service.
[0118] It should be noted that by predicting the future enabling time of the target GPU inference service through a prediction model and enabling the target GPU inference service according to the enabling time, the enabling of the disabled GPU inference service is realized in an automated manner, avoiding the problem of low efficiency caused by enabling the disabled GPU inference service manually, improving the efficiency of idle service management, and reducing the management cost.
[0119] Optionally, after enabling the target GPU inference service, the system can generate an enabling prompt message based on the enabling information of the target GPU inference service and send the enabling prompt message to the user to prompt the user that the target GPU inference service has been re-enabled.
[0120] Through the idle service management method for GPU inference services provided by the embodiments of the present application, it is realized that by monitoring the call status of the GPU inference services deployed on each node in the K8S cluster in real time, the service call volume of each GPU inference service within a first preset period can be automatically counted, and the idle GPU inference services in the K8S cluster can be accurately identified; then when it is detected that the identification information of the target GPU inference service (i.e., the idle GPU inference service) does not exist in the target list, the target GPU inference service can be automatically disabled, realizing accurate idle service management in an automated manner, thus avoiding the problem of low management efficiency caused by managing idle GPU inference services manually, and improving the efficiency of idle service management for GPU inference services. Through the target list mechanism, it is ensured that the GPU inference services that are prohibited from being disabled will not be mis-disabled, avoiding the problem of service interruption caused by manually mis-disabling the GPU inference services in the target list, reducing the risk of service interruption, and ensuring the stable operation of the service.
[0121] In addition, through the idle service management method for GPU inference services provided by the embodiments of the present application, the enabling of the disabled GPU inference service can also be realized in an automated manner, without manually determining whether the disabled GPU inference service is called again, avoiding the problem of low efficiency caused by enabling the disabled GPU inference service manually, improving the efficiency of idle service management, and reducing the management cost.
[0122] Device embodiments
[0123] The embodiments of the present application provide an idle service management device for GPU inference services, where Figure 6 is a schematic structural diagram of an idle service management device for GPU inference services provided by the embodiments of the present application. As Figure 6 shown, the device includes: a monitoring module 11, a statistics module 12, a determination module 13, a detection module 14, and a disabling module 15. From Figure 6The connection relationships between several modules can be seen.
[0124] Among them, the monitoring module 11 is used to monitor the call status of the GPU inference services deployed on each node in the K8S cluster;
[0125] The statistics module 12 is used to count the service call volume of each GPU inference service within a first preset period based on the call status;
[0126] The determination module 13 is used to, if it is determined that the GPU inference service is an idle service based on the service call volume, use the GPU inference service as the target GPU inference service;
[0127] The detection module 14 is used to detect whether the identification information of the target GPU inference service exists in the target list; the target list includes the identification information corresponding to each of multiple GPU inference services that are prohibited from being deactivated;
[0128] The deactivation module 15 is used to, if the identification information of the target GPU inference service does not exist in the target list, deactivate the target GPU inference service.
[0129] Optionally, the idle service management device for the GPU inference service further includes: a reading module, a log acquisition module, and an enabling module.
[0130] Among them, the reading module is used to read the service deactivation record within a second preset period; the service deactivation record is used to record the information of all the deactivated GPU inference services in the K8S cluster;
[0131] The log acquisition module is used to, if the information of the deactivated GPU inference service exists in the service deactivation record, obtain the abnormal call log from the first preset storage area; the abnormal call log includes the information of the GPU inference service with a call failure;
[0132] The enabling module is used to, if it is detected that the identification information of the target GPU inference service exists in the abnormal call log, enable the target GPU inference service.
[0133] Optionally, the idle service management device for the GPU inference service further includes: a first acquisition module, a prediction module, and an execution module.
[0134] Among them, the first acquisition module is used to acquire the historical operation status information of the target GPU inference service; the historical operation status information includes the deactivation information and enabling information of the target GPU inference service within the historical period;
[0135] A prediction module, configured to input historical running status information into a prediction model to obtain a prediction result output by the prediction model; the prediction result includes the future enabling time of the target GPU inference service, and the prediction model is a machine learning model obtained through model training based on the historical running status information of multiple idle GPU inference services and their future actual enabling times, and the identification information of each idle GPU inference service is not in the target list;
[0136] An execution module, configured to enable the target GPU inference service according to the enabling time.
[0137] Optionally, the statistics module includes: a first detection unit, a statistics unit, and an analysis unit.
[0138] Wherein, the first detection unit is configured to detect whether there is a service call volume of the GPU inference service in the second preset storage area;
[0139] The statistics unit is configured to directly count the service call volume of each GPU inference service within the first preset period if there is a service call volume of the GPU inference service in the second preset storage area;
[0140] The analysis unit is configured to, if there is no service call volume of the GPU inference service in the second preset storage area, obtain the gateway logs of each GPU inference service within the first preset period and analyze the gateway logs to obtain the service call volume of each GPU inference service.
[0141] Optionally, the determination module includes: a judgment unit, a first determination unit, and a second determination unit.
[0142] Wherein, the judgment unit is configured to judge whether the service call volume of each GPU inference service is less than a preset threshold;
[0143] The first determination unit is configured to determine that the GPU inference service is a non-idle service if the service call volume of the GPU inference service is greater than or equal to the preset threshold;
[0144] The second determination unit is configured to determine that the GPU inference service is an idle service if the service call volume of the GPU inference service is less than the preset threshold, and use the GPU inference service as the target GPU inference service.
[0145] Optionally, the idle service management device for the GPU inference service further includes: a generation module and a sending module.
[0146] Wherein, the generation module is configured to generate an alarm prompt message according to the identification information of the target GPU inference service; the alarm prompt message is used to prompt the user that the target GPU inference service has been deactivated;
[0147] A sending module, configured to send an alarm prompt message to a user according to a preset alarm notification method.
[0148] Embodiment of the storage medium
[0149] An embodiment of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, some or all of the steps in the idle service management method for GPU inference services introduced in the foregoing method embodiment of the present application are implemented. The storage medium may be various media capable of storing program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc.
[0150] Embodiment of the processor
[0151] An embodiment of the present application provides a processor, which is used to run a program. When the program runs, some or all of the steps in the idle service management method for GPU inference services introduced in the foregoing method embodiment are executed.
[0152] It should be noted that the embodiments in this specification are all described in a progressive manner. For the same or similar parts between the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiments. The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components described as unit prompts may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative work.
[0153] The above is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.
Claims
1. A method for managing idle services for GPU inference services, characterized in that: include: Monitor the call status of the GPU inference service deployed on each node in the K8S cluster; Based on the calling status, counting the service calling volume of each of the GPU inference services within a first preset time period; If it is determined based on the service call volume that the GPU inference service is an idle service, then using the GPU inference service as a target GPU inference service; Detecting whether there is identification information of the target GPU inference service in the target list; the target list includes identification information corresponding to each of a plurality of GPU inference services that are prohibited from being deactivated; If the identification information of the target GPU inference service does not exist in the target list, the target GPU inference service is disabled.
2. The method according to claim 1, characterized in that The method further comprises: Reading a service deactivation record within a second preset time period; the service deactivation record is used to record information of all deactivated GPU inference services in the K8S cluster; If the service deactivation record contains information about the deactivated GPU inference service, obtaining an abnormal call log from the first preset storage area; the abnormal call log includes information about the GPU inference service that failed to be called; If it is detected that the identification information of the target GPU inference service exists in the abnormal call log, the target GPU inference service is enabled.
3. The method according to claim 1, characterized in that The method further comprises: Acquire historical running status information of the target GPU inference service; the historical running status information includes deactivation information and activation information of the target GPU inference service in a historical period; Input the historical running status information into a prediction model to obtain a prediction result output by the prediction model; the prediction result includes the activation time of the target GPU inference service in the future, the prediction model is a machine learning model obtained by model training based on the historical running status information of multiple idle GPU inference services and their actual activation time in the future, and the identification information of each of the idle GPU inference services is not in the target list; The target GPU inference service is enabled according to the enabling time.
4. The method according to claim 1, characterized in that: The counting of the service call volume of each of the GPU inference services within the first preset time period based on the call status includes: Detect whether there is a service call amount of the GPU inference service in the second preset storage area; If the service call volume of the GPU inference service exists in the second preset storage area, directly counting the service call volume of each GPU inference service in the first preset time period; If the service call volume of the GPU reasoning service does not exist in the second preset storage area, the gateway log of each of the GPU reasoning services within the first preset time period is obtained, and the gateway log is parsed to obtain the service call volume of each of the GPU reasoning services.
5. The method according to claim 1, characterized in that If it is determined based on the service call amount that the GPU inference service is an idle service, taking the GPU inference service as a target GPU inference service, comprising: Determine whether the service call volume of each of the GPU inference services is less than a preset threshold; If the service call volume of the GPU reasoning service is greater than or equal to the preset threshold, determining that the GPU reasoning service is a non-idle service; If the service call volume of the GPU reasoning service is less than the preset threshold, the GPU reasoning service is determined to be the idle service, and the GPU reasoning service is used as the target GPU reasoning service.
6. The method according to claim 1, characterized in that After deactivating the target GPU inference service, the method further includes: Generate warning prompt information according to the identification information of the target GPU inference service; the warning prompt information is used to prompt the user that the target GPU inference service has been deactivated; The alarm prompt information is sent to the user according to a preset alarm notification method.
7. An idle service management device for GPU inference service, characterized in that: include: The monitoring module is used to monitor the call status of the GPU inference service deployed on each node in the K8S cluster; A statistics module, configured to count the service call volume of each of the GPU inference services within a first preset time period based on the call status; a determination module, configured to use the GPU inference service as a target GPU inference service if it is determined that the GPU inference service is an idle service based on the service call amount; A detection module, used to detect whether the identification information of the target GPU inference service exists in the target list; the target list includes identification information corresponding to each of a plurality of GPU inference services that are prohibited from being deactivated; A deactivation module is used to deactivate the target GPU inference service if the identification information of the target GPU inference service does not exist in the target list.
8. The device according to claim 7, characterized in that The device also includes: A reading module, used to read a service deactivation record within a second preset time period; the service deactivation record is used to record information of all deactivated GPU inference services in the K8S cluster; A log acquisition module, configured to acquire an abnormal call log from the first preset storage area if the service deactivation record contains information about the deactivated GPU inference service; the abnormal call log includes information about the GPU inference service that failed to be called; The enabling module is used to enable the target GPU inference service if it is detected that the identification information of the target GPU inference service exists in the abnormal call log.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the idle service management method for GPU inference service according to any one of claims 1 to 6 is implemented.
10. A processor, characterized in that: Used to run a computer program, which, when running, executes the idle service management method for GPU inference service according to any one of claims 1 to 6.