Resource management methods, devices, servers and storage media

Abstract

This application discloses a resource management method, apparatus, server, and storage medium, belonging to the field of computer technology. The method includes: responding to a target instruction, determining an estimated remaining resource; if the estimated remaining resource is less than the reserved resource, acquiring data from multiple launched inference models across multiple dimensions to obtain multiple sets of data; the reserved resource refers to the resources sufficient for the system to operate normally; based on the multiple sets of data, reducing the number of instances of the multiple inference models and allocating resources for the execution of the target instruction. The method provided by this application can minimize the problem of slow system operation or even crashes due to resource scarcity, and can also reduce the failure to launch new inference models and expand inference model instances, thereby reducing inference request waiting time and improving inference efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a resource management method, apparatus, server, and storage medium. Background Technology

[0002] Model-as-a-service (MAS) refers to deploying multiple inference models or multiple versions of the same inference model on a unified server to provide model inference services to users, thereby meeting their model inference needs. However, as the number of inference models and instances deployed on the server increases, server resource consumption also increases, potentially leading to system slowdowns or even crashes. Furthermore, starting new inference models and expanding inference model instances may fail, resulting in prolonged inference request wait times and reduced inference efficiency. Therefore, a resource management method is urgently needed. Summary of the Invention

[0003] This application provides a resource management method, apparatus, server, and storage medium, which can avoid the problem of slow system operation or even crash due to resource shortage, reduce the failure to start new inference models and expand inference model instances, and improve inference efficiency. The technical solution is as follows:

[0004] On the one hand, a resource management method is provided, the method comprising:

[0005] In response to a target instruction, an estimated remaining resource is determined, wherein the target instruction indicates that an instance of a first inference model needs to be started or an instance of an already started second inference model needs to be expanded, and the estimated remaining resource refers to the estimated remaining resources after the target instruction is executed;

[0006] If the estimated remaining resources are less than the reserved resources, then data from multiple inference models that have been started are obtained in multiple dimensions to obtain multiple sets of data. The reserved resources refer to the resources that enable the system to run normally. The multiple inference models do not include the second inference model.

[0007] Based on the multiple sets of data, instances of the multiple inference models are reduced, and resources are allocated for the execution of the target instruction, wherein the estimated remaining resources after instance reduction are greater than or equal to the reserved resources.

[0008] Optionally, the target instruction indicates that the first inference model needs to be started at present; the determination of the estimated remaining resources includes:

[0009] Obtain the current remaining resources, the file size of the first inference model, and the number of the first instances, where the number of the first instances refers to the number of instances configured for the first inference model;

[0010] The estimated remaining resources are obtained based on the current remaining resources, the file size of the first inference model, and the number of the first instances.

[0011] Optionally, the target instruction indicates an instance where the second inference model needs to be expanded; determining the estimated remaining resources includes:

[0012] Get the current remaining resources, the memory usage of the second inference model and the number of second instances. The number of second instances refers to the number of instances that need to be added for the second inference model.

[0013] The estimated remaining resources are obtained based on the current remaining resources, the memory usage of the second inference model, and the number of the second instances.

[0014] Optionally, the step of reducing the number of instances of the multiple inference models based on the multiple sets of data includes:

[0015] The priority of the multiple inference models is determined based on the multiple sets of data;

[0016] Based on the priority of the multiple inference models, instances of the multiple inference models are reduced.

[0017] Optionally, determining the priority of the multiple inference models based on the multiple sets of data includes:

[0018] The multiple sets of data are input into a trained classification model to obtain the categories of the multiple inference models, and the categories of the multiple inference models indicate the priority of the multiple inference models.

[0019] Optionally, the step of reducing the number of instances of the plurality of inference models based on their priority includes:

[0020] Based on the priority of the multiple inference models and the estimated released resources, at least one inference model of the multiple inference models to be deleted and the number of instances to be deleted in the at least one inference model are determined. The estimated released resources refer to the difference between the reserved resources and the estimated remaining resources.

[0021] The instances of the at least one inference model are reduced according to the number of instances to be reduced in the at least one inference model.

[0022] Optionally, before reducing the number of instances of the plurality of inference models based on their priorities, the method further includes:

[0023] Based on the priority of the multiple inference models, the maximum resource to be released is determined, whereby the maximum resource that can be released at present is the maximum resource that can be released.

[0024] If the estimated released resources are less than or equal to the maximum released resources, then the step of reducing the instances of the multiple inference models based on their priorities is executed, where the estimated released resources refer to the difference between the reserved resources and the estimated remaining resources.

[0025] Optionally, the data in the multiple dimensions includes at least one of the following: attribute data, operational data, and inference data.

[0026] On the other hand, a resource management device is provided, the device comprising:

[0027] A determination module is used to determine the estimated remaining resources in response to a target instruction, wherein the target instruction indicates that the first inference model needs to be started or the instance of the already started second inference model needs to be expanded, and the estimated remaining resources refer to the estimated remaining resources after the target instruction is executed;

[0028] The acquisition module is used to acquire data from multiple inference models that have been started in multiple dimensions if the estimated remaining resources are less than the reserved resources, so as to obtain multiple sets of data. The reserved resources refer to the resources that enable the system to run normally. The multiple inference models do not include the second inference model.

[0029] The deletion module is used to delete instances of the multiple inference models based on the multiple sets of data, and allocate resources for the execution of the target instruction, wherein the estimated remaining resources after instance deletion are greater than or equal to the reserved resources.

[0030] Optionally, the target instruction indicates that the first inference model needs to be started; the determining module is specifically used for:

[0031] Obtain the current remaining resources, the file size of the first inference model, and the number of the first instances, where the number of the first instances refers to the number of instances configured for the first inference model;

[0032] The estimated remaining resources are obtained based on the current remaining resources, the file size of the first inference model, and the number of the first instances.

[0033] Optionally, the target instruction indicates an instance where the second inference model needs to be expanded; the determining module is specifically used for:

[0034] Get the current remaining resources, the memory usage of the second inference model and the number of second instances. The number of second instances refers to the number of instances that need to be added for the second inference model.

[0035] The estimated remaining resources are obtained based on the current remaining resources, the memory usage of the second inference model, and the number of the second instances.

[0036] Optionally, the deletion module is specifically used for:

[0037] The priority of the multiple inference models is determined based on the multiple sets of data;

[0038] Based on the priority of the multiple inference models, instances of the multiple inference models are reduced.

[0039] Optionally, the removal of modules is specifically used for:

[0040] The multiple sets of data are input into a trained classification model to obtain the categories of the multiple inference models, and the categories of the multiple inference models indicate the priority of the multiple inference models.

[0041] Optionally, the deletion module is specifically used for:

[0042] Based on the priority of the multiple inference models and the estimated released resources, at least one inference model of the multiple inference models to be deleted and the number of instances to be deleted in the at least one inference model are determined. The estimated released resources refer to the difference between the reserved resources and the estimated remaining resources.

[0043] The instances of the at least one inference model are reduced according to the number of instances to be reduced in the at least one inference model.

[0044] Optionally, the deletion module is specifically used for:

[0045] Based on the priority of the multiple inference models, the maximum resource to be released is determined, whereby the maximum resource that can be released at present is the maximum resource that can be released.

[0046] If the estimated released resources are less than or equal to the maximum released resources, then the step of reducing the instances of the multiple inference models based on their priorities is executed, where the estimated released resources refer to the difference between the reserved resources and the estimated remaining resources.

[0047] Optionally, the data in the multiple dimensions includes at least one of the following: attribute data, operational data, and inference data.

[0048] On the other hand, a server is provided, the server including a memory and a processor, the memory for storing computer programs, and the processor for executing the computer programs stored in the memory to implement the steps of the resource management method described above.

[0049] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of the resource management method described above.

[0050] On the other hand, a computer program product containing instructions is provided, which, when executed on a computer, cause the computer to perform the steps of the resource management method described above.

[0051] The technical solution provided in this application can bring at least the following beneficial effects:

[0052] If the estimated remaining resources are less than the reserved resources, it indicates that executing the target instruction will lead to resource scarcity, potentially causing system slowdowns or even crashes. Therefore, by acquiring data from multiple dimensions of the multiple running inference models, and then reducing the number of instances of these models based on this data, resources can be freed up. This minimizes the risk of system slowdowns or crashes due to resource scarcity, and also reduces the likelihood of failures in starting new or expanding inference models, thereby reducing inference request waiting time and improving inference efficiency. In other words, by setting reserved resources and ensuring that these resources are not occupied, the system can be guaranteed to run normally at all times, effectively preventing system slowdowns or crashes due to resource scarcity. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart of a resource management method provided in an embodiment of this application;

[0055] Figure 2 This is a flowchart of another resource management method provided in the embodiments of this application;

[0056] Figure 3 This is a schematic diagram of the structure of a resource management device provided in an embodiment of this application;

[0057] Figure 4 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0059] Before providing a detailed explanation of the resource management method provided in the embodiments of this application, the application scenarios involved in the embodiments of this application will be introduced first.

[0060] Model-as-a-service (MAS) refers to deploying multiple inference models or multiple versions of the same inference model on a unified server to provide model inference services to users, thereby meeting their model inference needs. In other words, a server can deploy multiple different inference models or different versions of the same inference model. Each inference model deployed on the server has at least one instance, which handles inference requests from users. If multiple instances of the inference model exist on the server, these instances can run simultaneously to handle multiple inference requests from users.

[0061] In practical applications, users can query the currently deployed inference models on the server and determine their current status—whether they are running or not. If the required inference model is running, the user can directly send an inference request to the server to initiate inference. If the required inference model is not running, the user can send a start command to the server to start the model, and then send an inference request. If the required inference model is not found on the server, the user can upload their desired model. The server can receive the uploaded model, start it, and then the user can send an inference request. Furthermore, when there are too many inference requests for a particular model or the average queuing time for a particular model is long, the number of instances of that model can be increased, allowing it to handle more inference requests, further reducing the waiting time and improving the server's inference efficiency.

[0062] Because new inference models or instances of expanded inference models can be continuously deployed on the server, the server's resource consumption will increase accordingly. This may lead to problems such as the system running too slowly or even crashing. Furthermore, there may be situations where starting new inference models or expanding instances of inference models fails, resulting in longer inference request waiting times and reduced inference efficiency.

[0063] To prevent the risks caused by the continuous growth of resource consumption, it is necessary to limit the upper limit of resources that model service can use, that is, to reserve enough idle resources to ensure the normal operation of the system. Therefore, with limited resources, it is necessary to manage the resource requests and usage of each inference model to ensure the rational allocation of resources. Thus, this application provides a resource management method. Using the resource management method provided by this application, based on data from multiple dimensions of multiple started inference models, instances of multiple started inference models can be reduced to free up server resources. This minimizes the problem of system slowdown or even crash due to resource shortages, and also reduces the failure to start new inference models and expand inference model instances, thereby reducing inference request waiting time and improving inference efficiency.

[0064] The execution subject of the resource management method provided in this application embodiment is a server. The server can be an independent server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, or a cloud computing service center.

[0065] It should be noted that the application scenarios and execution entities described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the emergence of new application scenarios and servers, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0066] The resource management method provided in the embodiments of this application will now be explained in detail.

[0067] Figure 1 This is a flowchart illustrating a resource management method provided in an embodiment of this application, which is applied to a server. Please refer to... Figure 1 The method includes the following steps.

[0068] Step 101: In response to the target instruction, determine the estimated remaining resources. The target instruction indicates that the first inference model needs to be started or the instance of the already started second inference model needs to be expanded. The estimated remaining resources refer to the resources that the server estimates will remain after executing the target instruction.

[0069] Based on the above description, since new inference models or instances of inference models can be continuously deployed on the server, a target instruction needs to be triggered when the server needs to start the first inference model or expand an instance of the already started second inference model.

[0070] Because the target instruction can have two different meanings—either instructing the server to start the first inference model or instructing the server to expand an instance of the already started second inference model—the methods for determining the estimated remaining resources differ depending on the meaning of the target instruction. These will be discussed separately below.

[0071] In the first interpretation, the target instruction indicates that the first inference model needs to be started. At this time, the server obtains the current remaining resources, the file size of the first inference model, and the number of the first instances. The number of the first instances refers to the number of instances configured for the first inference model. The estimated remaining resources are obtained by subtracting the product of the file size of the first inference model and the number of the first instances from the current remaining resources.

[0072] Since the server may have started at least one inference model when responding to the target command, and this inference model will consume resources on the server, it is possible to obtain the remaining resources of the server and thus know the remaining available resources on the server.

[0073] In practical applications, if the inference model required by the user is not yet started, the user can send a start command to the server to activate the inference model, and then send an inference request to the server for that model. If the user cannot find the required inference model on the server, the user can upload their desired inference model to the server. The server can then receive the uploaded inference model, activate it, and the user can then send an inference request to the server for that model. In other words, the first inference module to be activated could be an existing but not yet activated inference model on the server, or it could be the inference model currently uploaded by the user.

[0074] Because users can configure the number of instances required for the inference model to run when it is uploaded to the server, and the server can know the file size of the inference model after it is uploaded, the server can obtain the file size and the number of instances of the first inference model, i.e., the number of first instances, regardless of whether the first inference model is an existing but not yet started inference model on the server or a currently uploaded inference model by the user.

[0075] Since an inference model may have multiple instances, these instances can run concurrently to handle multiple inference requests from users. In other words, multiple instances of a particular inference model running on the server can be considered as the server running multiple instances of that inference model simultaneously. Furthermore, since the first inference model has not yet started, the memory size occupied by an instance of the first inference model after startup cannot be determined. Therefore, the server can use the product of the first inference model's file size and the number of its first instances as the resource size occupied by the first inference model after startup. Then, it can subtract the product of the first inference model's file size and the number of its first instances from the current remaining resources to obtain the estimated remaining resources after the first inference model starts.

[0076] The second interpretation is that the target instruction indicates that the number of instances of the already launched second inference model needs to be expanded. In this case, the server obtains the current remaining resources, the memory usage of the second inference model, and the number of second instances. The number of second instances refers to the number of instances that need to be expanded for the second inference model. The estimated remaining resources are obtained by subtracting the product of the memory usage of the second inference model and the number of second instances from the current remaining resources.

[0077] Since the second inference model has already started, the server can obtain the memory size occupied by one instance of the second inference model after startup, i.e., the memory usage of the second inference model. In other words, the memory usage of the second inference model refers to the memory size occupied by one instance of the second inference model. Therefore, the server can use the product of the memory usage of the second inference model and the number of second instances as the resource size required for the current expansion of the second inference model's instances. Then, the server subtracts the product of the memory usage of the second inference model and the number of second instances from the current remaining resources to obtain the estimated remaining resources after expanding the instances of the second inference model.

[0078] Step 102: If the estimated remaining resources are less than the reserved resources, then obtain data from multiple inference models that have been started in multiple dimensions to obtain multiple sets of data. The reserved resources refer to the resources that enable the system to run normally. The multiple inference models do not include the second inference model.

[0079] Optionally, if the estimated remaining resources are greater than or equal to the reserved resources, the server executes the target instruction.

[0080] If the estimated remaining resources are less than the reserved resources, it indicates that executing the target instruction may cause resource strain on the server, posing a risk of system slowdown or even crash. In this case, the server can obtain data from multiple inference models across multiple dimensions to obtain multiple sets of data. If the estimated remaining resources are greater than or equal to the reserved resources, it indicates that the server currently has sufficient resources and executing the target instruction will not cause resource strain, thus preventing the system from running normally. Therefore, the server can directly execute the target instruction.

[0081] In some instances, the data in the above multiple dimensions includes at least one of the following: attribute data, operational data, and inference data.

[0082] In some instances, attribute data may include the inference model's name, version, upload date, file size, number of configuration instances, etc. The upload date refers to the date the user uploaded the inference model to the server. The number of configuration model instances refers to the number of instances the user configured the inference model to require for runtime when it was uploaded to the server.

[0083] In some instances, runtime data may include the memory usage of the inference model, the last startup status, online duration, availability of a new version, and the current number of instances. Memory usage refers to the amount of server memory occupied by a single instance of the inference model after startup and before processing inference requests. The last startup status indicates the state of the inference model's last startup, i.e., whether it started successfully or failed. The current number of instances refers to the number of instances of the inference model currently running on the server.

[0084] In some instances, inference data may include the number of IP (Internet Protocol) addresses sending inference requests, the total number of inference requests, the number of successful inferences, the number of failed inferences, the average end-to-end time for inference requests, the average queuing time for inference requests, the average computation time for inference requests, and so on. The total number of inference requests refers to the number of all inference requests that require the inference model to perform inference. The average end-to-end time for inference requests is the average difference between the time the server receives the inference request and the time the server issues the inference result. The average computation time for each request is the average time from when the inference model begins inference to when it obtains the inference result.

[0085] Of course, the data in the above multiple dimensions may also include other dimensions, and this application embodiment does not limit this.

[0086] It should be noted that the data across these multiple dimensions can refer to all data from the time the inference model starts until it retrieves data from the server. Alternatively, the data can be specific to a target time period. This target time period is a pre-set value and can be adjusted according to different needs. For example, the target time period could be from the current time to 5 minutes prior to the current time; that is, retrieving data from the inference model within the 5 minutes prior to the current time. If retrieving data from the inference model within the target time period, then the aforementioned inference data represents the inference data within that target time period. Reserved resources are pre-set and can also be adjusted according to different needs.

[0087] Step 103: Based on the multiple sets of data, reduce the number of instances of the multiple inference models and allocate resources for the execution of the target instruction. After reducing the number of instances, the estimated remaining resources are greater than or equal to the reserved resources.

[0088] In some embodiments, the server can determine the priority of the multiple inference models based on the multiple sets of data, and reduce the number of instances of the multiple inference models based on the priority of the multiple inference models.

[0089] As an example, the server feeds the multiple sets of data into a trained classification model to obtain the categories of the multiple inference models, which indicate the priority of the multiple inference models.

[0090] Before inputting multiple sets of data into a trained classification model, the classification model to be trained needs to be trained. That is, the server can obtain data from each launched inference model in multiple dimensions and the corresponding category of each inference model, and then use the data from each launched inference model in multiple dimensions as input and the corresponding category of each inference model as output to train the classification model to be trained.

[0091] During training, different weights can be assigned to the data across these multiple dimensions based on actual needs, enabling the classification model to more accurately classify the inference model. For example, if the total number of inference requests and memory usage are of greater importance, the weights corresponding to these two dimensions can be increased. Of course, the classification model can also be trained without setting weights. This application does not limit this approach.

[0092] Optionally, after obtaining the trained classification model, the model can be evaluated. That is, data from multiple dimensions is input into the classification model to obtain the category predicted by the classification model. Then, the category predicted by the classification model is compared with the pre-determined category of the inference model to determine the quality of the classification model, so as to improve the classification model.

[0093] It should be noted that when training the classification model, data from the already started inference model within a set time period can be obtained. This way, the obtained data can better represent the latest situation of the inference model, thus enabling the final classification model to classify the inference model more accurately.

[0094] As an example, the classification model can categorize multiple launched inference models into four classes (A, B, C, and D) based on data across multiple dimensions, with priority decreasing in that order. Class A represents the highest priority, characterized by a high number of inference request addresses, a high total number of inference requests, a high number of successful inferences, a long average queuing time for inference requests, low memory usage, and no new version updates. Class B represents a medium priority, with the above dimensions at a moderate level. Class C represents a low priority, with the above dimensions at a low level. Class D represents the lowest priority, with zero inference requests and the existence of a new version update.

[0095] In some embodiments, the process of reducing instances of the plurality of inference models based on their priorities may include: determining at least one inference model with instances to be reduced and the number of instances to be reduced in the plurality of inference models based on their priorities and estimated released resources, wherein the estimated released resources refer to the difference between reserved resources and estimated remaining resources, and reducing instances of the at least one inference model according to the number of instances to be reduced in the at least one inference model.

[0096] Optionally, the memory usage of the multiple inference models can be obtained, and then the multiple inference models can be sorted based on their priority and memory usage to obtain a sorting result. Based on the sorting result and the estimated release of resources, at least one inference model with instances to be deleted and the number of instances to be deleted in the at least one inference model can be determined.

[0097] In some embodiments, the process of sorting the multiple inference models based on their priorities and memory footprints to obtain a sorting result may include: sorting the multiple inference models in ascending order of priority; and for inference models of the same priority, sorting them in descending order of memory footprint, ultimately obtaining a sorting result. The inference model ranked first in the sorting result has the lowest priority and the largest memory footprint, while the inference model ranked last has the highest priority and the smallest memory footprint.

[0098] In some embodiments, the process of determining at least one inference model with instances to be deleted and the number of instances to be deleted in the at least one inference model based on the ranking results of the plurality of inference models and the estimated release resources may include: determining a first release resource corresponding to each of the plurality of inference models based on the priority of the plurality of inference models, wherein the first release resource refers to the maximum resource that can be released after the inference model performs instance deletion; determining a plurality of cumulative release resources based on the ranking results of the plurality of inference models, wherein the plurality of cumulative release resources refers to the sum of the first release resources corresponding to different numbers of inference models in the order of the ranking results from front to back; determining one cumulative release resource as the target release resource from the plurality of cumulative release resources; and determining at least one inference model with instances to be deleted and the number of instances to be deleted in the at least one inference model based on the target release resource.

[0099] Because users have different needs for different inference models, higher-priority inference models receive more inference requests and require more instances to improve inference efficiency, while lower-priority inference models receive fewer inference requests and can have fewer instances. Therefore, when reducing the number of instances of an inference model, it is necessary to specify the minimum number of instances to be retained for each priority model to ensure resource management on the server while minimizing the impact on the model-servoir system.

[0100] In some embodiments, the server pre-stores a correspondence between priorities and minimum instance counts, indicating the minimum number of instances that need to be retained for inference models of different priorities. For any one of the multiple inference models, based on its priority, the minimum instance count corresponding to that inference model is determined from the correspondence between priorities and minimum instance counts. The current instance count and memory usage of that inference model are obtained. The minimum instance count is subtracted from the current instance count to obtain the third instance count. The third instance count is multiplied by the memory usage of that inference model to obtain the first released resource corresponding to that inference model. After processing each of the multiple inference models in the same way, the first released resources corresponding to each of the multiple inference models can be obtained.

[0101] As an example, the process of determining multiple cumulative resource releases can be represented by the following formula (1).

[0102]

[0103] In formula (1) above, A x Q represents the first released resource corresponding to the x-th inference model in the sorted results from front to back. i It represents the i-th cumulatively released resource among multiple cumulatively released resources, and also represents the cumulative sum of the first released resources corresponding to the first i inference models in the sorting results from front to back.

[0104] For example, the correspondence between priority and minimum instance count is as follows: the minimum instance count for classes A, B, and C is 1, while the minimum instance count for class D is 0. Since class D indicates that the total number of inference requests for this inference model is 0 and a new version exists, it means that users frequently use the new version for inference and are less likely to use the old version. Therefore, the instance count for the inference model in class D can be reduced to 0.

[0105] Based on the above description, the server can set a minimum number of instances for inference models of different priorities. That is, when reducing the number of instances, the server cannot reduce the number of instances for all inference models to 0. Alternatively, the server can set the minimum number of instances for inference models of different priorities to 0 or not set a minimum number of instances for inference models of different priorities; in other words, the number of instances for all inference models can be reduced to 0. This application does not limit this aspect.

[0106] The process of determining a target release resource from multiple cumulative release resources can include: identifying the cumulative release resource that is greater than or equal to the estimated release resource and has the smallest difference from the estimated release resource as the target release resource. Of course, in practical applications, other methods can also be used to determine the target release resource. For example, any one of the multiple cumulative release resources that is greater than or equal to the estimated release resource can be identified as the target release resource.

[0107] Since the cumulative released resources represent the cumulative sum of the first released resources corresponding to at least one inference model, and the target released resource is one of multiple cumulative released resources, after determining the target released resource, it is possible to determine the inference model corresponding to the target released resource and the number of third instances corresponding to the inference model. In this way, the inference model corresponding to the target released resource can be directly determined as at least one inference model of the instances to be deleted among the multiple inference models, and the number of third instances corresponding to the at least one inference model can be determined as the number of instances to be deleted in the at least one inference model.

[0108] Since the target released resources are the maximum resources that can be released by at least one inference model of the instance to be deleted, although the above method can ensure that the resources released in the end meet the execution of the target instruction, some unnecessary instances may be deleted. That is to say, for the last inference model in at least one inference model of the instance to be deleted, it may be possible to ensure that the remaining resources of the server meet the execution of the target instruction by deleting some instances of that inference model, without needing to delete the instances of that inference model to the minimum. Therefore, to avoid unnecessary deletion of instances and ensure the accuracy of the final number of instances in the inference model to be deleted, before determining at least one inference model of instances to be deleted and the number of instances to be deleted in the at least one inference model, the memory usage of the last inference model of the at least one inference model of instances to be deleted can also be determined. If the difference between the target released resources and the estimated released resources divided by the memory usage is greater than or equal to 1, then the integer part of the difference between the target released resources and the estimated released resources divided by the memory usage is determined as the number of instances to be retained. The number of instances to be retained is subtracted from the number of the third instance of the last inference model of the at least one inference model of instances to be deleted to obtain the number of instances to be deleted in the last inference model.

[0109] If the difference between the target released resources and the estimated released resources divided by the memory usage size is less than 1, then the inference model corresponding to the target released resources is directly determined as at least one inference model of the instances to be deleted among the multiple inference models, and the number of third instances corresponding to the at least one inference model is determined as the number of instances to be deleted among the at least one inference model.

[0110] If the difference between the target released resources and the estimated released resources, divided by the memory footprint of the last inference model, is greater than or equal to 1, it means the difference is greater than or equal to the memory footprint of the last inference model. Therefore, it is unnecessary to reduce the number of instances in the last inference model to a minimum, and there are instances that need to be unnecessary to be deleted. Thus, the integer part of the difference between the target released resources and the estimated released resources, divided by the memory footprint, can be determined as the number of instances to be retained. Subtracting the number of instances to be retained from the number of third instances in the last inference model yields the number of instances to be deleted in that last inference model. Conversely, if the difference between the target released resources and the estimated released resources, divided by the memory footprint of the last inference model, is less than 1, it means the difference is less than the memory footprint of the last inference model. The number of instances in the last inference model needs to be reduced to a minimum, and there are no instances that need to be unnecessary to be deleted. Therefore, the number of third instances corresponding to the last inference model can be determined as the number of instances to be deleted in that last inference model.

[0111] Based on the above description, the server can first determine at least one inference model that requires instance deletion to execute the target instruction, as well as the number of instances to be deleted in that inference model, and then delete the instances of that inference model. Alternatively, the server can also delete the number of instances of the inference models sequentially from front to back based on the sorting results of the multiple inference models. After each deletion, the server's estimated remaining resources are determined until the server's estimated remaining resources are greater than or equal to the reserved resources, at which point the deletion of the number of instances is complete. That is, instances of the inference models are deleted sequentially until the server's remaining resources can satisfy the execution of the target instruction. This application does not limit this approach.

[0112] In practical applications, executing a target instruction may require significant resources. If instances of inference models are directly reduced based on priority, the remaining server resources might still be insufficient to execute the target instruction even after reducing the number of already started inference model instances. In this case, not only will the server be unable to execute the target instruction, but the reduction in the number of already started inference model instances could also decrease inference efficiency. Therefore, to avoid situations where the target instruction cannot be executed even after reducing inference model instances, the server can determine the maximum release resource based on the priorities of multiple inference models before reducing their instances. This maximum release resource refers to the maximum resource that the server can currently release. If the estimated release resource is less than or equal to the maximum release resource, the step of reducing the instances of multiple inference models based on their priorities is executed. If the estimated release resource is greater than the maximum release resource, an alarm message is generated to indicate that the target instruction execution has failed.

[0113] In some embodiments, the maximum released resource can be the sum of the first released resources corresponding to the plurality of inference models. The process for determining the first released resources corresponding to the plurality of inference models is described in the preceding text and will not be repeated here.

[0114] If the estimated released resources are less than or equal to the maximum released resources, it means that after releasing the resources, the server's remaining resources will definitely be sufficient to satisfy the execution of the target instruction. Therefore, the step of reducing the instances of the multiple inference models based on their priorities can be executed. If the estimated released resources are greater than the maximum released resources, it means that even after releasing all available resources, the server will still be unable to satisfy the execution of the target instruction. Therefore, an alarm message can be generated to indicate that the target instruction execution has failed.

[0115] In practical applications, a server may need to simultaneously expand multiple instances of already running inference models. In other words, the target instruction indicates the need to simultaneously expand multiple instances of already running second inference models. In this case, before determining the estimated remaining resources, the server can obtain data on these multiple second inference models across multiple dimensions. Based on this data, the server determines the priority of each second inference model and then allocates resources to them sequentially according to their priority from highest to lowest. The implementation process of determining the priority of the multiple second inference models based on their data across multiple dimensions is detailed above and will not be repeated here.

[0116] Since there may be at least two second inference models that belong to the same priority, the multiple second inference models can be sorted based on their priority and the average queuing time of their inference requests to obtain a sorting result. Then, resources are allocated to the second inference models in order from front to back according to the sorting result.

[0117] In some embodiments, the multiple second inference models can be sorted according to their priority from highest to lowest. For second inference models of the same priority, they are further sorted according to their average inference request queuing time from longest to shortest, resulting in a final sorting result. The second inference model ranked first in the sorting result has the highest priority and the longest average inference request queuing time, while the second inference model ranked last has the lowest priority and the shortest average inference request queuing time.

[0118] In other words, when the target instruction indicates the need to simultaneously expand multiple instances of already launched second inference models, resources can be allocated to these second inference models sequentially in descending order of priority, ultimately completing the execution of the target instruction. This ensures that higher-priority inference models receive server resources first, further improving the server's inference efficiency.

[0119] Next Figure 2 Taking an example, this application provides a resource management method according to an embodiment. Figure 2 As shown, in response to the target instruction, the server determines the estimated remaining resources. If the estimated remaining resources are less than the reserved resources, it indicates that executing the target instruction may cause resource strain on the server, posing a risk of system slowdown or even crash. In this case, the server can obtain data from multiple running inference models across multiple dimensions to obtain multiple sets of data. Based on these multiple sets of data, the server can then reduce the number of instances of the multiple inference models and allocate resources for the execution of the target instruction, i.e., execute the target instruction. If the estimated remaining resources are greater than or equal to the reserved resources, it indicates that the server currently has sufficient resources, and executing the target instruction will not cause resource strain, thus preventing the system from running normally. Therefore, the server can directly execute the target instruction.

[0120] In this embodiment, when the estimated remaining resources are less than the reserved resources, executing the target instruction will lead to resource scarcity, potentially causing the system to run slowly or even crash. Therefore, data from multiple dimensions of the multiple launched inference models is obtained, and instances of these models are reduced based on this data to release resources. This minimizes the risk of system slowdown or crash due to resource scarcity and reduces the likelihood of failures in launching new or expanding inference models, thereby reducing inference request waiting time and improving inference efficiency. In other words, by reserving resources for the server and ensuring these resources are not occupied, the server can always operate normally, effectively preventing system slowdown or crash due to resource scarcity. Furthermore, the server can set minimum instance counts for inference models of different priorities and reduce instances based on model priority. This allows lower-priority inference models to have their instances reduced first, releasing server resources and managing server resources while maintaining inference efficiency.

[0121] The method provided in this application embodiment can also determine the number of inference model instances to be retained, thereby ensuring the accuracy of the final number of inference model instances to be deleted, avoiding unnecessary deletion of instances, and managing server resources while ensuring inference efficiency. Furthermore, by determining the maximum release resource, and based on the relationship between the estimated release resource and the maximum release resource, it is determined whether to delete inference model instances. When the estimated release resource is greater than the maximum release resource, it indicates that even after releasing all available resources, the server cannot satisfy the execution of the target instruction. In this case, instances of already started inference models are not deleted. This avoids the situation where the target instruction still cannot be executed after deleting a certain number of instances, avoiding unnecessary deletion of instances and ensuring server inference efficiency. When the target instruction indicates the need to simultaneously expand multiple instances of already started second inference models, resources can be allocated to these multiple second inference models in descending order of priority, ultimately completing the execution of the target instruction. This ensures that higher-priority inference models obtain server resources first, further improving server inference efficiency.

[0122] Figure 3 This is a schematic diagram of a resource management device provided in an embodiment of this application. This resource management device can be implemented as part or all of a server by software, hardware, or a combination of both. Please refer to... Figure 3 The device includes: a determination module 301, an acquisition module 302, and a deletion module 303.

[0123] The determination module 301 is used to determine the estimated remaining resources in response to a target instruction. This target instruction indicates whether the first inference model needs to be started or whether an instance of the already started second inference model needs to be expanded. The estimated remaining resources refer to the estimated resources remaining after executing the target instruction. Detailed implementation processes are described in the corresponding contents of the above embodiments and will not be repeated here.

[0124] The acquisition module 302 is used to acquire data from multiple inference models across multiple dimensions if the estimated remaining resources are less than the reserved resources, thereby obtaining multiple sets of data. The reserved resources refer to the resources necessary for the system to operate normally. The multiple inference models do not include the second inference model. For detailed implementation processes, please refer to the corresponding content in the above embodiments; they will not be repeated here.

[0125] The deletion module 303 is used to delete instances of the multiple inference models based on multiple sets of data and allocate resources for the execution of the target instruction. After instance deletion, the estimated remaining resources are greater than or equal to the reserved resources. Detailed implementation processes are described in the corresponding contents of the above embodiments and will not be repeated here.

[0126] Optionally, the target instruction indicates that the first inference model needs to be started at this time;

[0127] The determination module 301 is specifically used for:

[0128] Get the current remaining resources, the file size of the first inference model, and the number of the first instances, where the number of the first instances refers to the number of instances configured for the first inference model;

[0129] The estimated remaining resources are obtained based on the current remaining resources, the file size of the first inference model, and the number of the first instances.

[0130] Optionally, the target instruction indicates the instance that currently needs to be expanded into the second inference model;

[0131] The determination module 301 is specifically used for:

[0132] Get the current remaining resources, the memory usage of the second inference model, and the number of second instances. The number of second instances refers to the number of instances that need to be added for the second inference model.

[0133] Based on the current remaining resources, the memory usage of the second inference model, and the number of second instances, the estimated remaining resources are obtained.

[0134] Optionally, module 303 is specifically used for:

[0135] Prioritizing multiple inference models based on multiple sets of data;

[0136] Based on the priority of multiple inference models, instances of multiple inference models are reduced.

[0137] Optionally, module 303 is specifically used for:

[0138] Multiple sets of data are input into a trained classification model to obtain multiple inference model categories, which indicate the priority of the multiple inference models.

[0139] Optionally, module 303 is specifically used for:

[0140] Based on the priorities of multiple inference models and the estimated released resources, at least one inference model with instances to be deleted and the number of instances to be deleted in at least one inference model are determined. The estimated released resources refer to the difference between reserved resources and estimated remaining resources.

[0141] Delete instances of at least one inference model according to the number of instances to be deleted in the at least one inference model.

[0142] Optionally, module 303 is specifically used for:

[0143] Based on the priority of multiple inference models, the maximum resource to be released is determined, which refers to the maximum resource that can be released at present.

[0144] If the estimated released resources are less than or equal to the maximum released resources, then a step is performed to reduce the number of instances of multiple inference models based on the priority of multiple inference models. The estimated released resources refer to the difference between the reserved resources and the estimated remaining resources.

[0145] Optionally, the multi-dimensional data includes at least one of the following: attribute data, operational data, and inference data.

[0146] In this embodiment, when the estimated remaining resources are less than the reserved resources, executing the target instruction will lead to resource scarcity, potentially causing the system to run slowly or even crash. Therefore, data from multiple dimensions of the multiple launched inference models is obtained, and instances of these models are reduced based on this data to release resources. This minimizes the risk of system slowdown or crash due to resource scarcity and reduces the likelihood of failures in launching new or expanding inference models, thereby reducing inference request waiting time and improving inference efficiency. In other words, by reserving resources for the server and ensuring these resources are not occupied, the server can always operate normally, effectively preventing system slowdown or crash due to resource scarcity. Furthermore, the server can set minimum instance counts for inference models of different priorities and reduce instances based on model priority. This allows lower-priority inference models to have their instances reduced first, releasing server resources and managing server resources while maintaining inference efficiency.

[0147] The method provided in this application embodiment can also determine the number of inference model instances to be retained, thereby ensuring the accuracy of the final number of inference model instances to be deleted, avoiding unnecessary deletion of instances, and managing server resources while ensuring inference efficiency. Furthermore, by determining the maximum release resource, and based on the relationship between the estimated release resource and the maximum release resource, it is determined whether to delete inference model instances. When the estimated release resource is greater than the maximum release resource, it indicates that even after releasing all available resources, the server cannot satisfy the execution of the target instruction. In this case, instances of already started inference models are not deleted. This avoids the situation where the target instruction still cannot be executed after deleting a certain number of instances, avoiding unnecessary deletion of instances and ensuring server inference efficiency. When the target instruction indicates the need to simultaneously expand multiple instances of already started second inference models, resources can be allocated to these multiple second inference models in descending order of priority, ultimately completing the execution of the target instruction. This ensures that higher-priority inference models obtain server resources first, further improving server inference efficiency.

[0148] It should be noted that the resource management device provided in the above embodiments is only illustrated by the division of the above functional modules when performing resource management. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the resource management device and the resource management method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0149] Figure 4 This is a schematic diagram of the structure of a server provided in an embodiment of this application. The server 400 includes a central processing unit (CPU) 401, a system memory 404 including random access memory (RAM) 402 and read-only memory (ROM) 403, and a system bus 405 connecting the system memory 404 and the central processing unit 401. The server 400 also includes a basic input / output system (I / O system) 406 that facilitates the transfer of information between various devices within the computer, and a mass storage device 407 for storing the operating system 413, application programs 414, and other program modules 415.

[0150] The basic input / output system 406 includes a display 408 for displaying information and an input device 409 for user input, such as a mouse or keyboard. Both the display 408 and the input device 409 are connected to the central processing unit 401 via an input / output controller 410 connected to the system bus 405. The basic input / output system 406 may also include the input / output controller 410 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 410 also provides output to a display screen, printer, or other types of output devices.

[0151] Mass storage device 407 is connected to central processing unit 401 via a mass storage controller (not shown) connected to system bus 405. Mass storage device 407 and its associated computer-readable media provide non-volatile storage for server 400. That is, mass storage device 407 may include computer-readable media (not shown) such as hard disk or CD-ROM drive.

[0152] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies, CD-ROM, DVD or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 404 and mass storage device 407 described above can be collectively referred to as memory.

[0153] According to various embodiments of this application, server 400 can also be connected to a remote computer on a network, such as the Internet. That is, server 400 can be connected to network 412 via network interface unit 411 connected to system bus 405, or it can use network interface unit 411 to connect to other types of networks or remote computer systems (not shown).

[0154] The aforementioned memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU.

[0155] In some embodiments, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the resource management method described above. For example, the computer-readable storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0156] It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application can be a non-volatile storage medium, in other words, it can be a non-transient storage medium.

[0157] It should be understood that all or part of the steps of the above embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially in the form of a computer program product. The computer program product includes one or more computer instructions. The computer instructions can be stored in the above-described computer-readable storage medium.

[0158] That is, in some embodiments, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of the resource management method described above.

[0159] It should be understood that "at least one" as mentioned herein refers to one or more, and "multiple" refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, in order to clearly describe the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., are not necessarily different.

[0160] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the data from multiple dimensions of the multiple inference models that have been activated involved in the embodiments of this application were obtained under full authorization.

[0161] The above descriptions are embodiments provided in this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A resource management method, characterized in that, The method comprises: In response to a target instruction, an estimated remaining resource is determined, wherein the target instruction indicates that an instance of a first inference model needs to be started or an instance of an already started second inference model needs to be expanded, and the estimated remaining resource refers to the estimated remaining resources after the target instruction is executed; If the estimated remaining resources are less than the reserved resources, then data from multiple inference models that have been started are obtained in multiple dimensions to obtain multiple sets of data. The reserved resources refer to the resources that enable the system to run normally. The multiple inference models do not include the second inference model. The data in multiple dimensions includes at least one of the following: attribute data, running data, and inference data. Based on the multiple sets of data, the priority of the multiple inference models is determined; The instances of the multiple inference models are reduced in order of priority from low to high, and resources are allocated for the execution of the target instruction. After the instance reduction, the estimated remaining resources are greater than or equal to the reserved resources. When reducing the instances of the multiple inference models, the minimum number of instances needs to be retained for inference models of different priorities.

2. The method as described in claim 1, characterized in that, The target instruction indicates that the first inference model needs to be started; the determination of the estimated remaining resources includes: Obtain the current remaining resources, the file size of the first inference model, and the number of the first instances, where the number of the first instances refers to the number of instances configured for the first inference model; The estimated remaining resources are obtained based on the current remaining resources, the file size of the first inference model, and the number of the first instances.

3. The method as described in claim 1, characterized in that, The target instruction indicates an instance where the second inference model needs to be expanded; the determination of the estimated remaining resources includes: Get the current remaining resources, the memory usage of the second inference model and the number of second instances. The number of second instances refers to the number of instances that need to be added for the second inference model. The estimated remaining resources are obtained based on the current remaining resources, the memory usage of the second inference model, and the number of the second instances.

4. The method as described in claim 1, characterized in that, Determining the priority of the multiple inference models based on the multiple sets of data includes: The multiple sets of data are input into a trained classification model to obtain the categories of the multiple inference models, and the categories of the multiple inference models indicate the priority of the multiple inference models.

5. The method as described in claim 4, characterized in that, The step of reducing the number of instances of the multiple inference models based on their priorities includes: Based on the priority of the multiple inference models and the estimated released resources, at least one inference model of the multiple inference models to be deleted and the number of instances to be deleted in the at least one inference model are determined. The estimated released resources refer to the difference between the reserved resources and the estimated remaining resources. The instances of the at least one inference model are reduced according to the number of instances to be reduced in the at least one inference model.

6. The method as described in claim 5, characterized in that, Before reducing the number of instances of the multiple inference models based on their priorities, the method further includes: Based on the priority of the multiple inference models, the maximum resource to be released is determined, whereby the maximum resource that can be released at present is the maximum resource that can be released. If the estimated released resources are less than or equal to the maximum released resources, then the step of reducing the instances of the multiple inference models based on their priorities is executed, where the estimated released resources refer to the difference between the reserved resources and the estimated remaining resources.

7. A resource management device, characterized in that, The device comprises: A determination module is used to determine the estimated remaining resources in response to a target instruction, wherein the target instruction indicates that the first inference model needs to be started or the instance of the already started second inference model needs to be expanded, and the estimated remaining resources refer to the estimated remaining resources after the target instruction is executed; The acquisition module is used to acquire data from multiple inference models that have been started in multiple dimensions if the estimated remaining resources are less than the reserved resources, so as to obtain multiple sets of data. The reserved resources refer to the resources that enable the system to run normally. The multiple inference models do not include the second inference model. The data in multiple dimensions includes at least one of the following: attribute data, running data, and inference data. The elimination module is used to determine the priority of the multiple inference models based on the multiple sets of data; Based on the priority of the multiple inference models, instances of the multiple inference models are reduced, and resources are allocated for the execution of the target instruction. After the instance reduction, the estimated remaining resources are greater than or equal to the reserved resources. When reducing the instances of the multiple inference models, it is necessary to retain the minimum number of instances for inference models with different priorities.

8. A server, characterized in that, The server includes a memory and a processor, the memory being used to store a computer program, and the processor being configured to execute the computer program stored in the memory to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

Images