Reasoning instance management method and device

Abstract

The invention provides a reasoning instance management method and device. The method is applied to inference service and comprises the steps that when a first message from an inference engine is received, a GPU resource pool is inquired, the inference engine is used for monitoring the GPU resource use condition of an inference instance, and the first message is sent out when the inference engine monitors that the video memory of the inference instance is insufficient; if the GPU card with the same type as the current GPU card exists in the GPU resource pool, the identification information of the GPU card is sent to the reasoning instance, so that the GPU card is newly added to the reasoning instance on the basis of the current GPU card; otherwise, the identification information of the GPU card with the larger video memory is sent to the reasoning instance, so that the reasoning instance replaces the current GPU card with the GPU card with the larger video memory. According to the method, intelligent monitoring, dynamic scheduling and fault recovery of GPU resources are realized through close cooperation of an inference engine and an inference service, so that a stable and reliable execution environment is provided for an inference task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a reasoning instance management method and device. BACKGROUND

[0002] In the field of artificial intelligence, especially in computationally intensive tasks such as deep learning and neural networks, the demand for high-performance computing hardware is growing. As a dedicated accelerator, GPU (Graphics Processing Unit) has been widely used in model reasoning processes to process user input and generate the required information. In the current scheme, the reasoning instance is tightly bound to a specific GPU card, and this rigid binding limits the flexible scheduling of resources. When the GPU card fails, the reasoning instance bound to it cannot continue to operate normally, resulting in service interruption. SUMMARY

[0003] To overcome the problems in the related art, the present application provides a reasoning instance management method and device.

[0004] According to a first aspect of an embodiment of the present application, a reasoning instance management method is provided, which is applied to a reasoning service, and the method comprises:

[0005] Upon receiving a first message from a reasoning engine, querying a GPU resource pool, wherein the reasoning engine is used to monitor the GPU resource usage of a reasoning instance, and the first message is sent by the reasoning engine when it detects that the reasoning instance is short of video memory;

[0006] If there is a GPU card of the same type as the current GPU card in the GPU resource pool, the identification information of the GPU card is sent to the reasoning instance, so that the reasoning instance adds the GPU card to the current GPU card;

[0007] If there is no GPU card of the same type as the current GPU card in the GPU resource pool, but there is a GPU card with more video memory, the identification information of the GPU card with more video memory is sent to the reasoning instance, so that the reasoning instance replaces the current GPU card with the GPU card with more video memory.

[0008] According to a second aspect of an embodiment of the present application, a reasoning instance management device is provided, which is applied to a reasoning service, and the device comprises:

[0009] A querying module is configured to query a GPU resource pool upon receiving a first message from a reasoning engine, wherein the reasoning engine is used to monitor the GPU resource usage of a reasoning instance, and the first message is sent by the reasoning engine when it detects that the reasoning instance is short of video memory;

[0010] The expansion module is configured to, if there is a GPU card of the same type as the current GPU card in the GPU resource pool, send identification information of the GPU card to the inference instance, so that the inference instance adds the GPU card to the current GPU card;

[0011] The replacement module is configured to, if there is no GPU card of the same type as the current GPU card in the GPU resource pool, but there is a GPU card with larger video memory, send identification information of the GPU card with larger video memory to the inference instance, so that the inference instance replaces the current GPU card with the GPU card with larger video memory.

[0012] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising:

[0013] a memory and one or more processors; the memory is coupled to the processor; wherein the memory stores computer program code, the computer program code includes computer instructions, when the computer instructions are executed by the processor, the electronic device executes the method as described above.

[0014] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, comprising computer instructions, when the computer instructions run on an electronic device, the electronic device executes the method as described above.

[0015] According to a fifth aspect of the embodiments of the present application, a computer program product is provided, when the computer program product runs on a computer, the computer executes the method as described above.

[0016] The technical solutions provided by the embodiments of the present application can include the following beneficial effects:

[0017] In the embodiments of the present application, the inference engine monitors the GPU resource usage of the inference instance in real time, and once it finds that the video memory is insufficient, the inference service automatically queries the GPU resource pool and allocates additional GPU cards of the same type to the inference instance, realizes the rapid expansion of GPU resources, if there is no GPU card of the same type available, allocates other types of GPU cards with larger video memory to the inference instance, and ensures business continuity. Dynamically adjust the matching relationship between the inference task and the GPU card, and maximize the use of resources. The purpose of intelligently managing the GPU resources occupied by the inference instance is achieved, and the flexibility, reliability and business continuity of the inference system are improved.

[0018] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate an embodiment consistent with the present application and, together with the description, serve to explain the principles of the application.

[0020] Figure 1 An architecture diagram of a reasoning system provided for an embodiment of the present application;

[0021] Figure 2 A first flowchart of a reasoning instance management method provided for an embodiment of the present application;

[0022] Figure 3 A second flowchart of a reasoning instance management method provided for an embodiment of the present application;

[0023] Figure 4 A third flowchart of a reasoning instance management method provided for an embodiment of the present application;

[0024] Figure 5 A schematic diagram of a reasoning instance management apparatus provided for an embodiment of the present application;

[0025] Figure 6 A structural schematic diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing the specific embodiments and are not intended to be limiting to the present application.

[0027] The present application provides a reasoning instance management scheme, which aims to realize efficient management and dynamic scheduling of GPU resources through the collaborative work of a reasoning engine and a reasoning service. As shown in Figure 1 , the reasoning system mainly includes three parts: a reasoning engine, a reasoning service, and a GPU resource pool.

[0028] The following describes each part.

[0029] The GPU resource pool is a collection of physical GPU resources, including multiple servers (sever), such as Figure 1 sever1 and sever2 shown in the figure, each of which is equipped with multiple GPU cards.

[0030] The reasoning engine is deployed inside the reasoning instance and is responsible for continuously monitoring the GPU resource usage (such as memory occupancy rate, GPU utilization rate, etc.) of the reasoning instance and the reasoning task execution performance (such as processing speed, response time, etc.), and can also report these monitoring information to the reasoning service.

[0031] The inference service is responsible for calling the GPU resources to provide services to the outside, and can also receive monitoring information provided by the inference engine through a management platform (such as Kubernetes), and dynamically adjust the GPU resources occupied by each inference instance according to actual needs, such as expansion or replacement of GPU cards with larger video memories.

[0032] The inference instance is a specific execution unit of the inference service, and runs on a host computer and can perform corresponding operations according to the instructions of the inference service, such as restarting to mount new GPU resources.

[0033] The present application uses a stateful service (StatefulSet) to identify and manage the inference instance. Since the StatefulSet provides a unique identity for each inference instance, the inference service can accurately identify and manage each inference instance. On this basis, the inference service manages and schedules the GPU resources used by each inference instance, optimizes the resource switching mechanism of the inference instance when encountering abnormal conditions or the current resources cannot meet the business requests, and ensures that the inference instance runs in a resource sufficient condition, thereby improving the overall business carrying capacity.

[0034] Specifically, in the process of constructing the inference instance, the inference service uniquely identifies the inference instance with the GPU card through the StatefulSet, thereby realizing effective management of the inference instance and ensuring the stability and high availability of the entire inference system.

[0035] Next, the embodiments of the present application will be described in detail.

[0036] The embodiments of the present application provide an inference instance management method, which is applied to an inference service, such as Figure 2 As shown in the figure, the method can include the following steps:

[0037] Step 210, when receiving a first message from an inference engine, querying a GPU resource pool, wherein the inference engine is used to monitor the GPU resource usage of the inference instance, and the first message is sent by the inference engine when the inference engine detects that the inference instance is short of video memory;

[0038] Specifically, when the inference engine detects a condition of insufficient video memory, such as when the GPU memory usage rate exceeds a preset threshold or cannot meet the needs of the current inference task, the inference engine will generate a first message and send it to the inference service. The message is used to notify the inference service that the current video memory resource is insufficient to support the normal operation of the inference instance, thereby triggering corresponding resource management or scheduling operations.

[0039] When the inference service receives the first message from the inference engine, the message indicates that the inference instance is experiencing a problem of insufficient memory, the inference service will query the GPU resource pool. The purpose of this step is to determine whether there is available GPU resource that can be allocated to the inference instance that encounters the problem of insufficient memory to solve its resource shortage.

[0040] That is, the first message indicates that the inference instance cannot handle concurrent requests due to too many requests, or there is other performance bottleneck that causes it to temporarily unable to respond to new requests.

[0041] Step 220, if there is a GPU card of the same type as the current GPU card in the GPU resource pool, the identification information of the GPU card is sent to the inference instance, so that the inference instance adds the GPU card to the current GPU card;

[0042] If there is another GPU card of the same type as the current GPU card in the GPU resource pool, the inference service sends the identification information of the GPU card to the corresponding inference instance. Wherein, the identification information of the GPU card is data for uniquely identifying the GPU card, which can be serial number, model, capacity, etc.

[0043] This step aims to control the inference instance to add GPU cards of the same type to expand its memory capacity, thereby alleviating or solving the problem of insufficient memory. Selecting GPU cards of the same type can maintain the consistency of the hardware environment and simplify management and maintenance. It is worth mentioning that the above current GPU card refers to the GPU card currently mounted by the inference instance.

[0044] Step 230, if there is no GPU card of the same type as the current GPU card in the GPU resource pool, but there is a GPU card with larger memory, the identification information of the GPU card with larger memory is sent to the inference instance, so that the inference instance replaces the current GPU card with the GPU card with larger memory.

[0045] If there is no other GPU card of the same type as the current GPU card in the GPU resource pool, but there is a GPU card of different type with larger memory capacity, the inference service sends the identification information of the GPU card with larger memory to the inference instance. The purpose of this step is to solve the problem of insufficient memory by replacing the current GPU card with a GPU card with larger memory.

[0046] On this basis, if there is no GPU card of the same type as the current GPU card in the GPU resource pool, and there is no GPU card with larger memory, the inference service can perform a throttling operation on the inference instance, such as limiting the request rate, limiting concurrent requests.

[0047] As a specific implementation, for inference instances, the embodiment of the present application introduces a new state identifier, i.e., a switchable state, on the basis of the normal running state. In this way, the state of each inference instance is explicitly identified, which facilitates the inference service to manage each inference instance accordingly.

[0048] Specifically, in step 220, before the inference instance adds the GPU card on the basis of the current GPU card, the state of the inference instance is modified from the normal running state to the switchable state; after the inference instance adds the GPU card on the basis of the current GPU card, the state of the inference instance is modified from the switchable state to the normal running state.

[0049] Correspondingly, in step 230, before the inference instance replaces the current GPU card with the GPU card with larger memory, the state of the inference instance is modified from the normal running state to the switchable state; after the inference instance replaces the current GPU card with the GPU card with larger memory, the state of the inference instance is modified from the switchable state to the normal running state.

[0050] For GPU cards, the embodiment of the present application defines the occupation state of the GPU card, including the occupied state and the unoccupied state, thereby facilitating the inference service to manage and schedule each GPU card in the GPU resource pool.

[0051] Specifically, in step 220, after the inference instance adds the GPU card on the basis of the current GPU card, the state of the GPU card is modified from the unoccupied state to the occupied state.

[0052] Correspondingly, in step 230, after the inference instance replaces the current GPU card with the GPU card with larger memory, the state of the current GPU card is modified from the occupied state to the unoccupied state, and the state of the GPU card with larger memory is modified from the unoccupied state to the occupied state.

[0053] In the embodiment of the present application, the inference engine is used to monitor the GPU resource usage. For example, the embodiment of the present application divides the GPU resource usage into the following three states:

[0054] Insufficient memory: refers to that the memory capacity of the GPU cannot meet the current or expected computing task requirements, resulting in the inability to load all necessary data or models into the memory.

[0055] Memory error: refers to that the GPU memory has an error when executing a task, which can include data corruption, address error, out-of-bound access, etc., which can be caused by hardware failure or software defects.

[0056] Normal video memory: refers to that the video memory of the GPU works in a normal state, no capacity shortage or error is detected, and the current computing task can be met.

[0057] The above describes the insufficient video memory, and the following describes the video memory error and the normal video memory.

[0058] For the video memory error, as shown in Figure 3 , the following steps can be included:

[0059] Step 310: upon receiving a second packet from the inference engine, querying the GPU resource pool, the second packet being sent by the inference engine when the inference instance video memory error is monitored;

[0060] The second packet indicates that the inference engine monitors that the GPU has a fault or a problem, which affects the inference instance to carry the business capability.

[0061] Step 320: if there is an unoccupied GPU card in the GPU resource pool on the current server, the identification information of the unoccupied GPU card is sent to the inference instance, so that the inference instance replaces the current GPU card with the unoccupied GPU card;

[0062] The above current server refers to the server where the current GPU card (the GPU card currently mounted by the inference instance) is located.

[0063] Step 330: if there is no unoccupied GPU card in the GPU resource pool on the current server, but there is an unoccupied GPU card on another server, the identification information of the unoccupied GPU card on the other server is sent to the inference instance, so that the inference instance replaces the current GPU card with the unoccupied GPU card on the other server.

[0064] For the normal video memory, as shown in Figure 4 , the following steps can be included:

[0065] Step 410: upon receiving a third packet from the inference engine, updating the working state of the inference engine recorded locally according to the third packet, the third packet being sent by the inference engine when the inference instance video memory is normal.

[0066] The third packet indicates that the inference instance is in good running state and can effectively execute the inference task.

[0067] The embodiment of the application provides a reasoning instance management method, when GPU memory is insufficient, reasoning service can automatically select a suitable GPU card from a GPU resource pool to replace or expand, ensuring the smooth execution of reasoning tasks. Through the close cooperation of reasoning engine and reasoning service, intelligent monitoring, dynamic scheduling and fault recovery of GPU resources are realized, thereby providing a stable and reliable execution environment for reasoning tasks.

[0068] The reasoning instance management method of the embodiment of the application is further described below with actual application as an example.

[0069] The specific implementation process is as follows:

[0070] Reasoning engine monitoring: the reasoning engine runs in the reasoning instance, monitors the running state of the current reasoning task, including GPU resource usage and reasoning task performance.

[0071] State reporting: the reasoning engine generates corresponding messages (such as Normal, Unable To Respond, and GPU Error) according to the monitoring results, and sends them to the reasoning service through the network. The Unable To Respond message is equivalent to the first message, the GPU Error message is equivalent to the second message, and the Normal message is equivalent to the third message.

[0072] Reasoning service decision: after receiving the message, the reasoning service makes a decision according to the message content and the pre-defined strategy. For example:

[0073] If the Unable To Respond message is received, a suitable GPU is selected from the GPU resource pool to replace or expand.

[0074] If the GPU Error message is received, the resource switching operation is triggered, and the reasoning task is migrated to other GPUs.

[0075] If the Normal message is received, the task is continued to be assigned to the reasoning instance.

[0076] Resource switching: the reasoning service calls the API of Kubernetes to extend the GPU resources of the reasoning instance, migrates the reasoning task from the current GPU to other available GPUs, or reschedules the reasoning instance to other host computers.

[0077] State update: the reasoning service updates the state information of the reasoning instance, ensuring that the reasoning engine can continue to monitor the new resource state.

[0078] The processing flow of the reasoning service receiving different messages is described in detail as follows. First, the reasoning service receives the Unable To Respond message

[0079] The inference service first determines whether there is a same GPU card type in the GPU resource pool. If there is, GPU expansion is performed, the number of GPU cards mounted by the instance is changed from 1 to 2, the identification information of the added GPU card is synchronized to the inference instance, and the service state of the inference instance is modified to a switchable state. After the inference instance receives the information, the inference instance is triggered to restart and mount 2 GPU cards to perform inference instance online again. After the online is completed, the state of the inference instance is a normal running state, and the information is synchronized to the inference engine. At this time, the inference service sets the state of the added second GPU card to an occupied state.

[0080] If there is no available GPU card of the same type in the GPU resource pool, the inference service determines whether there is a GPU card of another type with larger memory. If there is a GPU card with larger memory, the inference service notifies the inference engine of the GPU card type information and identification information. After receiving the information, the inference engine judges and calculates, and if the GPU card can replace the previous GPU card and start the instance, the inference engine synchronizes the related information to the inference service. Then the inference service changes the service state of the inference instance to a switchable state, and sends information to the inference instance carrying the GPU card identification information and the GPU card model information. After receiving the switching information, the inference instance triggers the inference instance to perform the online process again, and mounts the new type of GPU card model to perform online. After the online is completed and the inference instance is running normally, the inference instance synchronizes the running state information to the inference engine. Finally, the inference service sets the state of the previous GPU card to an unoccupied state, and sets the state of the added GPU card with larger memory to an occupied state.

[0081] If there is neither a GPU card of the same type nor a GPU card with larger memory in the GPU resource pool, the inference service cannot perform the GPU switching operation, and synchronizes the information to the inference engine. In order to avoid problems in business response caused by large concurrent requests of the inference instance, the inference service performs a flow limiting operation on the corresponding inference instance, and sends the flow limiting information to the inference instance.

[0082] Second, the inference service receives a GPU Error message

[0083] If the GPU card currently mounted by the inference instance fails, causing the inference instance to fail to work normally, the inference engine sends a GPU Error message to the inference service, notifying the inference service that the current inference instance cannot continue to provide services. Subsequently, the inference service starts to query the GPU types and occupation in the GPU resource pool.

[0084] If there are unoccupied and allocable GPU resources in the current server, the inference service will directly reallocate a normal GPU card to the inference instance on the server and start the inference service. After the inference instance is successfully started and normally runs, the inference service will resume providing services to the outside.

[0085] If there are no available GPU resources in the current server, the inference service will expand its query range to other servers in the cluster to check whether there are available GPU resources. If there are no available GPU cards in other servers, the inference service will be unable to reallocate resources, and the inference instance will be unable to normally provide services.

[0086] If normal available GPU resources are found in other servers of the cluster, the inference service synchronizes the server information and GPU information to the corresponding inference instance and guides the inference instance to restart according to the new hardware information. Once the inference instance is successfully started and normally runs on the new GPU, the inference service will resume providing services to the outside.

[0087] Third, the inference service receives the Normal message

[0088] When the inference instance normally runs, the inference engine sends the Normal message every 1 minute to synchronize the current inference task state and GPU usage state to the inference service. Such a synchronization mechanism ensures that the inference service can know the running state and resource usage of the inference instance in real time, thereby performing effective resource management and scheduling.

[0089] In the embodiments of the application, the inference service adopts the following processing methods when facing the three situations of insufficient video memory, video memory failure and normal video memory:

[0090] Insufficient video memory: the inference service receives the Unable To Respond message from the inference engine, queries the GPU resource pool, and finds available GPU cards of the same type or with larger video memory. According to the resource pool situation, GPU expansion or replacement with a GPU card with larger video memory is automatically performed.

[0091] Video memory failure: the inference service receives the GPU Error message, triggers resource switching operation, and migrates the inference task to other healthy GPU. The faulty GPU is isolated and replaced to maintain the continuity of the service.

[0092] Normal video memory: the inference service receives the Normal message, continues to allocate the inference task to the inference instance that normally runs. The resource usage is monitored to prevent future possible resource shortage.

[0093] In summary, the inference service effectively handles the insufficient GPU memory, GPU memory failure and normal GPU memory through automatic and intelligent resource management, and improves the processing efficiency of the inference task and the stability of the system.

[0094] Based on the same inventive concept, the present application also provides an inference instance management device. The device is applied to an inference service, and a structural schematic diagram thereof is as shown in Figure 5 The device specifically comprises:

[0095] The query module 510 is configured to query a GPU resource pool when receiving a first message from an inference engine. The inference engine is configured to monitor GPU resource usage of an inference instance, and the first message is sent by the inference engine when the inference engine detects insufficient GPU memory of the inference instance.

[0096] The expansion module 520 is configured to send identification information of a GPU card of the same type as the current GPU card to the inference instance if there is a GPU card of the same type as the current GPU card in the GPU resource pool, so that the inference instance adds the GPU card to the current GPU card.

[0097] The replacement module 530 is configured to send identification information of a GPU card with larger GPU memory to the inference instance if there is no GPU card of the same type as the current GPU card in the GPU resource pool, but there is a GPU card with larger GPU memory, so that the inference instance replaces the current GPU card with the GPU card with larger GPU memory.

[0098] As a specific embodiment, the device further comprises:

[0099] The flow limiting module is configured to perform a flow limiting operation on the inference instance if there is no GPU card of the same type as the current GPU card in the GPU resource pool, and there is also no GPU card with larger GPU memory.

[0100] As a specific embodiment, the device further comprises:

[0101] The instance state updating module is configured to modify a state of the inference instance from a normal running state to a switchable state before the inference instance adds the GPU card to the current GPU card, and modify the state of the inference instance from the switchable state to the normal running state after the inference instance adds the GPU card to the current GPU card.

[0102] As a specific embodiment, the device further comprises:

[0103] The GPU card state updating module is configured to modify a state of the GPU card from an unoccupied state to an occupied state after the inference instance adds the GPU card to the current GPU card.

[0104] As a specific implementation, the apparatus further comprises:

[0105] The second query module is configured to query the GPU resource pool upon receiving a second message from the inference engine, the second message being sent by the inference engine when the inference engine detects a memory error of the inference instance;

[0106] The second replacement module is configured to, if there is an unoccupied GPU card on the current server in the GPU resource pool, send identification information of the unoccupied GPU card to the inference instance, so that the inference instance replaces the current GPU card with the unoccupied GPU card.

[0107] The third replacement module is configured to, if there is no unoccupied GPU card on the current server in the GPU resource pool, but there is an unoccupied GPU card on another server, send identification information of the unoccupied GPU card on the other server to the inference instance, so that the inference instance replaces the current GPU card with the unoccupied GPU card on the other server.

[0108] As a specific implementation, the apparatus further comprises:

[0109] The synchronization module is configured to, upon receiving a third message from the inference engine, update the locally recorded working state of the inference engine according to the third message, the third message being sent by the inference engine when the inference engine detects that the memory of the inference instance is normal.

[0110] Embodiments of the present application provide an electronic device, which can include a memory and one or more processors. The memory is configured to store computer program code, the computer program code including computer instructions. When the processor executes the computer instructions, the electronic device can perform various functions or steps of the above-mentioned method embodiments.

[0111] The structure of the electronic device can refer to the structure of the electronic device 100 shown in Figure 6 .

[0112] Exemplarily, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; or a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0113] The embodiment of the present application further provides a computer readable storage medium, which comprises computer instructions, and when the computer instructions run on an electronic device, the electronic device executes various functions or steps of the method embodiment.

[0114] The computer readable storage medium comprises but is not limited to any one of the following: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk and various storage program codes.

[0115] The embodiment of the present application further provides a computer program product, which, when running on a computer, enables the computer to execute various functions or steps of the method embodiment.

[0116] The electronic device, the computer readable storage medium and the computer program product provided by the embodiment of the present application are all used to execute the corresponding method provided above. Therefore, the beneficial effects achieved thereby can refer to the beneficial effects in the corresponding method provided above, which will not be described herein again.

[0117] The above is merely an embodiment of the present application and is not used to limit the present application. The present application can have various changes or replacements for those skilled in the art. Any modification, equivalent replacement, improvement and the like within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method for managing reasoning instances, characterized in that: The method is applied to an inference service, and the method includes: Upon receiving a first message from an inference engine, querying a GPU resource pool, wherein the inference engine is used to monitor GPU resource usage of an inference instance, the first message being sent by the inference engine when detecting insufficient video memory of the inference instance; If there is a GPU card of the same type as the current GPU card in the GPU resource pool, send the identification information of the GPU card to the inference instance, so that the inference instance adds the GPU card based on the current GPU card; If there is no GPU card of the same type as the current GPU card in the GPU resource pool, but there is a GPU card with larger video memory, the identification information of the GPU card with larger video memory is sent to the inference instance, so that the inference instance replaces the current GPU card with the GPU card with larger video memory.

2. The method according to claim 1, characterized in that The method further comprises: If the GPU resource pool does not contain a GPU card of the same type as the current GPU card, nor does it contain a GPU card with larger video memory, the inference instance is throttled.

3. The method according to claim 1, characterized in that The method further comprises: Before adding the GPU card to the inference instance based on the current GPU card, modify the state of the inference instance from the normal running state to the switchable state; After the GPU card is added to the inference instance based on the current GPU card, the state of the inference instance is changed from the switchable state to the normal running state.

4. The method according to claim 1, wherein The method further includes: after the inference instance is enabled to add the GPU card on the basis of the current GPU card, changing the state of the GPU card from an unoccupied state to an occupied state.

5. The method according to claim 1, wherein The method further comprises: Upon receiving a second message from the inference engine, querying the GPU resource pool, wherein the second message is sent by the inference engine when detecting a memory error in the inference instance; If there is an unoccupied GPU card on the current server in the GPU resource pool, the identification information of the unoccupied GPU card is sent to the inference instance, so that the inference instance replaces the current GPU card with the unoccupied GPU card; If there is no unoccupied GPU card on the current server in the GPU resource pool, but there is an unoccupied GPU card on other servers, the identification information of the unoccupied GPU card on the other server is sent to the inference instance, so that the inference instance replaces the current GPU card with the unoccupied GPU card on the other server.

6. The method according to claim 1, characterized in that The method further comprises: When a third message is received from the inference engine, the working status of the inference engine recorded locally is updated according to the third message, wherein the third message is sent when the inference engine detects that the inference instance display memory is normal.

7. A reasoning instance management device, characterized in that: The device is applied to reasoning services, and the device includes: a query module configured to query a GPU resource pool upon receiving a first message from an inference engine configured to monitor GPU resource usage of an inference instance, wherein the first message is sent by the inference engine when detecting insufficient video memory of the inference instance; The expansion module is configured to send the identification information of a GPU card of the same type as the current GPU card to the inference instance if the GPU resource pool contains the GPU card of the same type as the current GPU card, so that the inference instance can add the GPU card based on the current GPU card; The replacement module is used to send the identification information of the GPU card with larger video memory to the inference instance if there is no GPU card of the same type as the current GPU card in the GPU resource pool, but there is a GPU card with larger video memory, so that the inference instance replaces the current GPU card with the GPU card with larger video memory.

8. The device according to claim 7, characterized in that The device further comprises: The current limiting module is used to limit the current of the inference instance if there is no GPU card of the same type as the current GPU card in the GPU resource pool, nor a GPU card with larger video memory.

9. The device according to claim 7, characterized in that The device further comprises: The instance status update module is used to modify the status of the inference instance from a normal operating state to a switchable state before the GPU card is added to the inference instance on the basis of the current GPU card; and to modify the status of the inference instance from a switchable state to a normal operating state after the GPU card is added to the inference instance on the basis of the current GPU card.

10. An electronic device, characterized in that: include: A memory, one or more processors; the memory is coupled to the processor; wherein the memory stores computer program code, the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 6.

11. A computer-readable storage medium comprising computer instructions, characterized in that: When the computer instructions are executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 6.

12. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 6.

Images