Model service calling method and device, electronic equipment and storage medium

Abstract

The invention provides a model service calling method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: under the condition that a gateway service instance receives a model reasoning request, the gateway service instance determines an ith service node from at least two service nodes, sending a model reasoning request to the ith service grid corresponding to the ith service node; under the condition that the ith service grid receives the model reasoning request, the ith service grid detects whether a first model service instance can process the model reasoning request within a first waiting duration or not, an ith detection result is determined, and the first model service instance is a model service instance used for processing the model reasoning request on the ith service node; and the ith service grid sends an ith model service calling result to the gateway service instance based on the ith detection result, and different ith detection results correspond to different ith model service calling results. The method can reduce the processing time delay of the model reasoning request.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a model service calling method, device, electronic device and storage medium. Background Art

[0002] With the popularity of Model as a Service (MaaS), AI (Artificial Intelligence) model services provide online reasoning in the form of microservices. The online reasoning process is as follows: the requester sends a model reasoning request to the gateway service. The gateway service selects a model service instance according to the load balancing strategy and forwards the model reasoning request to the model service instance. The model service instance processes the model reasoning request and obtains the model reasoning result. However, the gateway service may have an uneven distribution problem, which leads to high latency in the processing of model reasoning requests. Summary of the invention

[0003] The present disclosure provides a model service invocation method, device, electronic device and storage medium, which can reduce the processing delay of model inference requests. The technical solution is as follows:

[0004] According to one aspect of the present disclosure, a model service calling method is provided, the method is applied to a model service calling system, the model service calling system includes at least one gateway service instance and at least two service nodes, each of the service nodes is deployed with a service grid and at least one model service instance; the method includes:

[0005] When the gateway service instance receives the model inference request, the gateway service instance determines an i-th service node from the at least two service nodes, and sends the model inference request to an i-th service grid corresponding to the i-th service node, where i is a positive integer;

[0006] When the i-th service grid receives the model inference request, the i-th service grid detects whether the first model service instance can process the model inference request within a first waiting time, and determines an i-th detection result, the first waiting time is much smaller than an average inference time of the model service instance, and the first model service instance is a model service instance on the i-th service node for processing the model inference request;

[0007] The i-th service grid sends the i-th model service call result to the gateway service instance based on the i-th detection result, and different i-th detection results correspond to different i-th model service call results. The gateway service instance is used to adopt different request processing methods according to different i-th model service call results.

[0008] According to another aspect of the present disclosure, a model service calling device is provided, the device is applied to a model service calling system, the model service calling system includes at least one gateway service instance and at least two service nodes, each of the service nodes is deployed with a service grid and at least one model service instance; the device includes:

[0009] a gateway processing module, configured to, when the gateway service instance receives a model inference request, determine the i-th service node from the at least two service nodes, and send the model inference request to the i-th service grid corresponding to the i-th service node, where i is a positive integer;

[0010] A first determination module is used for, when the i-th service grid receives the model inference request, detecting whether the first model service instance can process the model inference request within a first waiting time, and determining an i-th detection result, wherein the first waiting time is much smaller than an average inference time of the model service instance, and the first model service instance is a model service instance on the i-th service node for processing the model inference request;

[0011] The first sending module is used for the i-th service grid to send the i-th model service call result to the gateway service instance based on the i-th detection result, and different i-th detection results correspond to different i-th model service call results. The gateway service instance is used to adopt different request processing methods according to different i-th model service call results.

[0012] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the model service calling method as described above.

[0013] According to another aspect of the present disclosure, a computer program product is provided, the computer program product comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned model service calling method.

[0014] The beneficial effects brought by the technical solution provided by the embodiments of the present disclosure include at least:

[0015] The disclosed embodiment provides a model service calling method: by adding a service grid in the model calling system, when the gateway service instance receives a model inference request, it first sends the model inference request to the service grid, and the service grid analyzes whether the model service instance can process the model inference request in a short time based on a preset first waiting time, obtains a detection result, and feeds back unused model service call results to the gateway service instance according to different detection results, so that the gateway service instance can update subsequent request processing methods in time; this can avoid the problem of high latency in the processing of model inference requests caused by the need for the model inference request to wait for the previous model inference request to be processed before processing, thereby reducing the processing latency of the model inference request. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0017] Figure 1 The following is an architecture diagram of a model service calling system in the related art;

[0018] Figure 2 It is an architecture diagram of a model service calling system provided by an embodiment of the present disclosure;

[0019] Figure 3 A flowchart of a model service calling method according to an exemplary embodiment of the present disclosure is shown;

[0020] Figure 4 A flowchart of another model service calling method according to an exemplary embodiment of the present disclosure is shown;

[0021] Figure 5 A flowchart of another model service calling method according to an exemplary embodiment of the present disclosure is shown;

[0022] Figure 6 It is a structural diagram of a model service calling device provided by an embodiment of the present disclosure;

[0023] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0024] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0025] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0026] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative and not restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more". The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are for illustrative purposes only, and are not used to limit the scope of these messages or information.

[0027] The solution of the present invention is described below with reference to the accompanying drawings, and the technical solution provided by the embodiment of the present invention is explained in detail through specific embodiments and their application scenarios.

[0028] With the popularity of model as a service, AI model services provide online reasoning in the form of microservices. The architecture diagram of the model service calling system is as follows: Figure 1 As shown in the figure, the model service call system consists of multiple gateway service instances and multiple model service instances. The gateway service instance implements common capabilities such as authentication, current limiting, and load balancing, and the model service instance implements AI reasoning capabilities. The reasoning request processing process is as follows:

[0029] (1) The requester (such as an upstream service, browser, APP, etc.) sends a model inference request to any gateway service instance;

[0030] (2) The gateway service instance performs authentication and rate limiting on the model inference request. If authentication fails or rate limiting is triggered, the model inference request fails.

[0031] (3) If there is no problem, the gateway service instance selects a model service instance based on the load balancing strategy (such as random, least request, round-robin, etc.);

[0032] (4) The gateway service instance forwards the model inference request to the model service instance;

[0033] (5) The model service instance receives the model inference request and puts the model inference request into a waiting queue;

[0034] (6) The model service instance processes the model inference requests in the waiting queue in turn;

[0035] (7) The model service instance returns the model inference result of the model inference request to the gateway service instance;

[0036] (8) The gateway service instance returns the model inference results to the requester.

[0037] For example, if the upstream service sends a model inference request to gateway service instance 2, if the authentication and current limiting of gateway service instance 2 are passed, the model inference request can be sent to the model service instance m2-1 located on computing node 1 according to the load balancing strategy; after receiving the model inference request, the model service instance m2-1 will put the model inference request into the waiting queue, and the model service instance m2-1 will process the model inference requests in the waiting queue in turn until the model inference result corresponding to the model inference request is obtained, and then fed back to the upstream service through gateway service instance 2.

[0038] When using the model service call system in the related technology, no matter which load balancing strategy the gateway service instance adopts, since multiple gateway service instances will process different model inference requests at the same time, these gateway service instances may forward the model inference request to the same model service instance, causing multiple model inference requests to enter the same waiting queue. Since it is necessary to wait for the previous inference request to be processed, the processing delay of the model inference request is greatly increased on the basis of the long average processing time of the AI ​​model service itself.

[0039] In order to improve the high latency problem caused by the uneven distribution of gateway service instances in related technologies, the model service calling system in related technologies has been improved. Please refer to Figure 2 , which is an architecture diagram of a model service calling system provided by an embodiment of the present disclosure. The model service calling system includes at least one gateway service instance and at least two service nodes, and each service node is deployed with a service grid and at least one model service instance. Exemplarily, the service node can be implemented as a physical server; the service grid deployed on the service node can be implemented as a software application or a software service.

[0040] By adding a service grid to the model service call system and configuring the service grid to determine whether to wait for the model service instance to process the model inference request or refuse to process the model inference request based on the timeout period, the requester can avoid the problem of uneven random distribution of gateway service instances through a short timeout attempt strategy, thereby reducing the average latency.

[0041] Please refer to Figure 3 , which shows a flow chart of a model service calling method according to an exemplary embodiment of the present disclosure. Figure 2 The model service call system shown in the figure is used as an example to illustrate. Figure 3 As shown, the method includes:

[0042] Step 301, when the gateway service instance receives a model inference request, the gateway service instance determines the i-th service node from at least two service nodes, and sends the model inference request to the i-th service grid corresponding to the i-th service node, where i is a positive integer.

[0043] When the upstream service, browser or APP (Application) has a model inference requirement, a model inference request can be sent to any gateway service instance. The corresponding gateway service instance receives the model inference request sent by the requester, and first performs authentication verification and current limiting verification on the model inference request. The authentication verification is to determine whether the requester is allowed to call the model service instance to process the model inference request, and the current limiting verification is to determine whether the current traffic allows the gateway service instance to send a new model inference request to the service node; if both the authentication verification and the current limiting verification are passed, the gateway service instance can determine the i-th service node from multiple service nodes according to the load balancing strategy (for example, random, least request or patrol strategies), and send the model inference request to the i-th service grid corresponding to the i-th service node.

[0044] Step 302, when the i-th service grid receives a model inference request, the i-th service grid detects whether the first model service instance can process the model inference request within the first waiting time, and determines the i-th detection result. The first waiting time is much smaller than the average inference time of the model service instance. The first model service instance is the model service instance on the i-th service node used to process the model inference request.

[0045] Unlike related technologies, the gateway service instance does not directly send the model inference request to a specific model service instance, but to the service grid deployed on the service node. The service grid first analyzes and determines whether the request distribution of the gateway service instance is uniform, that is, whether the model service instance on the current service node can immediately process the model inference request, so as to further determine the model service call result. In a corresponding possible implementation, the i-th service grid receives the model inference request sent by the gateway service instance, and detects whether the first model service instance on the i-th service node can start processing the model inference request within the first waiting time, and the first model service instance is a model service instance with the function of processing the model inference request, so as to obtain the analysis result of whether the distribution of the gateway service instance is uniform, that is, to determine the i-th detection result.

[0046] Since the first waiting time is much shorter than the average inference time of a single model inference request processed by the model service instance, a shorter trial strategy can be used to quickly determine whether the model inference request can be processed quickly, thereby avoiding the problem of high latency in the model inference request due to the need for the model inference request to wait for the previous model inference request to be processed before processing.

[0047] Exemplarily, the first waiting time may be 2 ms, while the average inference time is 300 ms, and the first waiting time is much shorter than the average inference time.

[0048] Step 303: The i-th service grid sends the i-th model service call result to the gateway service instance based on the i-th detection result, and different i-th detection results correspond to different i-th model service call results.

[0049] After the i-th service grid determines the i-th detection result, different i-th model service call results can be sent to the gateway service instance according to different i-th detection results, so that the gateway service instance can execute different request processing methods for subsequent model inference requests according to different i-th model service call results.

[0050] To sum up, the embodiments of the present disclosure provide a model service calling method: by adding a service grid to the model calling system, when the gateway service instance receives a model inference request, it first sends the model inference request to the service grid, and the service grid analyzes whether the model service instance can process the model inference request in a short time based on a preset first waiting time, obtains the detection result, and feeds back unused model service call results to the gateway service instance according to different detection results, so that the gateway service instance can update the subsequent request processing method in time; it can avoid the problem that the model inference request needs to wait for the previous model inference request to be processed before processing, resulting in high latency in the processing of the model inference request.

[0051] There are two results for the i-th detection result: one can be processed immediately, and the other cannot be processed immediately. For the one that can be processed immediately, there is no waiting and high latency problem. The first model service instance processes the model inference request normally and obtains the model inference result. The corresponding service grid only needs to feed back the model inference result to the gateway service instance. For the one that cannot be processed immediately, in order to avoid the high latency problem caused by waiting, it is necessary to promptly feed back the uneven distribution problem to the gateway service instance, and the gateway service instance will allocate a new service node for the model inference request for processing.

[0052] Please refer to Figure 4 , which shows a flowchart of another model service calling method according to an exemplary embodiment of the present disclosure. Figure 2 The model service call system shown in the figure is used as an example to illustrate. Figure 4 As shown, the method includes:

[0053] Step 401, when the gateway service instance receives a model inference request, the gateway service instance determines the i-th service node from at least two service nodes, and sends the model inference request to the i-th service grid corresponding to the i-th service node, where i is a positive integer.

[0054] The implementation of step 401 may refer to step 301 , and this embodiment will not be described in detail here.

[0055] Step 402: When the i-th service grid receives the model inference request, the i-th service grid puts the model inference request into a first waiting queue and sets a first waiting time.

[0056] Among them, the first waiting time is configured by the gateway service instance, and the gateway service instance will send the model inference request and the first waiting time to the i-th service grid. After receiving the model inference request and the first waiting time, the corresponding i-th service grid first puts the model inference request into the first waiting queue, and sets the maximum waiting time of the model inference request in the first waiting queue to the first waiting time; so as to subsequently determine whether the model inference request in the first waiting queue has been consumed by the first model service instance within the first waiting time. Consumption means that the first model service instance obtains the model inference request from the first waiting queue and starts processing.

[0057] Exemplarily, the first waiting time may be 2 ms.

[0058] Step 403: If the model inference request is consumed by the first model service instance within the first waiting time, the i-th service grid determines that the i-th detection result is a processed model inference request.

[0059] If the model inference request can be consumed by the first model service instance within the first waiting time, that is, before the first waiting time is reached, the first model service instance obtains the model inference request from the first waiting queue and starts normal processing, then the i-th service grid determines that the i-th detection result is a processed model inference request. In this case, the normal request processing method can be followed to wait for the first model service instance to complete processing the model inference request and generate a model inference result to feed back to the gateway service instance.

[0060] Step 404: If the model inference request is not consumed by the first model service instance within the first waiting period, the i-th service grid determines that the i-th detection result is an unprocessed model inference request.

[0061] If the model inference request is not consumed by the first model service instance within the first waiting time, that is, after the first waiting time is reached, the first model service instance still does not obtain the model inference request from the first waiting queue for processing, then the i-th service grid determines that the i-th detection result is an unprocessed model inference request.

[0062] Exemplarily, the i-th service grid can set a timer with a duration equal to the first waiting duration. After the timer reaches the timing duration, if the model inference request has been taken away from the first waiting queue by the first model service instance, the i-th detection result is determined to be a processed model inference request; if the model inference request has not been taken away from the first waiting queue by the first model service instance, the i-th detection result is determined to be an unprocessed model inference request.

[0063] Step 405: If the i-th detection result is that the model inference request has been processed, the i-th service grid obtains the model inference result generated after the first model service instance processes the model inference request, and feeds back the model inference result to the gateway service instance.

[0064] If the i-th detection result is a processed model inference request, it means that the model inference request can be processed immediately by the first model service instance, and there is no processing delay waiting for the previous one or more requests. In this case, the normal request processing method can be followed to wait for the first model service instance to process the model inference request and generate a model inference result; the corresponding i-th service grid obtains the model inference result, and feeds back the model inference result to the gateway service instance; that is, when the i-th detection result is a processed model inference request, the i-th model service call result fed back by the i-th service grid to the gateway service instance is the model inference result corresponding to the model inference request; and after the gateway service instance receives the model inference result, the model inference result can be fed back to the requester who sent the model inference request.

[0065] Step 406: If the i-th detection result is that the model inference request is not processed, the i-th service grid feeds back waiting timeout information to the gateway service instance, and the waiting timeout information is used to indicate that the gateway service instance has failed to call the first model service instance.

[0066] If the i-th detection result is an unprocessed model inference request, it means that the first model service instance may be processing the previous one or more model inference requests and cannot immediately process the new model inference request currently received. If you continue to wait for the first model service instance to process the model inference request, the inference time required for model inference request processing is relatively long, and continued waiting may cause high latency problems. In order to avoid high latency problems, the gateway service instance should promptly assign a new service node to process the model inference request. In a corresponding possible implementation, if the i-th detection result is an unprocessed model inference request, the i-th service grid will feedback waiting timeout information to the gateway service instance. The waiting timeout information is used to indicate that the gateway service instance failed to call the first model service instance and needs to replace a new service node to process the model inference request. That is, in the case where the i-th detection result is an unprocessed model inference request, the i-th model service call result fed back by the i-th service grid to the gateway service instance is waiting timeout information.

[0067] Step 407: When the gateway service instance receives the waiting timeout information, the gateway service instance detects the number of requests for sending the model inference request.

[0068] In order to ensure high availability of the service, there are generally model service instances on multiple service nodes that can process the same model inference request. In this case, if the first model service instance on the current i-th service node cannot immediately process the current model inference request, the gateway service instance can try to select other service nodes to process the model inference request to avoid the delay caused by waiting. However, in order to avoid the high delay caused by an unlimited number of attempts, a target number of requests is also set, so that when the gateway service instance receives a waiting timeout message, the gateway service instance first detects the number of requests for sending the model inference request, and then determines how to process the model inference request by comparing the relationship between the number of requests and the target number of requests.

[0069] Step 408, when the number of requests is less than the target number of requests, the gateway service instance determines the i+1th service node from at least two service nodes, and sends a model inference request to the i+1th service grid corresponding to the i+1th service node to obtain the i+1th model service call result. The i-th service node is different from the i+1-th service node.

[0070] The target number of requests can be set by the business personnel according to the maximum acceptable number of attempts; or, the target number of requests can be set according to the target number of service nodes that can process model inference requests, and the target number of requests is less than or equal to the target number. For example, the target number of requests can be 4 times.

[0071] Optionally, after receiving the model inference request, the gateway service instance can determine the target number of service nodes that have the processing capability of the model inference request from at least two service nodes based on the model inference request, and then determine the target number of requests based on the target number of nodes; in order to avoid repeated attempts, the target number of requests is less than or equal to the target number of nodes. Exemplarily, if the gateway service instance determines that there are 4 service nodes that can process the model inference request, the target number of requests can be set to 4 times.

[0072] Optionally, when determining the target request number based on the number of target nodes, in order to avoid high latency problems caused by too many attempts due to a large number of target nodes, an attempt number threshold can be set; after determining the number of target nodes, if the number of target nodes is greater than the attempt number threshold, the attempt number threshold is determined as the target request number; if the number of target nodes is less than or equal to the attempt number threshold, the target node number can be determined as the target request number.

[0073] Exemplarily, the threshold of the number of attempts can be 10 times. If the number of target nodes is 20, the target request number is 10 times to avoid the high latency problem caused by too many attempts; if the number of target nodes is 4, the target request number is 4 to avoid multiple attempts at the same service node.

[0074] If the number of requests is less than the target number of requests, it means that the gateway service instance can try other service nodes to process the model inference request again. The corresponding gateway service instance determines the i+1th service node different from the i-th service node from at least two service nodes, and sends a model inference request to the i+1th service grid corresponding to the i+1th service node to obtain the i+1th model service call result.

[0075] It should be noted that the process of obtaining the i+1th model service call result is the same as the process of obtaining the i+1th model service call result, that is, steps 402 to 406 are repeatedly executed.

[0076] Step 409, when the number of requests reaches the target number of requests, the gateway service instance configures a second waiting time, determines the target service node from at least two service nodes, and sends a model inference request to the target service grid corresponding to the target service node to obtain the model inference result of the model service instance on the target service node for the model inference request, and the second waiting time is greater than the average inference time.

[0077] If the number of requests has reached the target number of requests, it means that the maximum number of attempts has been reached. It is possible that multiple service nodes are now under full load. Continuing to try will add more delays on top of the original high delay. In a possible implementation, the gateway service instance configures a second waiting time, and selects a target service node for processing the model inference request from at least two service nodes, and sends a model inference request to the target service grid corresponding to the target service node, so that the model service instance on the target service node processes the model inference request and obtains the model inference result.

[0078] Since the gateway service instance selects a target service node to process the model inference request, rather than the previous attempts, the second waiting time currently set is greater than the average inference time. Exemplarily, the second waiting time may be twice the average inference time.

[0079] The target service node may be a service node different from the i-th service node selected in the previous attempts, or may be the same service node as the i-th service node selected in the previous attempts.

[0080] After the target service grid corresponding to the target service node receives the model inference request, the specific processing process may also include the following steps 1 to 3.

[0081] Step 1: When the target service grid receives the model inference request, the target service grid puts the model inference request into a second waiting queue and sets a second waiting time, where the second waiting queue is different from the first waiting queue.

[0082] When the target service grid receives a model inference request, the target service grid can put the model inference request into the second waiting queue and set a second waiting time. Since the second waiting time corresponding to this model inference request is much longer than the first waiting time, in order to facilitate distinction, the target service grid creates different waiting queues to place the model inference request, and the corresponding second waiting queue is different from the first waiting queue.

[0083] Although the purpose of the target service grid receiving the model inference request this time is to process the model inference request, it cannot wait for the processing result indefinitely. A second waiting time is set accordingly to avoid invalid high latency caused by reasons such as failure to process the model inference request or failure of the model service instance to run.

[0084] Step 2: If the model inference request is consumed by the second model service instance within the second waiting time, the target service grid feeds back the model inference result to the gateway service instance. The second model service instance is the model service instance on the target service node used to process the model inference request.

[0085] If the model inference request is consumed by the second model service instance on the target service node within the second waiting time, the target service grid obtains the model inference result fed back by the second model service instance, and feeds back the model inference result to the gateway service instance, so that the gateway service instance can feed back the model inference result to the requester of the model inference request. Step 3: If the model inference request is not consumed by the second model service instance within the second waiting time, the target service grid feeds back the model call failure information to the gateway service instance.

[0086] On the contrary, if the model inference request is not consumed by the second model service instance within the second waiting period, it means that there may be problems such as model service instance operation failure, which causes the model inference request to be delayed. The corresponding target service grid feeds back the model call failure information to the gateway service instance so that the gateway service instance can select other service nodes, or directly feeds back the model inference request failure processing information to the requester.

[0087] In this embodiment, if the detection result is that the model inference request has been processed, the service grid can wait for the first model service instance to process the model inference request and generate a model inference result in a normal request processing manner, and then feedback the model inference result to the gateway service instance; if the detection result is that the model inference request has not been processed, the service grid will promptly feedback the waiting timeout information to the gateway service instance to remind the gateway service instance that the selected service node is heavily loaded, and the processing delay of the model inference request can be reduced by replacing the service node; moreover, a target number of requests is also set to avoid the gateway service instance from attempting to request different service nodes without limit, thereby avoiding the delay problem caused by unlimited attempts.

[0088] Based on the model service calling method described in the above embodiment, the improved model inference request processing process can be as follows to solve the high latency problem caused by the unbalanced distribution of gateway service instances:

[0089] (1) The requester (such as an upstream service, browser, APP, etc.) sends a model inference request to any gateway service instance;

[0090] (2) The gateway service instance performs authentication and rate limiting on the model inference request. If the authentication fails or rate limiting is triggered, the model inference request fails.

[0091] (3) If there is no problem, the gateway service instance sends the model inference request to the service grid and sets a very short maximum timeout waiting time t1 (much smaller than the average inference time). The waiting time t1 is set through the HTTP Header.

[0092] (4) After receiving the model inference request, the service grid puts the model inference request into the fast-failure queue waiting queue Q1 (i.e., the first waiting queue in the above embodiment).

[0093] (5) If the model service instance can process the model inference request before the waiting time t1 is reached, it will be processed normally to obtain the model inference result;

[0094] (6) If the model service instance fails to process the model inference request before the waiting time t1 is reached, the service grid returns a waiting timeout message to the gateway service instance;

[0095] (7) Repeat steps (3) to (6) N times. As long as one of them can be processed normally, the model inference request is processed and the process ends.

[0096] N times is the target number of requests in the above embodiment, that is, if the gateway service instance receives a timeout message and the number of requests has not reached N times, steps (3) to (6) can be repeated, and the gateway service instance can continue to send model inference requests to other service grids, and continue to try whether other service nodes can quickly start processing model inference requests.

[0097] (8) The gateway service instance sets a waiting time t2 that is twice the average inference time and sends the model request to the service grid;

[0098] If the process cannot be processed normally after N repetitions, the gateway service instance sets a second waiting time (t2) and sends the model inference request to one of the service grids.

[0099] (9) After receiving the model inference request, the service grid puts the model inference request into the regular waiting queue Q2 (i.e., the second waiting queue in the above embodiment), waiting for the model service instance on the service node to process it and feedback the model inference result.

[0100] Since t1*N<<average inference time (also called average inference duration), multiple attempts can be made to avoid the problem of increased latency of several hundred milliseconds in the case of load balancing conflicts among multiple gateway instances. Using the method disclosed in the present invention, in the same case, the increased latency is no more than 10 milliseconds.

[0101] Since the operating environment that the AI ​​model service relies on is more complex, the environment startup takes a long time. Using the same Docker image environment, the image size required by the AI ​​model service is about 10 times that of the business software service. And the AI ​​model service needs to load the model weight file. Therefore, the cold start time of the AI ​​model service reaches minutes; when the model service instance is used on demand, the cold start will bring a higher startup delay. In order to avoid the time-consuming cold start problem of on-demand use, the embodiment of the present disclosure provides a time-sharing multiplexing and dynamic loading solution, which can effectively improve the GPU utilization rate while solving the problems of long cold start time and long-term idle resources in the on-demand use scenario.

[0102] Please refer to Figure 5 , which shows a flowchart of another model service calling method according to an exemplary embodiment of the present disclosure. Figure 2 The model service call system shown in the figure is used as an example to illustrate. Figure 4 As shown, the method includes:

[0103] Step 501: For each service node, the service grid counts the GPU occupancy rates of at least two model service instances and the request information for processing model inference requests.

[0104] Considering that the model service instance needs to exclusively occupy GPU (Graphics Processing Unit) resources when running, and the GPU memory cost is higher than the CPU (Central Processing Unit) memory, in order to increase the number of model service instances that can be accommodated by a single machine, a single computing node can usually achieve a CPU memory and GPU memory capacity ratio of about 5:1, so as to store different model service instances on the CPU and GPU according to the needs and characteristics of each model service instance; that is, when multiple model service instances are deployed in a service node, the storage space of the service node is divided into GPU memory space and CPU memory space, and the space capacity ratio between the CPU memory space and the GPU memory space can be 5:1. Moreover, the service grid can dynamically adjust the target deployment location of each model service instance in the storage space according to the needs of the model service instance, deploying some model service instances in the CPU memory space and other model service instances in the GPU memory space, so that during actual operation, the running status of the model service instance can be checked through the service grid. For the model service instance that has been loaded (deployed) in the GPU memory space, the model inference request can be directly sent to the model service instance for processing; and for the model service instance stored (deployed) in the CPU memory space, the model service instance can be first loaded from the CPU memory space to the GPU memory space, and then the model inference request can be sent to the model service instance for processing. The shorter the time required to load the CPU memory space into the GPU memory space and check the running status, the cold start time of the AI ​​model service instance can be reduced.

[0105] Considering that it takes a certain amount of time to load the model service instance from the CPU memory space to the GPU memory space, in order to avoid the deployment of frequently called model service instances in the CPU memory space and increase the inference latency of the model service; for each service node, the service grid can collect the call frequency of each model service instance deployed in the service node, and preferentially deploy the model service instances with higher call frequency in the GPU memory space, and deploy the model service instances with lower call frequency in the CPU memory space. Specifically, a frequency threshold can be set. If the call frequency is higher than the frequency threshold, the model service instance is deployed in the GPU memory space. If the call frequency is lower than the frequency threshold, the model service instance is deployed in the CPU memory space. By dividing the storage space of the service node into two types of storage space, and the service grid dynamically adjusts the target deployment location of the model service instance, the high cost problem caused by the full use of the GPU memory space can be avoided; and the response latency problem of the model service instance with high frequency call can be reduced.

[0106] Optionally, the GPU memory space in the service node may be composed of multiple GPUs, and the CPU memory space may also be composed of multiple CPUs.

[0107] Considering that the model service instance will monopolize GPU resources when running, if a single GPU only stores one model service instance, it will obviously cause a waste of free storage space. In order to improve GPU utilization, a GPU memory sharing and time-sharing multiplexing mechanism is introduced to dynamically adjust the deployment mode of the model service instance on the GPU. That is, by deploying multiple model service instances in a single GPU (memory sharing), and the request times of multiple model service instances receiving model inference requests do not overlap (time-sharing multiplexing), while improving GPU utilization, it can also ensure that only one model service instance on a single GPU will receive a model inference request at the same time, meeting its exclusive GPU resource needs.

[0108] In order to achieve the effects of video memory sharing and time-sharing multiplexing, for each service node, the service grid needs to count the GPU occupancy rates of multiple model service instances on the service node in advance (that is, the GPU video memory usage of each model service instance during actual operation) and the request information for processing model inference requests; so as to analyze the way to deploy each model service instance based on the GPU occupancy rate and request information.

[0109] Exemplarily, the request information includes at least information such as the request time and processing time period of the historical model inference requests processed by each model service instance.

[0110] Step 502: The service grid determines a target model service combination from at least two model service instances based on the GPU occupancy rate and the request information, where the target model service combination includes at least two model service instances.

[0111] After collecting statistics such as GPU occupancy and request information, the service grid can automatically analyze multiple model service instances that can be deployed on the same GPU based on the GPU occupancy and request time indicated by the request information, and determine the multiple model service instances as the target model service combination.

[0112] Optionally, after determining the model service instance that needs to be deployed in the GPU memory space first according to the calling frequency, the service grid determines multiple model service instances that can be deployed on the same GPU, i.e., the target model service combination, according to the GPU occupancy rate and request information of the model service instance. Exemplarily, if there are 7 model service instances deployed in the service node, among which model service instance 1, model service instance 2, model service instance 3 and model service instance 4 are all model service instances with higher calling frequencies, and model service instance 5, model service instance 6 and model service instance 7 are model service instances with lower calling frequencies, model service instance 1, model service instance 2, model service instance 3 and model service instance 4 can be deployed in the GPU memory space, while model service instance 5, model service instance 6 and model service instance 7 can be deployed in the CPU memory space; further, the service grid determines the target model service combination that can be deployed in the same GPU according to the GPU occupancy rate and request information of model service instance 1, model service instance 2, model service instance 3 and model service instance 4.

[0113] Step 503: The service grid deploys the target model service combination on the target GPU in the service node. The GPU occupancy rate of the model service instances included in the target model service combination is less than or equal to the GPU capacity of the target GPU, and the request time overlap of the model inference requests is lower than a preset threshold.

[0114] Then, the determined target model service combination is deployed on the target GPU in the service node, and the sum of the GPU occupancy rates of the model service instances included in the target model service combination needs to be less than or equal to the GPU capacity of the target GPU, and the request time overlap of the model inference request is lower than the preset threshold. Exemplarily, the preset threshold can be 10%; or 0%. Since the request time overlap of multiple model service instances deployed on the target GPU is low, it is highly likely that only one model service instance will receive a model inference request at the same time, and therefore only one model service will process the inference request at the same time, which can achieve the effect of time-sharing multiplexing.

[0115] For example, if the GPU occupancy rate of model service instance 1 is 30%, and the request time is concentrated in the morning, the GPU occupancy rate of model service instance 2 is 70%, and the request time is concentrated in the afternoon, the GPU occupancy rate of model service instance 3 is 40%, and the request time is concentrated in the morning, the GPU occupancy rate of model service instance 4 is 20%, and the request time is concentrated in the afternoon; the GPU occupancy rate of model service instance 5 is 40%, and the request time is concentrated in the evening; then model service instance 1 and model service instance 2 can be deployed on one GPU, and model service instance 3, model service instance 4 and model service instance 5 can be deployed on one GPU.

[0116] Optionally, as the model inference requests in the model service call system change, the service grid on each service node needs to dynamically adjust the shared composition configuration, that is, dynamically adjust the target model service composition deployed on each GPU to reduce overlap and improve GPU utilization. For example, you can set a dynamic adjustment interval so that the target model service composition is dynamically adjusted based on historical inference requests at preset intervals.

[0117] For model service instances deployed on GPUs, if the model service instances have not been running for a long time, if they continue to occupy GPU resources, it will obviously waste GPU memory resources; the service grid will also periodically check idle model service instances to unload them to the CPU. In a corresponding possible implementation, the service grid counts the idle usage time of each third model service instance deployed on the target GPU, and the third model service instance belongs to the target model service combination; if there is a third model service instance whose idle usage time is greater than the target preset duration, the service grid controls the third model service instance to be unloaded from the target GPU to the target CPU; so as to deploy model service instances with high utilization rates on the GPU and improve the GPU memory utilization rate.

[0118] The deployment method of the model service instance in this embodiment is applied, and the process after the service grid receives the model inference request is as follows:

[0119] (1) The AI ​​service grid receives a model inference request;

[0120] (2) Based on the model service instance specified by the model inference request, the AI ​​service grid checks the running status of the model service instance;

[0121] (3) If the model service instance is in the ready (loaded) state, that is, it has been loaded into the GPU memory, the model inference request is sent to the model service instance for processing;

[0122] (4) If the model service is in the offload state, that is, it has been unloaded from the GPU memory to the CPU memory, the AI ​​service grid sends a load instruction to the model service instance;

[0123] (5) The model service instance performs a loading operation, loads the model weight file from the CPU memory to the GPU memory, and informs the AI ​​service grid that it has changed to the ready state;

[0124] (6) The AI ​​service grid sends the model inference request to the model service instance for processing;

[0125] (7) The AI ​​service grid periodically checks the idle status of each model service instance in the GPU memory, and sends an uninstallation instruction to the idle model service instance.

[0126] It should be noted that steps (1) to (7) correspond to the process of the AI ​​service grid calling the model service instance after receiving the model inference request.

[0127] In this embodiment, by adopting time-sharing multiplexing and dynamic loading solutions, GPU resources are fully utilized and GPU memory idleness is reduced. Among them, time-sharing multiplexing is achieved by setting multiple model service instances to share a GPU card, and controlling a GPU card to only process one model inference request at the same time. Moreover, the service grid can unload model service instances that have no requests for a long time from the GPU memory to the CPU memory, thereby achieving a higher service load for a single GPU card. Through these two methods, the GPU utilization rate can be effectively improved and the cost can be reduced. At the same time, the problem of long cold start time and long-term idle resources in the on-demand use scenario is solved.

[0128] In addition, the service grid also has diverse observation and monitoring capabilities. Since the service grid and the model service instance run on the same computing node, and the service grid can obtain all model inference requests and model inference results sent to the model service instance, the service grid can implement general diverse observation and monitoring capabilities based on type plug-ins. The specific process is as follows:

[0129] (1) Implement different plug-ins for different processing data types, inference service types, and application scenarios indicated by model inference requests;

[0130] (2) The service grid finds the corresponding plug-in for the received model inference request (the model inference request includes the data to be processed) and extracts the information of the data type;

[0131] (3) The service grid finds the corresponding plug-in according to the inference model type (such as based on transformers, ViT architecture, etc.) and obtains runtime monitoring indicators, such as the sparsity level of each layer, to monitor various operating indicators of each model service instance during runtime.

[0132] (4) The service grid finds the corresponding plug-in for the received model inference results and extracts information about the data type (such as text images, extracting image quality information).

[0133] The information obtained from the above monitoring can generate visual charts or graphics to be displayed in the front-end device interface for business personnel to view.

[0134] Please refer to Figure 6, which is a structural diagram of a model service invocation device provided by an embodiment of the present disclosure. The device is applied to a model service invocation system, which includes at least one gateway service instance and at least two service nodes, each of which is deployed with a service grid and at least one model service instance. For example, Figure 6 As shown, the device 600 includes:

[0135] The gateway processing module 601 is configured to, when the gateway service instance receives a model inference request, determine the i-th service node from the at least two service nodes, and send the model inference request to the i-th service grid corresponding to the i-th service node, where i is a positive integer;

[0136] A first determination module 602 is configured to detect, when the i-th service grid receives the model inference request, whether the first model service instance can process the model inference request within a first waiting time, and determine an i-th detection result, wherein the first waiting time is much smaller than an average inference time of the model service instance, and the first model service instance is a model service instance on the i-th service node for processing the model inference request;

[0137] The first sending module 603 is used for the i-th service grid to send the i-th model service call result to the gateway service instance based on the i-th detection result, and different i-th detection results correspond to different i-th model service call results. The gateway service instance is used to adopt different request processing methods according to different i-th model service call results.

[0138] Optionally, the first determining module 602 is further configured to:

[0139] The i-th service grid places the model inference request into a first waiting queue, and sets the first waiting time;

[0140] If the model inference request is consumed by the first model service instance within the first waiting period, the i-th service grid determines that the i-th detection result is that the model inference request has been processed;

[0141] If the model inference request is not consumed by the first model service instance within the first waiting period, the i-th service grid determines that the i-th detection result is that the model inference request is not processed.

[0142] Optionally, the first sending module 603 is further configured to:

[0143] If the i-th detection result is that the model inference request has been processed, the i-th service grid obtains the model inference result generated after the first model service instance processes the model inference request, and feeds back the model inference result to the gateway service instance;

[0144] If the i-th detection result is that the model inference request is not processed, the i-th service grid feeds back waiting timeout information to the gateway service instance, where the waiting timeout information is used to indicate that the gateway service instance failed to call the first model service instance.

[0145] Optionally, the device further comprises:

[0146] A detection module, configured to detect, when the gateway service instance receives the waiting timeout information, the number of requested times for sending the model inference request by the gateway service instance;

[0147] A second sending module is used for, when the number of requests is less than the target number of requests, the gateway service instance determines the i+1th service node from the at least two service nodes, and sends the model inference request to the i+1th service grid corresponding to the i+1th service node to obtain the i+1th model service call result, and the i-th service node is different from the i+1th service node;

[0148] The third sending module is used to configure a second waiting time for the gateway service instance when the number of requests reaches the target number of requests, and to determine a target service node from the at least two service nodes, and to send the model inference request to the target service grid corresponding to the target service node to obtain the model inference result of the model service instance on the target service node for the model inference request, wherein the second waiting time is greater than the average inference time.

[0149] Optionally, the device further comprises:

[0150] A setting module, configured to, when the target service grid receives the model inference request, cause the target service grid to place the model inference request into a second waiting queue and set the second waiting time, wherein the second waiting queue is different from the first waiting queue;

[0151] A fourth sending module, configured to cause the target service grid to feed back a model inference result to the gateway service instance if the model inference request is consumed by the second model service instance within the second waiting period, wherein the second model service instance is a model service instance on the target service node for processing the model inference request;

[0152] The fifth sending module is used to, if the model inference request is not consumed by the second model service instance within the second waiting period, the target service grid feeds back model call failure information to the gateway service instance.

[0153] Optionally, the device further comprises:

[0154] A second determination module is used for, when the gateway service instance receives a model reasoning request, the gateway service instance determines the target number of service nodes having the processing capability of the model reasoning request from the at least two service nodes;

[0155] The third determination module is used for the gateway service instance to determine the target request number based on the target node quantity.

[0156] Optionally, at least two model service instances are deployed in the service node, the storage space of the service node includes GPU memory space and CPU memory space, and the service grid is used to adjust the target deployment positions of the at least two model service instances in the storage space.

[0157] Optionally, the device further comprises:

[0158] A first statistics module, configured to, for each of the service nodes, count the GPU occupancy rates of the at least two model service instances and request information for processing model inference requests by the service grid;

[0159] A fourth determination module, configured for the service grid to determine a target model service combination from the at least two model service instances based on the GPU occupancy rate and the request information, wherein the target model service combination includes at least two model service instances;

[0160] A deployment module is used for the service grid to deploy the target model service combination on the target GPU in the service node, the GPU occupancy rate of the model service instances included in the target model service combination is less than or equal to the GPU capacity of the target GPU, and the request time overlap of the model inference request is lower than a preset threshold.

[0161] Optionally, the device further comprises:

[0162] A second statistics module is used for the service grid to count the idle usage time of each third model service instance deployed on the target GPU, and the third model service instance belongs to the target model service combination;

[0163] The unloading module is used to control the service grid to unload the third model service instance from the target GPU to the target CPU if the idle usage time of the third model service instance is greater than the target preset duration.

[0164] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

[0165] It should be noted that the electronic device provided by the exemplary embodiments of the present disclosure may be a gateway service instance in the above embodiments, or may be a service node in the above embodiments.

[0166] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.

[0167] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.

[0168] refer to Figure 7 , a block diagram of an electronic device 700 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0169] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0170] A plurality of components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 may be any type of device capable of inputting information to the electronic device 700, and the input unit 706 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 707 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 may include but is not limited to a disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0171] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, Figure 3 , Figure 4 , Figure 5 , Figure 6 The method shown may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 700 via ROM 702 and / or communication unit 709. In some embodiments, computing unit 701 may be configured to execute the computer program by any other suitable means (e.g., by means of firmware). Figure 3 , Figure 4 , Figure 5 , Figure 6 The method shown.

[0172] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0173] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0174] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0175] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0176] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0177] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A model service calling method, characterized in that: The method is applied to a model service calling system, the model service calling system includes at least one gateway service instance and at least two service nodes, each of the service nodes is deployed with a service grid and at least one model service instance; the method includes: When the gateway service instance receives the model inference request, the gateway service instance determines an i-th service node from the at least two service nodes, and sends the model inference request to an i-th service grid corresponding to the i-th service node, where i is a positive integer; When the i-th service grid receives the model inference request, the i-th service grid detects whether the first model service instance can process the model inference request within a first waiting time, and determines an i-th detection result, the first waiting time is much smaller than an average inference time of the model service instance, and the first model service instance is a model service instance on the i-th service node for processing the model inference request; The i-th service grid sends the i-th model service call result to the gateway service instance based on the i-th detection result, and different i-th detection results correspond to different i-th model service call results. The gateway service instance is used to adopt different request processing methods according to different i-th model service call results.

2. The method according to claim 1, characterized in that The i-th service grid detects whether the first model service instance can process the model inference request within the first waiting time, and determines the i-th detection result, including: The i-th service grid places the model inference request into a first waiting queue, and sets the first waiting time; If the model inference request is consumed by the first model service instance within the first waiting time, the i-th service grid determines that the i-th detection result is that the model inference request has been processed; If the model inference request is not consumed by the first model service instance within the first waiting period, the i-th service grid determines that the i-th detection result is that the model inference request is not processed.

3. The method according to claim 2, characterized in that The i-th service grid sends the i-th model service call result to the gateway service instance based on the i-th detection result, including: If the i-th detection result is that the model inference request has been processed, the i-th service grid obtains the model inference result generated after the first model service instance processes the model inference request, and feeds back the model inference result to the gateway service instance; If the i-th detection result is that the model inference request is not processed, the i-th service grid feeds back waiting timeout information to the gateway service instance, where the waiting timeout information is used to indicate that the gateway service instance failed to call the first model service instance.

4. The method according to claim 3, characterized in that The method further comprises: When the gateway service instance receives the waiting timeout information, the gateway service instance detects the number of requests for sending the model inference request; In a case where the number of requested times is less than the target number of requests, the gateway service instance determines an i+1th service node from the at least two service nodes, and sends the model inference request to an i+1th service grid corresponding to the i+1th service node to obtain an i+1th model service call result, the i-th service node being different from the i+1th service node; When the number of requests reaches the target number of requests, the gateway service instance configures a second waiting time, determines a target service node from the at least two service nodes, and sends the model inference request to the target service grid corresponding to the target service node to obtain the model inference result of the model service instance on the target service node for the model inference request, and the second waiting time is greater than the average inference time.

5. The method according to claim 4, characterized in that The method further comprises: When the target service grid receives the model inference request, the target service grid puts the model inference request into a second waiting queue and sets the second waiting time, where the second waiting queue is different from the first waiting queue; If the model inference request is consumed by the second model service instance within the second waiting time, the target service grid feeds back the model inference result to the gateway service instance, and the second model service instance is the model service instance on the target service node for processing the model inference request; If the model inference request is not consumed by the second model service instance within the second waiting period, the target service grid feeds back model call failure information to the gateway service instance.

6. The method according to claim 4, characterized in that The method further comprises: When the gateway service instance receives a model reasoning request, the gateway service instance determines the target number of service nodes having the processing capability of the model reasoning request from the at least two service nodes; The gateway service instance determines the target request number based on the target node quantity.

7. The method according to any one of claims 1 to 6, characterized in that: At least two model service instances are deployed in the service node, the storage space of the service node includes GPU memory space and CPU memory space, and the service grid is used to adjust the target deployment positions of the at least two model service instances in the storage space.

8. A model service calling device, characterized in that: The device is applied to a model service calling system, the model service calling system includes at least one gateway service instance and at least two service nodes, each of the service nodes is deployed with a service grid and at least one model service instance; the device includes: a gateway processing module, configured to, when the gateway service instance receives a model inference request, determine the i-th service node from the at least two service nodes, and send the model inference request to the i-th service grid corresponding to the i-th service node, where i is a positive integer; A first determination module is used for, when the i-th service grid receives the model inference request, detecting whether the first model service instance can process the model inference request within a first waiting time, and determining an i-th detection result, wherein the first waiting time is much smaller than an average inference time of the model service instance, and the first model service instance is a model service instance on the i-th service node for processing the model inference request; The first sending module is used for the i-th service grid to send the i-th model service call result to the gateway service instance based on the i-th detection result, and different i-th detection results correspond to different i-th model service call results. The gateway service instance is used to adopt different request processing methods according to different i-th model service call results.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images