Model reasoning service optimization method and related device

Abstract

The invention discloses a model reasoning service optimization method and a related device, relates to the field of artificial intelligence, and can determine a model version according to a model version parameter in a reasoning request. Firstly, according to a request identifier representing a reasoning request, whether reasoning result data corresponding to the request identifier exists or not is determined from a cache, and the reasoning result data is preferentially searched from the cache, so that the burden of a large number of repeated reasoning requests on a large model is reduced, and the processing efficiency of a thrust request is improved. When it is determined that no corresponding reasoning result data exists in the cache, a target service node is determined from all service nodes of a target model instance according to a load balancing strategy, the reasoning request is sent to the target service node to be processed, and the target model instance is a model instance represented by the model version. Balanced distribution of reasoning request processing is achieved, the phenomenon of overload or waste of node resources can be effectively reduced, and then the service efficiency of the model is effectively improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for optimizing model inference services. Background Technology

[0002] With the widespread application of large language models in fields such as natural language processing and generative AI (artificial intelligence), inference services based on large language models have developed rapidly. However, since large model inference frameworks often focus only on model loading and the inference service itself, there are still shortcomings in the management of access gateways, which can easily lead to overload or waste of some node resources, thereby reducing the service efficiency of the model. Summary of the Invention

[0003] In view of the above problems, this application provides a model inference service optimization method and related apparatus to improve resource balance and model inference efficiency. The specific solution is as follows:

[0004] The first aspect of this application provides a method for optimizing model inference services, including:

[0005] The model version is determined based on the model version parameter in the inference request;

[0006] Based on the request identifier that represents the inference request, determine from the cache whether there is inference result data corresponding to the request identifier;

[0007] When it is determined that the corresponding inference result data does not exist in the cache, a target service node is determined from each service node of the target model instance according to the load balancing strategy, and the inference request is sent to the target service node for processing. The target model instance is the model instance represented by the model version.

[0008] In one possible implementation, determining the target service node from each service node of the target model instance according to the load balancing strategy includes:

[0009] The load weight value of each service node is polled according to the scheduling algorithm to determine the service node allocation queue within a period, and the target service node is determined according to the order of service nodes in the service node allocation queue.

[0010] In one possible implementation, the process of determining the load weight value for each service node includes:

[0011] The load value of the service node is determined based on the index data characterizing the load of the service node.

[0012] The load value is weighted according to the weighting formula to obtain the initial weight value;

[0013] The initial weight value is normalized to obtain the load weight value.

[0014] In one possible implementation, determining the model version based on the model version parameter in the inference request includes:

[0015] The model version corresponding to the model version parameter is determined from the model version table.

[0016] In one possible implementation, the model inference service optimization method further includes:

[0017] The model versions in the model version table are updated based on the service discovery mechanism.

[0018] In one possible implementation, the process of retrieving the inference result data from the cache includes:

[0019] The signature of the inference request is used as the key, and the inference result data corresponding to the inference request is used as the value. The key and the value are stored in the cache, and the cache is cleared after the storage validity period expires.

[0020] In one possible implementation, the model inference service optimization method further includes:

[0021] If the cache reaches its upper limit before the storage validity period is reached, the inference result data will be cleaned up sequentially according to the number of times each inference result data is used.

[0022] A second aspect of this application provides a model inference service optimization apparatus, comprising:

[0023] The model version determination module is used to determine the model version based on the model version parameter in the inference request;

[0024] The result data lookup module is used to determine from the cache whether reasoning result data corresponding to the request identifier exists, based on the request identifier representing the reasoning request; and,

[0025] The inference request processing module is used to determine the target service node from each service node of the target model instance according to the load balancing strategy when it is determined that the corresponding inference result data does not exist in the cache, and send the inference request to the target service node for processing. The target model instance is the model instance represented by the model version.

[0026] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the model inference service optimization method of the first aspect or any implementation thereof.

[0027] A fourth aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0028] The memory is used to store computer programs;

[0029] The processor is used to execute the computer program so that the electronic device can implement the model inference service optimization method of the first aspect or any implementation thereof.

[0030] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the model inference service optimization method of the first aspect or any implementation thereof.

[0031] By employing the above technical solution, the model inference service optimization method provided in this application can determine the model version based on the model version parameter in the inference request. First, based on the request identifier representing the inference request, it checks whether inference result data corresponding to the request identifier exists in the cache, prioritizing the search for inference result data in the cache. This reduces the burden on large models from numerous repetitive inference requests and improves the processing efficiency of push requests. If it is determined that no corresponding inference result data exists in the cache, a target service node is determined from the service nodes of the target model instance according to a load balancing strategy, and the inference request is sent to the target service node for processing. The target model instance is the model instance represented by the model version. This balanced distribution of inference request processing effectively reduces the overload or waste of node resources, thereby improving the service efficiency of the model. Attached Figure Description

[0032] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0033] Figure 1 An architecture diagram of a model inference service optimization system provided in this application;

[0034] Figure 2 A structural diagram of a terminal provided in this application;

[0035] Figure 3 A structural diagram of a gateway provided in this application;

[0036] Figure 4 A flowchart of a model inference service optimization method provided in this application;

[0037] Figure 5 A flowchart for model version management provided for this application;

[0038] Figure 6 A flowchart for caching the inference result data provided in this application;

[0039] Figure 7 A flowchart of the reasoning request distribution provided for this application;

[0040] Figure 8 A flowchart illustrating the processing of inference requests provided for this application;

[0041] Figure 9 A structural diagram of a model inference service optimization device provided in this application;

[0042] Figure 10 This is a structural diagram of an electronic device provided in this application. Detailed Implementation

[0043] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0044] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0045] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0046] See Figure 1 , Figure 1A schematic diagram of the architecture of a model inference service optimization system is shown. The system may include a terminal 100, a gateway 200, and a server 300. The gateway 200 can provide the model inference service optimization method provided in this embodiment to one or more terminals, and the server 300 runs various service nodes of a large language model.

[0047] The terminal 100 may have a large language model application installed. The application and webpage can provide an interface. The terminal 100 can receive relevant parameters input by the user on the interface of the large language model and send the parameters to the gateway 200. The gateway 200 can obtain the processing result based on the received parameters and return the processing result to the terminal 100.

[0048] The following description Figure 1 The product form of the mid-terminal 100;

[0049] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, in-vehicle equipment, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0050] Figure 2 A schematic diagram of an optional hardware structure for terminal 100 is shown.

[0051] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), a headphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190, and other components. Those skilled in the art will understand that... Figure 2 These are merely examples of terminals or multi-functional devices and do not constitute a limitation on terminals or multi-functional devices. They may include more or fewer components than shown in the illustration, or combine certain components, or use different components.

[0052] The input unit 130 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the portable multi-functional device. Specifically, the input unit 130 may include a touchscreen 131 (optional) and / or other input devices 132. The touchscreen 131 can collect touch operations performed by the user on or near it (such as operations performed by the user using fingers, knuckles, styluses, or any suitable object on or near the touchscreen), and drive the corresponding connection devices according to a pre-set program. The touchscreen can detect the user's touch actions, convert the touch actions into touch signals and send them to the processor 170, and can receive and execute commands sent by the processor 170; the touch signal includes at least touch point coordinate information. The touchscreen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, various types of touchscreens, such as resistive, capacitive, infrared, and surface acoustic wave, can be used to implement the touchscreen. Besides the touchscreen 131, the input unit 130 may also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0053] Among them, the input device 132 can receive input data, etc.

[0054] The display unit 140 can be used to display information input by the user or information provided to the user, various menus of the terminal 100, interactive interfaces, file display, and / or playback of any multimedia file. In this embodiment, the display unit 140 can be used to display the interface of a large language model, processing results, etc.

[0055] The memory 120 can be used to store instructions and data. The memory 120 may primarily include an instruction storage area and a data storage area. The data storage area can store various types of data, such as multimedia files and text. The instruction storage area can store software units such as operating systems, applications, and instructions required for at least one function, or subsets or extended sets thereof. It may also include non-volatile random access memory. It provides the processor 170 with hardware, software, and data resources for managing the computing device, supporting control software and applications. It is also used for storing multimedia files, as well as storing running programs and applications.

[0056] The processor 170 is the control center of the terminal 100. It connects various parts of the terminal 100 via various interfaces and lines. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it performs various functions and processes data of the terminal 100, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 170. In some embodiments, the processor and memory can be implemented on a single chip; in some embodiments, they can also be implemented separately on independent chips. The processor 170 can also be used to generate corresponding operation control signals, send them to the corresponding components of the computing processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that the various functional modules therein perform corresponding functions, thereby controlling the corresponding components to act according to the instructions.

[0057] The radio frequency unit 110 (optional) can be used for receiving and transmitting signals during information transmission or calls. For example, it can receive downlink information from the base station and process it for the processor 170; additionally, it can transmit uplink data to the base station. Typically, the RF circuit includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the radio frequency unit 110 can also communicate wirelessly with network devices and other devices. This wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0058] In this embodiment of the application, the radio frequency unit 110 can send inference request data to the gateway 200 and receive the processing result sent by the gateway 200.

[0059] It should be understood that the radio frequency unit 110 is optional and can be replaced with other communication interfaces, such as a network port.

[0060] The terminal 100 also includes a power supply 190 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0061] Terminal 100 also includes an external interface 180, which can be a standard Micro USB interface or a multi-pin connector, which can be used to connect terminal 100 to other devices for communication or to connect a charger to charge terminal 100.

[0062] Although not shown, terminal 100 may also include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with various functions, etc., which will not be described in detail here. Some or all of the methods described below can be applied to, for example... Figure 2 In the terminal 100 shown.

[0063] The following description Figure 1 The product form of the Zhongjiewang 200;

[0064] Figure 3 A schematic diagram of the gateway 200 is provided, as follows: Figure 3 As shown, the gateway 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0065] Bus 201 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0066] The processor 202 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0067] Memory 204 may include volatile memory, such as random access memory (RAM). Memory 204 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0068] The memory 204 can be used to store software code related to the model inference service optimization method, and the processor 202 can execute the steps of the chip's model inference service optimization method, and can also schedule other units to achieve the corresponding functions.

[0069] It should be understood that the aforementioned terminal 100 and gateway 200 can be centralized or distributed devices. The processors (e.g., processor 170 and processor 202) in the aforementioned terminal 100 and gateway 200 can be hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the processor can be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0070] API gateways, as a core component of interface services, play a crucial role in AI inference services. However, existing general-purpose API gateways are insufficient to meet the specific needs of large-scale model inference services in terms of dynamic resource management and load optimization. For example, existing gateway routing strategies cannot be dynamically adjusted based on inference service performance metrics (such as GPU load and response time), potentially leading to resource overload or waste on some nodes. High-frequency, repetitive inference requests result in redundant consumption of computing resources.

[0071] To address the aforementioned problems, this application provides a method for optimizing model inference services. The method described below, with reference to the accompanying drawings, will be described in detail.

[0072] Reference Figure 4 , Figure 4 This application provides a flowchart illustrating a model inference service optimization method, as shown in the embodiments below. Figure 4 As shown in the embodiment of this application, a model inference service optimization method may include steps 401 to 403, which are described in detail below.

[0073] 401. Determine the model version based on the model version parameter in the inference request.

[0074] Specifically, refer to Figure 5 As shown, by adding a model version parameter to the model request (i.e., inference request), the user can specify the desired model version in the request. Then, by parsing the model version parameter in the inference request, the user's desired model version is obtained. If the model version parameter does not carry a model version identifier or the user has not used the model version parameter, the latest model version will be used by default.

[0075] Specifically, when determining the model version, the model version corresponding to the model version parameter can be determined from the model version table. Then, the inference request can be routed to the appropriate model instance for processing. The model version in the model version table can be updated through the service discovery mechanism.

[0076] In risk control applications, different model versions correspond to different strategy requirements. By integrating model version parameters, dynamic version switching is achieved, supporting the selection of lightweight or complex models based on risk level, thus achieving a balance between real-time performance and accuracy.

[0077] It is understood that those skilled in the art can adjust and select the settings of the above model version parameters as needed, and no restrictions are imposed here.

[0078] 402. Based on the request identifier representing the reasoning request, determine from the cache whether there is reasoning result data corresponding to the request identifier.

[0079] For details, please refer to Figure 6 As shown, the signature of the inference request can be used as the key, and the corresponding inference result data can be used as the value, stored in the cache, and the key and value can be cleared after the storage expiration period. Furthermore, if the cache reaches its limit before the storage expiration period, the inference result data will be cleared sequentially according to the number of times each inference result data has been used.

[0080] For example, by using the hash value of the inference request as its signature, it can be ensured that even requests with minimal changes in content can be identified as distinct requests. A hash table is then used as the cache structure, with the inference request signature as the key and the inference result data as the value. When the cache reaches its capacity limit, the least recently used cache item is replaced using the LRU (Least Recently Used) algorithm. Furthermore, before processing an inference request, the cache is checked for the inference result data for that request. If a cache hit occurs, the cached result is returned directly; otherwise, the request continues to be processed.

[0081] In intelligent customer service scenarios, user queries for frequently asked questions can trigger a large number of repetitive inference requests. By adopting this distributed caching mechanism to reduce invalid computation, the inference latency for frequently asked questions is reduced from an average of 500ms to 50ms.

[0082] 403. When it is determined that there is no corresponding inference result data in the cache, the target service node is determined from each service node of the target model instance according to the load balancing strategy, and the inference request is sent to the target service node for processing. The target model instance is the model instance represented by the model version.

[0083] Specifically, when the corresponding inference result data is not found in the cache, the inference request can be distributed to the appropriate target service node for inference processing based on the load of each service node. For example, refer to... Figure 7 As shown, based on the obtained model request (i.e. inference request), the inference request is routed to the corresponding target service node for inference processing through the request distribution strategy. Then, the monitoring status of each service node can be updated according to the response of the service node.

[0084] As can be seen above, this model inference service optimization method, through a caching mechanism based on inference requests, first retrieves inference result data in the cache, reducing repeated inference calls and improving service efficiency. Furthermore, by dynamically distributing inference requests based on the load of each service node, it balances the load on hardware resources such as GPUs, significantly reducing service latency and improving the processing efficiency of the model inference service. It also supports switching between multiple versions of large models, meeting the needs of more business scenarios.

[0085] In some embodiments, to effectively reduce the overload or waste of service node resources and thus improve the service efficiency of large models, the process of determining the target service node from each service node of the target model instance according to the load balancing strategy in the above embodiments may include:

[0086] The load weight value of each service node is polled according to the scheduling algorithm to determine the service node allocation queue within a period, and the target service node is determined according to the order of service nodes in the service node allocation queue.

[0087] Specifically, the load value of a service node can be determined based on metrics that characterize its load. Then, the load value is weighted according to a weighting formula to obtain an initial weight value. This initial weight value is then normalized to obtain the final load weight value.

[0088] Taking the concurrency of service nodes, average response time, video memory usage, and key-value cache usage as examples, the process of determining the target service node is explained:

[0089] Multiple inference service nodes for large models are deployed on the inference platform, for example:

[0090] Node A (GPU model A100, 80GB video memory);

[0091] Node B (GPU model V100, 32GB video memory);

[0092] Node C (CPU-only fallback, lowest performance).

[0093] Each node periodically (e.g., every second) updates its response status, including:

[0094] Current number of concurrent requests;

[0095] Average response time;

[0096] GPU memory usage;

[0097] KV Cache (key-value cache) utilization.

[0098] The load value ranges from [0, +∞). To avoid the denominator being zero, a constant 1 is introduced: weight = 1 / (1 + load), where load is the load value and weight is the initial weight value.

[0099] The process for determining the load weight value is as follows:

[0100] The current system has received the following load reports from the three nodes:

[0101]

[0102] The process for determining the load value is as follows:

[0103] load = 0.4*(concurrency / 20)+0.3*(response time / 1000)+0.2*video memory usage+0.1*KV cache usage.

[0104] Substituting the load value into the weight calculation formula, the initial weight values ​​for each service node are obtained as follows:

[0105]

[0106] Normalize the weights to use weighted round-robin: Total weight = 0.625 + 0.454 + 0.333 = 1.412.

[0107] The allocation probability, or weight, of each node is:

[0108]

[0109] Thus, for every 100 new inference requests: approximately 44 are allocated to node A, approximately 32 to node B, and approximately 24 to node C.

[0110] The Weighted Round Robin scheduling algorithm is used to poll nodes based on their weights within a cycle. For example, the scheduler maintains a request allocation queue as follows:

[0111] AAABABCAABCABCAAB ....

[0112] The request distribution strategy periodically updates the node status and weight, and adjusts the request distribution order accordingly to achieve dynamic, load-aware balanced scheduling.

[0113] It is understood that those skilled in the art can adjust the process for determining the above load weight values ​​as needed, and will not be elaborated here.

[0114] As a specific application of the aforementioned model inference service optimization methods, the overall system architecture of the large model inference service platform consists of a distributed large model inference gateway and a backend large model service. Large model inference service requests pass through the distributed large model inference gateway, where authentication, rate limiting, and intelligent routing to the backend large model service are performed. (Refer to...) Figure 8 As shown, the current inference gateway consists of the following modules:

[0115] Access control module: Verifies the security of requests, ensuring they comply with access policies. Provides multiple authentication methods. Prevents unauthorized requests from proceeding through subsequent processes.

[0116] Request control module: Performs initial request processing, supporting URL rewriting, request header modification, etc. It parses requests according to routing rules and maps them to the corresponding services. It can be integrated with custom plugins to flexibly define request content.

[0117] Proxy control module: Determines whether a request can access the target service. Supports features such as IP rate limiting, blacklists / whitelists, and rate limiting. Handles traffic control logic, such as rate limiting and circuit breaking policies.

[0118] Proxy Request Module: Implements the actual proxy forwarding, forwarding requests to upstream services. Supports functions such as load balancing and service discovery.

[0119] Response control module: Processes response content, supports modifying response headers (such as adding CORS cross-domain headers, removing sensitive information), and modifying response body content, such as compression, decoding, or inserting custom content.

[0120] Log monitoring module: Records request and response log information, including access time, latency, status code, etc. Supports log collection to external monitoring systems such as ELK.

[0121] The processing flow of inference requests for each module of the above inference gateway is as follows:

[0122] When a client sends an inference request to the gateway, it first passes through the access control module. If authentication is successful, processing continues. If it fails, an error response is returned directly. After successful access control verification, the request control module matches the request path, method, and other information, adjusting the request path, headers, or parameters according to configuration rules. After the request route is determined, the proxy control module checks the client's access permissions based on its identity and status. If the client meets the restriction rules, access is denied; otherwise, processing continues. After the request passes through the proxy control module, it enters the proxy request module, selects an appropriate node, forwards the request, and waits for a response. After the response is returned, the response control module modifies the response headers and body according to the configuration. Finally, the log monitoring module is responsible for recording the request and response status.

[0123] Based on the above modules, refer to Figure 8 As shown, a multi-model version management plugin and a distributed caching plugin are added to the request control module, and a dynamic routing module is added to the proxy request module.

[0124] The multi-model version management plugin controls model versions, allowing users to specify the appropriate major model version. By adding a model version parameter to the inference request, users can select the desired model version by specifying this parameter in the request.

[0125] Then, the model version parameter in the request is parsed. Based on this parameter, the model version table is consulted, and the request is routed to the corresponding model instance. The model version table is then automatically updated through the service discovery mechanism.

[0126] Distributed caching plugin: Used to store frequently used inference results, avoiding duplicate calls and improving overall inference throughput. By using the hash value of the inference request as a unique signature, it ensures that even requests with minor changes in content can be identified as distinct requests. A hash table is used as the cache structure, with the hash value of the request signature as the key and the request result as the value. When the cache reaches its capacity limit, the least recently used cache item is replaced using the LRU (Least Recently Used) algorithm. This way, before sending an inference request to the large model for processing, the cache is checked for the result of the request. If the cache is hit, the cached result is returned directly; if not, the request is processed. When a request is processed, its result is stored in the cache, using the request signature as the key. A timeout is set for cached items, which are automatically removed from the cache after expiration.

[0127] Dynamic routing plugin: Based on metrics such as GPU utilization and current node response time, it assigns dynamic weights to inference service nodes to achieve dynamic load balancing for downstream inference services. The specific implementation method can be found in the target service node determination process described in the above embodiments, and will not be repeated here.

[0128] The above describes a model inference service optimization method provided by the embodiments of this application. The following describes the apparatus for performing the above model inference service optimization method.

[0129] Please see Figure 9 , Figure 9 This is a schematic diagram of a model inference service optimization device provided in an embodiment of this application. Figure 9 As shown, the model inference service optimization device includes:

[0130] The model version determination module 901 is used to determine the model version based on the model version parameter in the inference request.

[0131] The result data lookup module 902 is used to determine from the cache whether reasoning result data corresponding to the request identifier exists, based on the request identifier of the representation reasoning request. And,

[0132] The inference request processing module 903 is used to determine the target service node from each service node of the target model instance according to the load balancing strategy when it is determined that there is no corresponding inference result data in the cache, and send the inference request to the target service node for processing. The target model instance is the model instance represented by the model version.

[0133] In one possible implementation, the process by which the inference request processing module 903 determines the target service node from among the service nodes of the target model instance according to the load balancing strategy includes:

[0134] The load weight value of each service node is polled according to the scheduling algorithm to determine the service node allocation queue within a period, and the target service node is determined according to the order of service nodes in the service node allocation queue.

[0135] In one possible implementation, the process of determining the load weight value of each service node in the inference request processing module 903 includes:

[0136] The load value of the service node is determined based on the indicator data that characterizes the load of the service node.

[0137] The load value is processed according to the weight calculation formula to obtain the initial weight value;

[0138] The initial weight values ​​are normalized to obtain the load weight values.

[0139] In one possible implementation, the process by which the model version determination module 901 determines the model version based on the model version parameter in the inference request includes:

[0140] The model version corresponding to the model version parameter is determined from the model version table.

[0141] In one possible implementation, the model version determination module 901 is also used for:

[0142] The model version in the model version table is updated based on the service discovery mechanism.

[0143] In one possible implementation, the process of retrieving inference result data from the cache in the result data lookup module 902 includes:

[0144] The signature of the inference request is used as the key, and the inference result data corresponding to the inference request is used as the value. The key and value are stored in the cache, and the cached key and value are cleared after the storage validity period expires.

[0145] In one possible implementation, the result data lookup module 902 is also used for:

[0146] If the cache reaches its limit before the storage validity period expires, the inference result data will be cleaned up sequentially according to the number of times each inference result data is used.

[0147] This application also provides a gateway in its embodiments. (See reference...) Figure 10 As shown, it illustrates a structural schematic diagram suitable for implementing the electronic device in the embodiments of this application. Figure 10The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0148] like Figure 10 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003. When the electronic device is powered on, the RAM 1003 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0149] Typically, the following devices can be connected to I / O interface 1005: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1007 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1008 including, for example, memory cards, hard drives, etc.; and communication devices 1009. Communication device 1009 allows electronic devices to exchange data via wireless or wired communication with other devices. Although Figure 10 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0150] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the model inference service optimization methods provided in this application.

[0151] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the model inference service optimization methods provided in this application.

[0152] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0153] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0154] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0155] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for optimizing model inference services, characterized in that, include: The model version is determined based on the model version parameter in the inference request; Based on the request identifier that represents the inference request, determine from the cache whether there is inference result data corresponding to the request identifier; When it is determined that the corresponding inference result data does not exist in the cache, a target service node is determined from each service node of the target model instance according to the load balancing strategy, and the inference request is sent to the target service node for processing. The target model instance is the model instance represented by the model version.

2. The model inference service optimization method according to claim 1, characterized in that, The step of determining the target service node from each service node of the target model instance according to the load balancing strategy includes: The load weight value of each service node is polled according to the scheduling algorithm to determine the service node allocation queue within a period, and the target service node is determined according to the order of service nodes in the service node allocation queue.

3. The model inference service optimization method according to claim 2, characterized in that, The process of determining the load weight value of each service node includes: The load value of the service node is determined based on the index data characterizing the load of the service node. The load value is weighted according to the weighting formula to obtain the initial weight value; The initial weight value is normalized to obtain the load weight value.

4. The model inference service optimization method according to any one of claims 1 to 3, characterized in that, Determining the model version based on the model version parameter in the inference request includes: The model version corresponding to the model version parameter is determined from the model version table.

5. The model inference service optimization method according to claim 4, characterized in that, Also includes: The model versions in the model version table are updated based on the service discovery mechanism.

6. The model inference service optimization method according to any one of claims 1 to 3, characterized in that, The process of retrieving the inference result data from the cache includes: The signature of the inference request is used as the key, and the inference result data corresponding to the inference request is used as the value. The key and the value are stored in the cache, and the cache is cleared after the storage validity period expires.

7. The model inference service optimization method according to claim 6, characterized in that, Also includes: If the cache reaches its upper limit before the storage validity period is reached, the inference result data will be cleaned up sequentially according to the number of times each inference result data is used.

8. A model inference service optimization device, characterized in that, include: The model version determination module is used to determine the model version based on the model version parameter in the inference request; The result data lookup module is used to determine from the cache whether there is reasoning result data corresponding to the request identifier based on the request identifier that represents the reasoning request; as well as, The inference request processing module is used to determine the target service node from each service node of the target model instance according to the load balancing strategy when it is determined that the corresponding inference result data does not exist in the cache, and send the inference request to the target service node for processing. The target model instance is the model instance represented by the model version.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the model inference service optimization method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the model inference service optimization method as described in any one of claims 1 to 7.

Images