Model reasoning optimization method, device, equipment, storage medium and program product

Abstract

The present disclosure provides a model reasoning optimization method, an optimization device, an equipment, a storage medium and a program product, which relate to the technical field of large language models. Among them, the model reasoning optimization method includes: in response to the received reasoning request of the model reasoning, the global scheduler determines the first execution instance based on the local perception strategy of the configured global hint tree, and the first execution instance can reuse the context cache of the reasoning request; the pre-filling task of the model reasoning is scheduled to the first execution instance, so as to perform the pre-filling operation based on the first execution instance to obtain a key value cache including the context cache; the decoding task of the model reasoning is scheduled to the second execution instance, so as to decode the key value cache including the context cache based on the second execution instance to obtain the reasoning result; the global scheduler feeds back the reasoning result to the request end. Through the technical solution of the present disclosure, the rationality of task allocation can be improved and the reasoning process can be made more efficient.

Description

Technical Field

[0001] The present disclosure relates to the technical field of large language models, and in particular to a model reasoning optimization method, a model reasoning optimization device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Large Language Models (LLMs) consume a lot of computing resources, storage resources, and energy during the reasoning process. Therefore, reasonable resource allocation is the key to ensuring that the model can achieve accurate reasoning. However, the current way of allocating resources for large language models usually adopts static resource allocation. This allocation method cannot dynamically adjust memory and computing resources according to the actual needs of the reasoning task, resulting in the inability to effectively utilize resources, which means that the reasoning performance of the model needs to be improved.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of the present disclosure is to provide a model reasoning optimization method, a model reasoning optimization device, an electronic device, a computer-readable storage medium and a computer program product, which at least to some extent overcome the problem that the reasoning performance of large language models in related technologies needs to be improved.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a model reasoning optimization method is provided, including: in response to a received inference request of model reasoning, a global scheduler determines a first execution instance based on a local perception strategy of a configured global hint tree, and the first execution instance reuses the context cache of the inference request; a pre-filling task of the model reasoning is scheduled to the first execution instance, so as to perform a pre-filling operation based on the first execution instance to obtain a key-value cache including the context cache; a decoding task of the model reasoning is scheduled to a second execution instance, so as to decode the key-value cache including the context cache based on the second execution instance to obtain an inference result; and the global scheduler feeds back the inference result to the request end.

[0007] In one embodiment of the present disclosure, in response to a received inference request for model inference, a global scheduler determines a first execution instance based on a local perception strategy of a configured global hint tree, including: in response to the inference request, the global scheduler parses a hint identifier of the inference request; searches for a tree node in the global hint tree that stores specified hint information, wherein the specified hint information matches the hint identifier; stores an instance location of a context cache of the inference request based on the tree node index; and perceives the resource environment and load information of the instance location based on the local perception strategy to determine the first execution instance based on the perception result.

[0008] In one embodiment of the present disclosure, the pre-filling task of the model reasoning is scheduled to the first execution instance to perform a pre-filling operation based on the first execution instance to obtain a key-value cache including the context cache, including: querying the reusable context cache based on the instance location as the first key-value cache; scheduling the pre-filling task to the first execution instance to perform an accelerated pre-filling operation on the reasoning request based on the first execution instance and the first key-value cache to obtain a second key-value cache; determining the key-value cache including the context cache based on the first key-value cache and the second key-value cache.

[0009] In one embodiment of the present disclosure, the pre-filling task is scheduled to the first execution instance to perform an accelerated pre-filling operation on the inference request based on the first execution instance and the first key-value cache to obtain a second key-value cache, including: performing correlation parsing on the context cache pair based on the inference request to obtain key-value pairs with different association relationships; and performing a pre-filling operation on the inference request by the first execution instance based on a reuse operation on the key-value pairs with different association relationships to obtain the second key-value cache.

[0010] In one embodiment of the present disclosure, before the global scheduler determines the first execution instance based on the local perception strategy of the configured global hint tree in response to the received inference request of the model inference, it also includes: registering the memory resources and computing power of each cloud node in the data center into the memory pool, generating a corresponding execution instance, and configuring the information of the execution instance in the global scheduler; building a tree framework based on a multi-layer radix tree structure; initializing the tree framework based on the information of the execution instance to obtain an initialized tree structure; configuring the initialized tree structure based on the mapping relationship between the hint information of the auxiliary reasoning and the context cache, and the execution instance storing the context cache, to obtain the global hint tree.

[0011] In one embodiment of the present disclosure, before the global scheduler determines the first execution instance based on the local awareness strategy of the configured global hint tree in response to the received inference request of the model inference, it also includes: detecting a key-value cache to be stored, determining the storage location of the key-value cache to be stored based on an index interface and a configured fixed amount of memory blocks; and establishing an index relationship between the key-value cache to be stored and the associated context cache based on the storage location.

[0012] In one embodiment of the present disclosure, the decoding task of the model inference is scheduled to a second execution instance, so as to decode the key value cache of the key value cache including the context cache based on the second execution instance to obtain an inference result, including: detecting the computing power of the registered execution instance based on the global scheduler; selecting the second execution instance based on the detection result; scheduling the decoding task to the second execution instance; and calling a transmission interface to transfer the key value cache including the context cache from the first execution instance to the second execution instance, so as to perform a decoding operation on the key value cache including the context cache based on the second execution instance to obtain the inference result.

[0013] In one embodiment of the present disclosure, before the global scheduler determines the first execution instance based on the local perception strategy of the configured global hint tree in response to a received inference request for model inference, it also includes: analyzing the scale and / or task characteristics of the inference request in response to the received inference request; pushing the inference request to the edge execution instance deployed on the edge node based on the analysis result, and / or pushing the inference request to the global scheduler so that the global scheduler can parse the inference request.

[0014] In one embodiment of the present disclosure, the inference request is pushed to an edge execution instance deployed on an edge node based on an analysis result, and / or the inference request is pushed to the global scheduler, including: the analysis result includes that the inference request is a delay-sensitive task, the pre-filled task is scheduled to the edge node, and the decoding task is scheduled to the global scheduler.

[0015] In one embodiment of the present disclosure, it also includes: during the execution of the pre-filling task and / or the decoding task, continuously monitoring the memory resources and computing power of the cloud node to adjust the task allocation based on the monitoring results.

[0016] According to another aspect of the present disclosure, a model reasoning optimization device is provided, including: a determination unit, for responding to a received inference request of model reasoning, and determining a first execution instance by a global scheduler based on a local perception strategy of a configured global hint tree, wherein the first execution instance reuses the context cache of the inference request; a first scheduling unit, for scheduling a pre-filling task of the model reasoning to the first execution instance, so as to perform a pre-filling operation based on the first execution instance to obtain a key-value cache including a context cache; a second scheduling unit, for scheduling a decoding task of the model reasoning to a second execution instance, so as to decode the key-value cache including the context cache based on the second execution instance to obtain an inference result; and a feedback unit, for the global scheduler to feed back the inference result to the requesting end.

[0017] According to another aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; the processor is configured to perform the above-mentioned model reasoning optimization method by executing the executable instructions.

[0018] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned model reasoning optimization method is implemented.

[0019] According to another aspect of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned model reasoning optimization method is implemented.

[0020] The model reasoning optimization solution provided by the embodiment of the present disclosure is that after receiving a model reasoning request, the global scheduler determines the first execution instance based on the configured global hint tree and local perception strategy. The global hint tree stores information including reasoning task type, historical data characteristics, model parameters and execution instance resource attributes. The local perception strategy considers the hardware resources and workload of the execution instance. The first execution instance uses the context cache to perform a pre-filling operation to process the new input data, generates a key-value cache containing the context cache, and schedules the decoding task of the model reasoning to the second execution instance. The second execution instance uses the key-value cache containing the context cache for decoding to obtain the reasoning result. By reusing the context cache, it is helpful to accelerate the data processing process and reduce the processing time. The global scheduler can accurately assign tasks to appropriate execution instances according to the global hint tree and local perception strategy, which can improve the rationality of task allocation and make the reasoning process more efficient to achieve reasoning optimization. In addition, separating the pre-filling and decoding tasks to different execution instances can be processed in parallel, which is also helpful to improve the overall processing speed.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0023] Figure 1 A schematic diagram showing a model reasoning optimization system in an embodiment of the present disclosure is shown;

[0024] Figure 2 A schematic flow chart of a model reasoning optimization method in an embodiment of the present disclosure is shown;

[0025] Figure 3 A schematic flow chart showing another model reasoning optimization method in an embodiment of the present disclosure;

[0026] Figure 4 A schematic flow chart showing another model reasoning optimization method in an embodiment of the present disclosure is shown;

[0027] Figure 5 A schematic diagram showing another model reasoning optimization system in an embodiment of the present disclosure;

[0028] Figure 6 A schematic flow chart showing another model reasoning optimization method in an embodiment of the present disclosure;

[0029] Figure 7 A schematic flow chart showing another model reasoning optimization method in an embodiment of the present disclosure;

[0030] Figure 8 A schematic diagram showing another model reasoning optimization system in an embodiment of the present disclosure;

[0031] Fig. 9 A schematic diagram of a model reasoning optimization device in an embodiment of the present disclosure is shown;

[0032] Fig.10 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0033] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0034] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0035] Large Language Models (LLMs) have become increasingly important due to their high performance in natural language processing tasks. However, the efficient service of LLMs faces key challenges in computing resource optimization, memory management, and inference speed due to the following defects.

[0036] First, the resources used by large language models are statically allocated, and it is impossible to dynamically adjust memory and computing resources according to the actual needs of the reasoning task, resulting in waste or shortage of resources.

[0037] Secondly, the use of state-agnostic reasoning services cannot effectively utilize the context information between requests, resulting in repeated consumption of computing resources.

[0038] In addition, the centralized reasoning architecture has limited scalability when processing large-scale or high-concurrency requests, and the scheduling strategy is too late and too simple, making it difficult to adapt to the growing computing needs and unable to achieve optimal resource allocation and task scheduling.

[0039] Based on the above problems, when processing LLM reasoning tasks, it is impossible to achieve the ideal speed and efficiency, which affects the user experience.

[0040] Figure 1 A schematic diagram of the structure of a model reasoning optimization system in an embodiment of the present disclosure is shown, including multiple terminals 120 and a server cluster 140.

[0041] The terminal 120 may be a mobile terminal such as a mobile phone, a game console, a tablet computer, an e-book reader, smart glasses, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a smart home device, an AR (Augmented Reality) device, a VR (Virtual Reality) device, or the terminal 120 may also be a personal computer (PC), such as a laptop computer and a desktop computer.

[0042] Among them, the terminal 120 may be installed with an application for providing model reasoning optimization.

[0043] The terminal 120 is connected to the server cluster 140 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0044] The server cluster 140 is a server, or is composed of several servers, or is a virtualization platform, or is a cloud computing service center. The server cluster 140 is used to provide background services for applications that provide model reasoning optimization. Optionally, the server cluster 140 undertakes the main computing work, and the terminal 120 undertakes the secondary computing work; or, the server cluster 140 undertakes the secondary computing work, and the terminal 120 undertakes the main computing work; or, the terminal 120 and the server cluster 140 adopt a distributed computing architecture for collaborative computing.

[0045] In some optional embodiments, the server cluster 140 is used to store model reasoning optimization models, etc.

[0046] Optionally, the client of the application installed in different terminals 120 is the same, or the client of the application installed on the two terminals 120 is the client of the same type of application of different control system platforms. Based on the different terminal platforms, the specific form of the client of the application may also be different, for example, the application client may be a mobile client, a PC client or a World Wide Web (Web) client.

[0047] Those skilled in the art will appreciate that the number of the above-mentioned terminals 120 may be more or less. For example, the above-mentioned terminal may be only one, or the above-mentioned terminals may be dozens or hundreds, or more. The embodiment of the present application does not limit the number and device type of the terminals.

[0048] Optionally, the system may also include a management device ( Figure 1(not shown), the management device is connected to the server cluster 140 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0049] Optionally, the above-mentioned wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0050] like Figure 2 As shown, a model reasoning optimization method according to an embodiment of the present disclosure includes:

[0051] Step S202, in response to the received inference request of the model inference, the global scheduler determines a first execution instance based on the local awareness strategy of the configured global hint tree, and the first execution instance can reuse the context cache of the inference request.

[0052] Among them, the global hint tree is used to store and organize various information related to model reasoning to assist task scheduling and resource allocation decisions. Each node in the tree stores specific hint information and a KV (key-value) cache index for context retrieval.

[0053] The local awareness strategy is a decision-making scheme based on the local resources and load conditions of the execution instance adopted by the global scheduler when allocating tasks. It takes into account the actual environment and status around each execution instance to ensure that tasks are allocated to the most appropriate execution unit.

[0054] The global scheduler is used to schedule the resources of cloud nodes in the data center.

[0055] In some embodiments, the first execution instance is a computing node or execution environment selected by the global scheduler for the pre-filled task of model reasoning according to the local awareness strategy of the global hint tree, and the first execution instance can be a memory instance.

[0056] In some embodiments, when the global scheduler receives an inference request for model inference, it first parses the hint identifier therein. The hint identifier contains key information of the request, such as the inference task type, etc. Based on the parsed hint identifier, the global hint tree is searched for a tree node storing the specified hint information matching it, and the instance location of the context cache of the inference request is obtained according to the index of the matching tree node found, and these instance locations point to the storage area storing the historical calculation results or intermediate data related to the current inference request.

[0057] Step S204: Schedule the pre-filling task of the model reasoning to the first execution instance, so as to perform the pre-filling operation based on the first execution instance to obtain a key-value cache including a context cache.

[0058] Among them, the context cache of the reasoning request can be understood as a cache mechanism that stores historical calculation results and intermediate data related to the reasoning request, which includes useful information generated by previous processing of similar reasoning requests or partial stages of the same reasoning request.

[0059] The Prefill task is used to preprocess the input data of the inference request and generate a richer key-value cache in combination with the reusable context cache to provide the necessary data support for subsequent decoding tasks.

[0060] In some embodiments, the queried context cache is analyzed to determine the portion thereof that is relevant to the current reasoning request, and the relevant key-value pairs are directly reused in the pre-fill operation, skipping the recalculation process of these data, thereby saving computing resources and time. In the process of processing new input data, the first execution instance generates new key-value pairs that record the results of processing the new input data, and merges these new key-value pairs with the key-value pairs reused from the context cache to form a key-value cache that includes the context cache.

[0061] Step S206, scheduling the decoding task of the model reasoning to the second execution instance, so as to decode the key-value cache including the context cache based on the second execution instance to obtain the reasoning result.

[0062] Among them, the second execution instance is a computing node used to execute the decoding task in model reasoning. It can share the same memory pool with the first execution instance, or be located in a different memory area.

[0063] The decoding task is the subsequent key stage in the model reasoning process. It is executed on the second execution instance and takes the key-value cache including the context cache generated by the pre-filled task as input. The reasoning result is obtained through a specific decoding algorithm and model structure.

[0064] In some embodiments, the second execution instance receives a key-value cache including a context cache. The decoding operation is performed using the cache data. Since the key-value cache has been enriched and optimized in the pre-filling stage, and contains a large amount of information related to the current reasoning request, the decoding process can use this information more efficiently. During the decoding process, the data in the key-value cache is processed according to the decoding algorithm to gradually restore the reasoning result.

[0065] Step S208: The global scheduler feeds back the inference result to the requesting end.

[0066] In this embodiment, after receiving a model inference request, the global scheduler determines the first execution instance based on the configured global hint tree and local perception strategy. The global hint tree stores information including the inference task type, historical data characteristics, model parameters, and execution instance resource attributes. The local perception strategy considers the hardware resources and workload of the execution instance. The first execution instance uses the context cache to perform a pre-filling operation to process the new input data, generates a key-value cache containing the context cache, and schedules the decoding task of the model inference to the second execution instance. The second execution instance uses the key-value cache containing the context cache for decoding to obtain the inference result. By reusing the context cache, it is helpful to accelerate the data processing process and reduce the processing time. The global scheduler can accurately assign tasks to appropriate execution instances according to the global hint tree and local perception strategy, which can improve the rationality of task allocation and make the inference process more efficient to achieve inference optimization. In addition, separating the pre-filling and decoding tasks to different execution instances can be processed in parallel, which is also helpful to improve the overall processing speed.

[0067] like Figure 3 As shown, in one embodiment of the present disclosure, in response to a received inference request for model inference, a global scheduler determines a first execution instance based on a local awareness strategy of a configured global hint tree, including:

[0068] Step S302: In response to the inference request, the global scheduler parses the hint identifier of the inference request.

[0069] In some embodiments, the global scheduler analyzes the inference request and extracts a prompt identifier therefrom. The prompt identifier may be an identifier of the request type (for example, text processing-related inference, image recognition-related inference, or other types), an identifier of the request source (a specific user, an application, etc.), a feature identifier of the input data (such as the language type of the text, the resolution range of the image, etc.), and an identifier of special attributes that may affect the processing method (such as requirements for real-time performance, special requirements for result accuracy, etc.).

[0070] Step S304: searching the global prompt tree for a tree node storing the specified prompt information, where the specified prompt information matches the prompt identifier.

[0071] In some embodiments, each node in the global hint tree stores certain hint information, which covers various possible reasoning scenarios, task characteristics, and related attributes. The scheduler tries to find a matching tree node by comparing the hint identifier of the reasoning request with the specified hint information stored in the tree node.

[0072] Step S306 : storing the instance location of the context cache having the inference request based on the tree node index.

[0073] In some embodiments, based on the determined matching tree node, the global scheduler can obtain the instance location of the context cache of the reasoning request according to the index information of the node. This instance location is a pointer or address to a storage area that stores context information related to the current reasoning request. The information stored in the context cache exists in the form of key-value pairs. This information is accumulated when similar reasoning requests were processed before, and has important reference value for the processing of the current reasoning request, which can help speed up the processing process.

[0074] Step S308: perceive the resource environment and load information of the instance location based on the local perception strategy, so as to determine the first execution instance based on the perception result.

[0075] In some embodiments, after obtaining the instance location of the context cache, the global scheduler uses a local perception strategy to perceive the resource environment and load information corresponding to the instance location. The local perception strategy perceives the computing power of the hardware device where the instance is located (such as the number of CPU cores and frequency, GPU performance parameters, etc.), available memory size, network bandwidth, and current workload (such as the number of tasks assigned to the instance, the degree of resource occupancy of these tasks, etc.). By comprehensively evaluating these resource environments and load information, the scheduler determines the most suitable first execution instance from all available execution instances. The selected first execution instance can both utilize the context cache and ensure sufficient resources and reasonable load.

[0076] In this embodiment, the context cache location is determined by searching for matching nodes in the global hint tree based on the hint identifier, and then the first execution instance is selected in combination with the local perception strategy. This allows accurate use of existing context cache information and reduces repeated calculations. At the same time, tasks are allocated according to the actual resource environment and load conditions of the instance, so that resources can be reasonably allocated, the reasoning speed is accelerated, and the user's waiting time is reduced.

[0077] In one embodiment of the present disclosure, a pre-filling task of model reasoning is scheduled to a first execution instance, so as to perform a pre-filling operation based on the first execution instance to obtain a key-value cache including a context cache, including:

[0078] A reusable context cache is queried based on the instance location as the first key-value cache; a pre-filling task is scheduled to the first execution instance to perform an accelerated pre-filling operation on the inference request based on the first execution instance and the first key-value cache to obtain a second key-value cache; a key-value cache including a context cache is determined based on the first key-value cache and the second key-value cache.

[0079] In some embodiments, a reusable context cache is searched in a corresponding storage area according to the previously determined instance location information. This instance location points to the historical data storage location related to the current reasoning request. The first execution instance analyzes the key-value pairs in the first key-value cache to determine which parts are related to the current reasoning request. For the related key-value pairs, they are directly reused in the pre-filling operation, skipping the recalculation process of these data. For the new input data part in the current reasoning request (i.e., the part not covered by the first key-value cache), the first execution instance uses the pre-filling algorithm to process it. In the process of processing the new input data, the first execution instance generates new key-value pairs, which record the results of processing the new input data.

[0080] In some embodiments, these new key-value pairs are merged with the key-value pairs reused from the first key-value cache to form a second key-value cache. During the merging process, the key-value pairs may be optimized, such as reorganizing them according to factors such as frequency of use and relevance of the key-value pairs to improve cache access efficiency and storage efficiency.

[0081] In this embodiment, by reusing the context cache (the first key-value cache), repeated calculation of existing calculation results is prevented, and the consumption of computing resources (such as the computing time of the CPU and GPU) is reduced. The accelerated pre-filling operation utilizes the information in the first key-value cache and skips some calculation steps, which can reduce the processing time of the pre-filling stage. Since the first key-value cache contains relevant information of previous processing of similar reasoning requests, this information is reused in the current reasoning request, which helps to maintain the consistency and accuracy of the reasoning process. In the process of generating the second key-value cache and integrating the final key-value cache, the optimized processing of data (such as organizing key-value pairs according to relevance and frequency of use) also helps to improve the accuracy of reasoning, so that the system can output more reliable reasoning results.

[0082] In one embodiment of the present disclosure, scheduling a pre-filling task to a first execution instance to perform an accelerated pre-filling operation on an inference request based on the first execution instance and a first key-value cache to obtain a second key-value cache includes:

[0083] Based on the inference request, the context cache pair is correlated and parsed to obtain key-value pairs with different association relationships; based on the reuse operation of the key-value pairs with different association relationships, the first execution instance pre-fills the inference request to obtain a second key-value cache.

[0084] In this embodiment, by performing relevance parsing and reuse on the context cache pairs to directly use the previous calculation results, the step of reprocessing these data is skipped, and it can be ensured that the information used in the pre-filling process is relevant to the current reasoning task and has a targeted reuse operation. Combined with the processing of new input data, the second key-value cache generated by the pre-filling operation contains more accurate and comprehensive information, providing better data support for subsequent reasoning steps, which helps to improve the accuracy of the reasoning results.

[0085] like Figure 4 As shown, in one embodiment of the present disclosure, before the global scheduler determines the first execution instance based on the local awareness strategy of the configured global hint tree in response to the received inference request of the model inference, it also includes:

[0086] Step S402, register the memory resources and computing capabilities of each cloud node in the data center into the memory pool, generate a corresponding execution instance, and configure the execution instance information in the global scheduler.

[0087] In some embodiments, in a data center, each cloud node has its own memory resources and computing power. First, the memory resources and computing power information of these cloud nodes are registered in a memory pool. The memory pool is a mechanism for uniformly managing memory resource allocation.

[0088] Through this registration process, corresponding execution instances are generated for each cloud node. These execution instances represent logical units that can execute tasks on cloud nodes. Then, the detailed information of these execution instances (including their memory resources, computing power and other related properties) is configured to the global scheduler. The global scheduler will use this information to make task allocation and scheduling decisions so that tasks can be reasonably allocated to the execution instances on each cloud node in the subsequent steps.

[0089] Step S404: construct a tree framework based on the multi-layer radix tree structure.

[0090] In some embodiments, a radix tree is a special data structure that can efficiently organize and store data, especially for data with a hierarchical structure. The multi-layer structure can divide and store information according to different dimensions or attributes, providing a basic architecture for subsequent storage and management of information related to execution instances and reasoning tasks.

[0091] Step S406: Initialize the tree framework based on the information of the execution instance to obtain an initialized tree structure.

[0092] In some embodiments, the constructed tree framework is initialized based on the execution instance information. This process may include storing relevant attributes of the execution instance (such as memory size, computing power type, etc.) in various nodes or levels of the tree framework according to certain rules. For example, according to the computing power of the execution instance, it can be placed in a specific level or branch of the tree to facilitate subsequent rapid search for suitable execution instances based on the computing requirements of the task. Through the initialization operation, the tree framework is given content related to the actual execution instance.

[0093] Step S408 : Based on the mapping relationship between the prompt information and the context cache, and the execution instance storing the context cache, an initialization tree structure is configured to obtain a global prompt tree.

[0094] In some embodiments, the initialization tree structure is further configured considering the mapping relationship between the prompt information and the context cache, as well as the execution instance information storing the context cache. The prompt information may include the type of reasoning task, the characteristics of the input data, the user's special requirements, etc. The prompt information is associated with the corresponding context cache and the execution instance storing these caches, and stored in the initialization tree structure.

[0095] In some embodiments, if an inference task is image recognition and the input is a color image of a specific resolution, then the prompt information related to this type of task, the context cache accumulated from previous processing of similar images, and the execution instance information storing the cache will all be integrated into the corresponding position of the tree structure. After such a configuration, the initialized tree structure becomes a global prompt tree, which can provide a comprehensive decision-making basis for the global scheduler when processing inference requests, help quickly find a suitable execution instance and use the context cache to accelerate the inference process.

[0096] In this embodiment, by registering the memory resources and computing power of each cloud node in the data center into the memory pool and generating execution instances to centrally manage resources, the global scheduler can better allocate tasks based on this information. In addition, the multi-layer radix tree structure can quickly locate and retrieve relevant information according to different attributes, reducing the time complexity in finding suitable execution instances and related cache information, and improving the response speed of the system. The generated global hint tree organically combines hint information, context cache and execution instances. When processing reasoning requests, the global scheduler can use the global hint tree to determine the most suitable execution instance and use the context cache to accelerate the reasoning process, which not only improves the processing speed of reasoning tasks, but also makes better use of historical data (context cache), further improving the accuracy and efficiency of reasoning, and enabling the entire data center to perform better when processing complex reasoning tasks.

[0097] In some embodiments, the global scheduler uses a multi-layer radix tree structure to build a global hint tree. Each node in the tree not only stores hint information, but also indexes the instance location of the relevant KV cache. When a new inference request arrives, the scheduler determines the first execution instance with the highest cache reuse potential by querying these trees. This strategy allows the system to intelligently assign requests to the most appropriate instances, thereby maximizing resource utilization and response speed.

[0098] like Figure 5 As shown, the present disclosure also provides a dependency framework to support and utilize dependencies between and within requests, so as to integrate a variety of existing inter-request and intra-request optimization technologies into a unified system.

[0099] To build a context cache on the parameter server (PD) shared inference instance, this can be achieved by using the index API, such as insert and match operations. In addition, to build discrete inference, the active key-value (KV) cache can be transferred from the Prefill instance to the Decode instance by calling the transfer API.

[0100] like Figure 5As shown in the figure, the dependency framework schedules PD (ParallelDistribution) instances, Prefill instances, Decode instances, and SP (Storage Provider) instances to the elastic memory pool based on the global scheduler and the global hint tree, and applies KV cache in combination with index operations to achieve inference optimization.

[0101] In one embodiment of the present disclosure, before the global scheduler determines the first execution instance based on the local awareness strategy of the configured global hint tree in response to the received inference request of the model inference, it also includes:

[0102] A key-value cache to be stored is detected, a storage location of the key-value cache to be stored is determined based on an index interface and a configured fixed amount of memory blocks; and an index relationship between the key-value cache to be stored and associated context information is established based on the storage location.

[0103] In some embodiments, fixed-size memory blocks are used to store the KV cache, and the allocation and release of these memory blocks are managed through simple API calls. By configuring an index mechanism, hint tags are mapped to corresponding KV cache locations, thereby quickly retrieving and updating the cache.

[0104] In some embodiments, large page technology can also be used to reduce memory fragmentation, and data transmission APIs can be optimized to reduce communication overhead, especially when processing KV cache transmission across instances. Due to the discreteness of the KV cache, only one memory block is transmitted for each network API call, which will result in too many network calls and increase communication overhead. By configuring the memory block aggregation strategy, multiple small memory blocks are aggregated into large memory blocks for transmission, reducing the number of network API calls. By designing a universal API, it can adapt to different network environments and hardware configurations, including networks of different speeds and different types of memory media. This not only improves the efficiency and scalability of the LLM service, but also significantly reduces latency and resource consumption through intelligent scheduling and memory management.

[0105] like Figure 6 As shown, in one embodiment of the present disclosure, the decoding task of the model inference is scheduled to the second execution instance, so as to decode the key-value cache of the key-value cache including the context cache based on the second execution instance to obtain the inference result, including:

[0106] Step S602: Detect the computing capability of the registered execution instance based on the global scheduler.

[0107] In some embodiments, the global scheduler is responsible for managing and monitoring the registered execution instances in the system. The global scheduler initiates a detection mechanism for the computing power of each execution instance. The computing power includes the number of CPU cores and frequency, the performance parameters of the GPU (if any), the current CPU and GPU load conditions, the amount of free memory, etc.

[0108] Step S604: Select a second execution instance based on the detection result.

[0109] In some embodiments, based on the detection results of the computing power of the execution instance, a suitable second execution instance is selected from it. The selection process will comprehensively consider the needs of the decoding task. For example, if the decoding task is to decode a complex deep learning model of a high-resolution image, it may be necessary to select an execution instance with powerful GPU computing power and sufficient memory to load the model and cache data. If it is a simple statistical analysis decoding of a large amount of text data, it may be more focused on selecting an execution instance with a large number of CPU cores, sufficient memory and a lower current load.

[0110] Step S606: Schedule the decoding task to the second execution instance.

[0111] In some embodiments, the global scheduler allocates and sends the decoding task to the selected second execution instance. This process involves the scheduling and delivery mechanism of the task. In addition, some parameters and information related to the decoding task may also be passed to the second execution instance, such as model configuration parameters, relevant descriptions of the input data, etc., so that the second execution instance can correctly prepare the decoding environment and perform the decoding operation.

[0112] Step S608: calling a transmission interface to transmit the key-value cache including the reuse context from the first execution instance to the second execution instance, so as to perform a decoding operation on the key-value cache including the reuse context based on the second execution instance to obtain an inference result.

[0113] In some embodiments, calling the transfer interface implements transfer of a key-value cache from a first execution instance to a second execution instance including reusing a context cache.

[0114] After receiving the key-value cache, the second execution instance performs a decoding operation. During the decoding process, the information in the key-value cache is used to gradually restore the inference result according to the predetermined decoding model and algorithm logic.

[0115] In this embodiment, the second execution instance is selected by detecting the computing power of the execution instance, ensuring that the decoding task is assigned to the execution environment that best suits its resource requirements, and the decoding task is specifically scheduled to the appropriate second execution instance, and the key-value cache including the context cache is transmitted to it. The second execution instance can make full use of the cache information for decoding, reducing the amount of data processing during the decoding process, thereby significantly improving the decoding speed and speeding up the execution of the entire reasoning process.

[0116] In one embodiment of the present disclosure, before the global scheduler determines the first execution instance based on the local awareness strategy of the configured global hint tree in response to the received inference request of the model inference, it also includes:

[0117] In response to the received inference request, the scale and / or task characteristics of the inference request are analyzed; based on the analysis result, the inference request is pushed to the edge execution instance deployed on the edge node, and / or the inference request is pushed to the global scheduler so that the inference request is parsed by the global scheduler.

[0118] In one embodiment of the present disclosure, based on the analysis results, the inference request is pushed to the edge execution instance deployed on the edge node, and / or the inference request is pushed to the global scheduler, including: the analysis results include that the inference request is a delay-sensitive task, the pre-filled task is scheduled to the edge node, and the decoding task is scheduled to the global scheduler.

[0119] In this embodiment, reasonable allocation is made according to the scale and characteristics of the inference request to prevent all tasks from rushing to the global scheduler or edge nodes. Small and real-time tasks are processed at the edge nodes, which is conducive to making full use of local resources at the edge. Large-scale and complex tasks are coordinated and processed by the global scheduler, which is conducive to better utilization of cloud resources and realizes optimal resource allocation.

[0120] In some embodiments, lightweight memory pool instances are deployed at edge nodes close to users to process small-scale or low-latency inference requests.

[0121] For requests that require a large amount of computing resources, the edge node can forward the task to the cloud-based system for processing.

[0122] The edge node and the cloud memory pool instance keep the KV cache data synchronized to ensure the consistency of context information.

[0123] For latency-sensitive inference tasks, the prefill stage is performed first at the edge node, and then the results are transmitted to the cloud for the decode stage.

[0124] The global scheduler can intelligently decide whether to execute tasks at the edge or in the cloud based on the characteristics of the tasks and the resource status of the edge nodes.

[0125] Data processed at edge nodes can reduce the amount of data transmitted to the cloud, thereby improving data security and privacy protection.

[0126] In this embodiment, the response time of inference requests is further reduced through the collaboration of edge computing and cloud computing, which is particularly suitable for application scenarios with high real-time requirements. Through data synchronization and intelligent task allocation, computing efficiency and data consistency are ensured, while the flexibility and reliability of the system are enhanced. The system can adapt to different workloads and performance requirements, whether in a cloud service platform or an edge computing environment. In addition, in the expansion plan, by processing part of the data at the edge node, the amount of data transmission is reduced, the security and privacy of the data are enhanced, and the integration of edge computing enables applications with high real-time requirements to be responded to quickly.

[0127] In one embodiment of the present disclosure, it also includes: during the execution of the pre-filling task and / or the decoding task, the memory resources and computing power of the cloud node are continuously monitored to adjust the task allocation based on the monitoring results.

[0128] In this embodiment, resources are adjusted according to actual needs by configuring the dynamic memory allocation and management mechanism of the memory pool.

[0129] If the cloud node that performs the pre-filling task has insufficient memory resources or overloaded computing power, first consider whether it is possible to optimize resources within the node. If internal optimization cannot meet the needs, find other idle or lightly loaded cloud nodes with sufficient memory and computing power. Migrate some pre-filling tasks to new nodes and redistribute input data and related computing tasks. During the migration process, ensure the integrity and consistency of the data, for example, through appropriate data transmission protocols and verification mechanisms, safely transfer unprocessed input data and generated intermediate results to the new node.

[0130] Regarding the problem of resource shortage in cloud nodes performing decoding tasks, similar to the pre-filling task, try to optimize within the node first. If the computing resources are tight, you can adjust the execution order of the decoding algorithm, give priority to the parts that have a greater impact on the results, or use some approximate algorithms to reduce the amount of calculation while ensuring that the quality of the decoding results is within an acceptable range. If the internal optimization effect is not good, find other suitable cloud nodes to share the task. Some subtasks in the decoding task (such as decoding of different data blocks or different parts) can be assigned to other nodes.

[0131] like Figure 7As shown, according to another embodiment of the present disclosure, the model reasoning optimization method is applied to a cloud service platform, the cloud service platform has multiple data centers, each data center is deployed with a high-performance GPU and CPU server cluster, and the model reasoning optimization method includes:

[0132] Step S702: deploying the inference system on the cloud service platform, with each data center configured with an elastic memory pool and an instance of the global scheduler.

[0133] Step S704, registering the memory resources and computing capabilities of each server into the memory pool, and establishing a resource index.

[0134] Step S706: The global scheduler is configured with the memory pool instance information of each data center and initializes the global hint tree.

[0135] Step S708, when the user submits an inference request, the request is first sent to the global scheduler, which analyzes the characteristics of the request, such as the hint prefix of the request, and searches for matching nodes in the global hint tree to determine the best memory pool instance.

[0136] Step S710 , the memory pool checks whether the requested hint prefix exists in the index, and if so, retrieves the corresponding KV cache.

[0137] Step S712, according to the scheduling result, the prefill task is assigned to the instance with sufficient memory resources to generate the KV cache, the decode task is assigned to the instance with stronger computing power, and the output is generated using the retrieved KV cache.

[0138] Step S714: The memory pool dynamically adjusts memory allocation based on the retrieved KV cache to optimize storage and access of the KV cache.

[0139] Step S716: The scheduler monitors the load of each instance in real time, dynamically adjusts task allocation, and achieves load balancing.

[0140] Step S718: output the inference result and return the inference result to the requesting end through the scheduler.

[0141] In this embodiment, through dynamic resource allocation and intelligent scheduling, the cloud platform system can utilize computing and memory resources more efficiently and reduce resource waste. The context cache and separated reasoning architecture significantly reduce repeated calculations and speed up reasoning, thereby reducing user waiting time. By separating tasks between Prefill-only and Decode-only instances, the system can focus on computing optimization at each stage and improve overall computing efficiency. The global scheduler monitors and adjusts task allocation in real time to achieve load balancing and avoid situations where some nodes are overloaded while other nodes are idle.

[0142] like Figure 8 As shown, according to an inference optimization system of another embodiment of the present disclosure, a global scheduler receives an inference request and matches the optimal execution (memory) instance based on a local awareness strategy and a global hint tree.

[0143] The prefill task is scheduled to the first execution (Prefill instance), and the decoding task is scheduled to the second execution instance (Decode instance). Combined with the PD-Colocated (Parallel Data Colocation) instance, the reasoning operation is performed to obtain the reasoning result, and the global hint tree is updated based on the K cache generated by the reasoning result.

[0144] Each instance structure includes a CPU and a GPU. The GPU is packaged with DRAM and HBM. The CPU memory pool optimizes memory management from the perspective of CPU memory access and use. In a multi-tasking environment, multiple processes or threads may compete for memory resources at the same time, and the CPU needs to efficiently obtain and process these memory data. The CPU memory pool can design the size and allocation strategy of the memory block based on factors such as the CPU cache structure and instruction execution mode. The historical KV cache and active KV cache are stored in DRAM and HBM respectively.

[0145] In order to achieve sequential parallelism (SP) reasoning, the Transfer API can be used to exchange the output of the attention mechanism between different instances, that is, to transfer KV. The general distributed transmission API simplifies the data transmission between different instances, hides the heterogeneity of the underlying hardware, and implements it seamlessly within the system.

[0146] In this embodiment, the distributed memory pool and the global scheduler support horizontal expansion of the system and can adapt to growing computing needs and high-concurrency scenarios. In addition, the dynamic memory management mechanism of the memory pool optimizes memory usage, reduces memory fragmentation, and improves memory access speed. By improving resource utilization and optimizing computing processes, the cloud platform system helps to reduce the operating costs of data centers. Whether in a large-scale cloud computing environment or in an edge computing environment close to users, the cloud platform system can provide efficient services. Fast response and efficient reasoning services can improve the end-user experience and satisfaction, so that the configuration idea cloud platform system not only promotes the development of large-scale language model service technology, but also provides a practical solution for the actual deployment and operation of artificial intelligence applications.

[0147] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0148] Refer to the following Fig. 9 To describe the model reasoning optimization device 900 according to an embodiment of the present disclosure. Fig. 9 The model reasoning optimization device 900 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0149] The model reasoning optimization device 900 is presented in the form of a hardware module or a software module. The components of the model reasoning optimization device 900 may include but are not limited to: a determination unit 902, which is used to respond to the received reasoning request of the model reasoning, and the global scheduler determines the first execution instance based on the local perception strategy of the configured global hint tree, and the first execution instance can reuse the context cache of the reasoning request; a first scheduling unit 904, which is used to schedule the pre-filling task of the model reasoning to the first execution instance, so as to perform the pre-filling operation based on the first execution instance to obtain the key value cache including the context cache; a second scheduling unit 906, which is used to schedule the decoding task of the model reasoning to the second execution instance, so as to decode the key value cache including the context cache based on the second execution instance to obtain the reasoning result; a feedback unit 908, which is used to feedback the reasoning result to the request end by the global scheduler.

[0150] like Fig.10 As shown, the electronic device 1000 is in the form of a general computing device. The components of the electronic device 1000 may include but are not limited to: the at least one processing unit 1010, the at least one storage unit 1020, and a bus 1030 connecting different system components (including the storage unit 1020 and the processing unit 1010).

[0151] The storage unit stores program codes, which can be executed by the processing unit 1010, so that the processing unit 1010 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 1010 can perform the following steps: Figure 2 The scheme described in .

[0152] The storage unit 1020 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 10201 and / or a cache storage unit 10202 , and may further include a read-only storage unit (ROM) 10203 .

[0153] The storage unit 1020 may also include a program / utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0154] Bus 1030 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0155] The electronic device 1000 may also communicate with one or more external devices 1070 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1000, and / or may communicate with any device that enables the electronic device 1000 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1050. In addition, the electronic device 1000 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1060. As shown, the network adapter 1060 communicates with other modules of the electronic device 1000 via a bus 1030. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0156] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or an electronic device, etc.) to execute the method according to the implementation of the present disclosure.

[0157] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on an electronic device, the program code is used to enable the electronic device to perform the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0158] According to the program product for implementing the above method according to the embodiment of the present disclosure, it can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0159] The program product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0160] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0161] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0162] The program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the IoT terminal, as an independent software package, partially on the user computing device and partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0163] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0164] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0165] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or an electronic device, etc.) to execute the method according to the implementation of the present disclosure.

[0166] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A model reasoning optimization method, characterized in that: include: In response to a received inference request for model inference, a global scheduler determines a first execution instance based on a local perception strategy of a configured global hint tree, including: in response to the inference request, the global scheduler parses a hint identifier of the inference request, the hint identifier includes at least one of an identifier of a request type of the inference request, an identifier of a request source, a feature identifier of input data, and a special attribute identifier; searching for a tree node storing specified hint information in the global hint tree, the specified hint information matches the hint identifier, the global hint tree is based on a mapping relationship between hint information for auxiliary inference and a context cache, and an execution instance storing the context cache, to configure an initialization tree structure to obtain the result; The tree structure is initialized by initializing the tree framework based on the information of the execution instance, wherein the prompt information includes the task characteristics of the reasoning scenario and / or the reasoning task, and the global prompt tree is used to store at least one of the reasoning task type, model parameters and execution instance resource attributes; based on the tree node index, the instance location of the context cache of the reasoning request is stored; the global scheduler perceives the resource environment and load information of the instance location based on the local perception strategy to determine the first execution instance based on the perception result, the first execution instance reuses the context cache of the reasoning request, and the global scheduler uses the execution instance resource attributes to allocate the reasoning request to the first execution instance on the cloud node; Scheduling the pre-filling task of the model reasoning to the first execution instance, so as to perform a pre-filling operation based on the first execution instance to obtain a key-value cache including the context cache; Scheduling the decoding task of the model inference to the second execution instance, so as to decode the key-value cache including the context cache based on the second execution instance to obtain the inference result, including: detecting the computing power of the registered execution instance based on the global scheduler; selecting the second execution instance based on the detection result; scheduling the decoding task to the second execution instance; and calling the transmission interface to transfer the key-value cache including the context cache from the first execution instance to the second execution instance, so as to perform a decoding operation on the key-value cache including the context cache based on the second execution instance to obtain the inference result; The global scheduler feeds back the inference result to the request end.

2. The model reasoning optimization method according to claim 1, characterized in that: Scheduling the pre-filling task of the model reasoning to the first execution instance, so as to perform a pre-filling operation based on the first execution instance to obtain a key-value cache including the context cache, including: Query the reusable context cache based on the instance location as a first key value cache; Scheduling the pre-filling task to the first execution instance to perform an accelerated pre-filling operation on the inference request based on the first execution instance and the first key-value cache to obtain a second key-value cache; A key-value cache including the context cache is determined based on the first key-value cache and the second key-value cache.

3. The model reasoning optimization method according to claim 2, characterized in that: Scheduling the pre-fill task to the first execution instance to perform an accelerated pre-fill operation on the inference request based on the first execution instance and the first key-value cache to obtain a second key-value cache includes: Performing correlation analysis on the context cache pair based on the inference request to obtain key-value pairs with different association relationships; Based on the reuse operation of the key-value pairs of different association relationships, the first execution instance performs a pre-filling operation on the inference request to obtain the second key-value cache.

4. The model reasoning optimization method according to claim 1, characterized in that: Before the global scheduler determines the first execution instance based on the local awareness strategy of the configured global hint tree in response to the received model inference request, the method further includes: Register the memory resources and computing power of each cloud node in the data center into the memory pool, generate a corresponding execution instance, and configure the information of the execution instance in the global scheduler; The tree framework is built based on a multi-layer radix tree structure.

5. The model reasoning optimization method according to claim 4, characterized in that: Before the global scheduler determines the first execution instance based on the local awareness strategy of the configured global hint tree in response to the received model inference request, the method further includes: Detecting that there is a key-value cache to be stored, determining a storage location of the key-value cache to be stored based on an index interface and a configured fixed amount of memory blocks; and An index relationship between the key-value cache to be stored and the associated context cache is established based on the storage location.

6. The model reasoning optimization method according to claim 1, characterized in that: Before the global scheduler determines the first execution instance based on the local awareness strategy of the configured global hint tree in response to the received model inference request, the method further includes: In response to the received reasoning request, analyzing the scale and / or task characteristics of the reasoning request; The inference request is pushed to an edge execution instance deployed on an edge node based on the analysis result, and / or the inference request is pushed to the global scheduler so that the global scheduler can parse the inference request.

7. The model reasoning optimization method according to claim 6, characterized in that: Pushing the inference request to an edge execution instance deployed on an edge node based on the analysis result, and / or pushing the inference request to the global scheduler, comprising: The analysis result includes that the inference request is a delay-sensitive task, the pre-filling task is scheduled to the edge node, and the decoding task is scheduled to the global scheduler.

8. The model reasoning optimization method according to claim 4, characterized in that: Also includes: During the execution of the pre-filling task and / or the decoding task, the memory resources and computing power of the cloud node are continuously monitored to adjust the task allocation based on the monitoring result.

9. A model reasoning optimization device, characterized in that: include: A determination unit is used to respond to a received inference request of a model inference, and a global scheduler determines a first execution instance based on a local perception strategy of a configured global hint tree, including: in response to the inference request, the global scheduler parses a hint identifier of the inference request, and the hint identifier includes at least one of an identifier of a request type of the inference request, an identifier of a request source, a feature identifier of input data, and a special attribute identifier; searching for a tree node storing specified hint information in the global hint tree, and the specified hint information matches the hint identifier; the global hint tree is based on a mapping relationship between the hint information of the auxiliary inference and the context cache, and an execution instance storing the context cache, to configure an initialization tree structure. The initialization tree structure is obtained by initializing the tree framework based on the information of the execution instance, the prompt information includes the task characteristics of the reasoning scenario and / or the reasoning task, and the global prompt tree is used to store at least one of the reasoning task type, model parameters and execution instance resource attributes; based on the tree node index, the instance location of the context cache of the reasoning request is stored; the global scheduler perceives the resource environment and load information of the instance location based on the local perception strategy to determine the first execution instance based on the perception result, the first execution instance reuses the context cache of the reasoning request, and the global scheduler uses the execution instance resource attributes to allocate the reasoning request to the first execution instance on the cloud node; A first scheduling unit, configured to schedule the pre-filling task of the model reasoning to the first execution instance, so as to perform a pre-filling operation based on the first execution instance to obtain a key-value cache including a context cache; A second scheduling unit is used to schedule the decoding task of the model inference to the second execution instance, so as to decode the key-value cache including the context cache based on the second execution instance to obtain the inference result, including: detecting the computing power of the registered execution instance based on the global scheduler; selecting the second execution instance based on the detection result; scheduling the decoding task to the second execution instance; and calling the transmission interface to transfer the key-value cache including the context cache from the first execution instance to the second execution instance, so as to perform a decoding operation on the key-value cache including the context cache based on the second execution instance to obtain the inference result; A feedback unit is used for the global scheduler to feed back the inference result to the request end.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the model reasoning optimization method described in any one of claims 1 to 8 by executing the executable instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model reasoning optimization method described in any one of claims 1 to 8 is implemented.

12. A computer program product having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model reasoning optimization method described in any one of claims 1 to 8 is implemented.

Images