A control method, device, and equipment for large model inference and a medium

Abstract

The application provides a large model inference control method and device, equipment and medium, and relates to the field of artificial intelligence. The current user input text to be inferred is divided into text segments; a unique identifier is generated for the text content of the text segment, obtaining the current identifier; and a current position number is generated according to the position of the text segment in the text to be inferred. Extract the target historical inference information corresponding to the target historical identifier from the historical cache; the target historical inference information includes a target historical position number, a target historical key and a target historical value, and the target historical key is generated according to the target historical position number and the target historical text content. According to the target historical text content and the current position number, a target key is generated; the historical value is used as the target value. The large model performs an inference operation on the text to be inferred based on the target key and the target value, and obtains an inference result. The reusable range of the cached historical inference information is improved, the repeated calculation is reduced, and the inference efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a control method, device, equipment, and medium for large-scale model reasoning. Background Technology

[0002] Large language models have been widely applied in fields such as natural language generation, question answering, and task planning. The reasoning process of a large language model can be divided into a pre-filling stage and a decoding stage. In the pre-filling stage, the large language model generates a corresponding key-value pair for each token in the reasoning request; in the decoding stage, new tokens are generated based on the key-value pairs of each token in the reasoning request. To reduce redundant attention computations during the reasoning process, the generated key-value pairs are typically cached so that they can be reused in the decoding stage or when reasoning for other requests.

[0003] Currently, cached key-value pairs corresponding to historical prefix sequences can only be reused when the current prefix sequence is identical to the historical prefix sequence (i.e., the content and order of each token in the prefix sequence are exactly the same). If two prefix sequences only share some non-contiguous tokens, or if all tokens have identical content but different orders, the cached key-value pairs corresponding to the historical prefix sequence cannot be reused. This results in a large number of cached key-value pairs not being fully utilized, leading to significant repetitive computations in the large model inference process and impacting inference efficiency.

[0004] Therefore, how to make full use of cached key values, reduce redundant calculations, and improve the inference efficiency of large language models is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, this application provides a control method for large model inference, the method comprising:

[0006] Obtain the text to be inferred from the current user input;

[0007] The text to be reasoned is divided into text segments; a unique identifier is generated for the text content of each text segment to obtain the current identifier; and a current position number is generated based on the position of each text segment in the text to be reasoned.

[0008] Extract target historical reasoning information corresponding to the target historical identifier that is the same as the current identifier from the historical cache; the target historical reasoning information includes target historical location number, target historical key, and target historical value; the target historical key is generated based on the target historical location number and the target historical text content.

[0009] Based on the target historical text content and the current position number, generate the target key of the text fragment; use the target historical value as the target value of the text fragment;

[0010] Based on the target key and the target value, a specified large model is invoked to perform reasoning on the text to be reasoned, and the reasoning result is obtained.

[0011] Optionally, generating the target key for the text fragment based on the target historical text content and the current location number includes:

[0012] The position offset is determined based on the current position number and the target historical position number;

[0013] The rotation angle is determined based on the position offset and the preset fundamental frequency;

[0014] Based on the rotation angle, construct a rotation matrix;

[0015] The product of the target history key and the rotation matrix is ​​taken as the target key.

[0016] Optionally, when the historical cache includes multiple target historical inference information, the step of extracting the target historical inference information corresponding to the target historical identifier that is the same as the current identifier from the historical cache includes:

[0017] A method for obtaining the historical reasoning information of each target; the method is forward computation or cache reuse; the cache reuse is used to indicate that the historical reasoning information of the target is extracted from the historical cache;

[0018] Determine whether there exists only one target historical reasoning information determined by the forward calculation method described above;

[0019] If so, the target historical reasoning information shall be taken as the final target historical reasoning information;

[0020] If not, then obtain the generation time point of each target historical reasoning information; and take the target historical reasoning information corresponding to the latest generation time point as the final target historical reasoning information.

[0021] Optionally, the history cache includes a low-level cache and a high-level cache; the low-level cache is used to store the historical identifiers and historical inference information corresponding to each historical text fragment in the inference session triggered by the current user; the high-level cache is used to store the identifiers and inference information corresponding to common text fragments that are consistent with predefined text in the inference sessions triggered by different users.

[0022] The step of retrieving target historical reasoning information from the historical cache corresponding to a target historical identifier that is the same as the current identifier includes:

[0023] Determine whether the target historical reasoning information exists in the low-level cache;

[0024] If it exists, the target historical reasoning information is extracted from the low-level cache;

[0025] If it does not exist, perform the following steps:

[0026] Determine whether the target historical reasoning information exists in the advanced cache;

[0027] If it exists, the target historical reasoning information is extracted from the advanced cache.

[0028] Optionally, extracting the target historical inference information corresponding to the target historical identifier that is the same as the current identifier from the historical cache includes:

[0029] The historical cache is divided into multiple cache pages;

[0030] The priority of a cache page is determined based on the number of times the historical reasoning information stored in the cache page is retrieved within a preset time period; and the cache pages are sorted according to the priority; the higher the number of times the information is retrieved, the higher the priority.

[0031] The top-ranked cache pages are selected as the current cache pages; and the current cache pages are loaded into the GPU to determine whether the target historical inference information exists in the current cache pages.

[0032] If it exists, then extract the target historical reasoning information from the current cache page;

[0033] If it does not exist, then a preset number of cache pages following the current cache page are taken as the current cache page; and the process proceeds to the step of loading the current cache page to the GPU to determine whether the target historical inference information exists in the current cache page, until all the cache pages are loaded to the GPU.

[0034] Optionally, before performing the reasoning operation on the text to be reasoned based on the target key and the target value, the method further includes:

[0035] Determine whether each of the text fragments corresponds to the generation of target inference information, and whether the target inference information passes a preset verification; the target inference information includes the target key and the target value;

[0036] If so, proceed to the step of inferring the text to be inferred by calling the specified large model based on the target key and the target value;

[0037] If not, the fragments that did not generate target inference information and the fragments that failed the preset verification are taken as key text fragments; forward calculation is performed on the key text fragments to obtain the target inference information of the key text fragments; and the step of determining whether each of the text fragments corresponds to the generation of target inference information and whether the target inference information passes the preset verification is returned.

[0038] Optionally, determining that the target inference information passes a preset verification includes:

[0039] Determine whether the target inference information satisfies the first verification condition for the target value and the second verification condition for the target key; the first verification condition includes that the text content is consistent with the target historical text content and that the current identifier is consistent with the target historical identifier; the second verification condition includes that the rotation matrix for generating the target key is successfully constructed and that the dimension of the target key is aligned with the dimension of the rotation matrix.

[0040] If so, then the target reasoning information is determined to have passed the preset verification.

[0041] This application also provides a control device for large-scale model inference, the device comprising:

[0042] The reasoning text acquisition module is used to acquire the text to be reasoned input by the current user;

[0043] The segmentation module is used to divide the text to be reasoned into text segments; generate a unique identifier for the text content of the text segment to obtain the current identifier; and generate a current position number based on the position of the text segment in the text to be reasoned.

[0044] The information extraction module is used to extract target historical inference information corresponding to the target historical identifier that is the same as the current identifier from the historical cache; the target historical inference information includes a target historical location number, a target historical key, and a target historical value; the target historical key is generated based on the target historical location number and the target historical text content;

[0045] The information determination module is used to generate a target key for the text fragment based on the target historical text content and the current location number; and to use the historical value as the target value of the text fragment.

[0046] The reasoning module is used to invoke a specified large model to reason about the text to be reasoned based on the target key and the target value, and obtain the reasoning result.

[0047] This application also provides an apparatus, the apparatus comprising:

[0048] Memory, used to store computer programs;

[0049] A processor, used to implement the control method for any of the above-described large model inferences when executing the computer program.

[0050] This application also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described large model inference control methods.

[0051] In summary, this application provides a control method, apparatus, device, and medium for large-scale model inference. It identifies reusable target historical inference information from the historical cache solely based on the text content of a text fragment, without requiring the current position number to match the target historical position number. The consistency between the target historical identifier and the current identifier indicates that the target historical text content matches the text content of the text fragment. Therefore, the target key of the text fragment can be obtained simply by adjusting the target historical key based on the current position number and the target historical position number, without needing to determine the target key through forward computation. Historical values ​​are generated based on the target historical text content, which matches the text content of the text fragment. Therefore, the historical value can be directly used as the target value of the text fragment, again without needing to determine the target value through forward computation. Thus, this application can reuse historical key values ​​with the same text content but different position numbers, greatly expanding the reusable range of historical inference information, reducing redundant calculations in the inference process, and improving inference efficiency. Attached Figure Description

[0052] Figure 1 A flowchart illustrating the control method for large-scale model inference provided in this application;

[0053] Figure 2 A schematic diagram illustrating the first principle of the control method for large-scale model inference provided in this application;

[0054] Figure 3 A schematic diagram illustrating the principle of searching target historical reasoning information based on multi-level caching provided in this application;

[0055] Figure 4 A schematic diagram of the second principle of the control method for large-scale model inference provided in this application;

[0056] Figure 5 A schematic diagram illustrating the dynamic loading of cached pages provided in this application;

[0057] Figure 6 A schematic diagram of the control device for large-scale model inference provided in this application. Detailed Implementation

[0058] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0059] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0060] Please refer to Figure 1 , Figure 1 A flowchart illustrating the control method for large-scale model inference provided in this application, the method comprising:

[0061] S101. Obtain the text to be inferred input by the current user.

[0062] S102. Divide the text to be reasoned into text segments; generate a unique identifier for the text content of each text segment to obtain the current identifier; generate the current position number based on the position of the text segment in the text to be reasoned.

[0063] This application does not impose any particular limitation on the method of dividing the text to be reasoned into text segments. For example, the text to be reasoned can be divided into text segments according to a preset number of characters; or, after converting the text to be reasoned into a token sequence, a sliding window can be set according to a preset step size to segment the token sequence to obtain text segments; or, the text to be reasoned can be divided into multiple text segments based on its semantics.

[0064] A unique identifier is generated based on the text content of the text fragment. This identifier is then used as the current identifier to identify reusable historical reasoning information. For example, a hash algorithm (such as MiniHash) can be used to generate a hash value based on the text content of the text fragment, and this hash value can be used as the current identifier.

[0065] Since the text to be inferred is divided into text segments, a current position number can be generated based on the position of each text segment within the text to be inferred. For example, if the text segment is the third text segment in the text to be inferred, then the current position number is 3.

[0066] S103. Extract the target history inference information corresponding to the target history identifier that is the same as the current identifier from the historical cache; the target history inference information includes the target history location number, the target history key and the target history value; the target history key is generated based on the target history location number and the target history text content.

[0067] The historical cache stores historical reasoning information and historical identifiers for each historical text fragment within the historical reasoning text. The historical reasoning information includes a historical position number, a historical key, and a historical value. The historical key is generated based on the historical position number and the text content of the historical text fragment, and the historical value is generated based on the text content of the historical text fragment. In this application, the aforementioned historical key can be generated based on the historical position number and the text content of the historical text fragment using RoPE (Rotary Position Embedding), embedding the historical position number into the historical key to represent the relative position information of the historical text fragment.

[0068] Based on this, the historical identifier in the historical cache that is identical to the current identifier is used as the target historical identifier; the historical inference information corresponding to the target historical identifier is used as the target historical inference information; and the target historical inference information is extracted from the historical cache. As mentioned earlier, the identifier of a text fragment is generated based on the text content of the text fragment. The fact that the current identifier and the target historical identifier are the same indicates that the current text content (the text content of the text fragment) is the same as the target historical text content (the text content of the target historical text fragment corresponding to the target historical inference information). Therefore, although the target historical position number embedded in the target historical key may be different from the current position number, the text content embedded in the target historical key is consistent with the current text content; similarly, the text content embedded in the target historical value is consistent with the current text content.

[0069] S104. Generate the target key of the text fragment based on the target historical text content and the current position number; use the target historical value as the target value of the text fragment.

[0070] As mentioned earlier, the target history key is generated based on the target history position number and the target history text content. Since the target history text content is identical to the current text content, to reuse the target history key, simply adjust the target history position number embedded in the target history key to match the current position number. Based on this, the target history key can be adjusted according to the positional deviation between the current position number and the target history position number to obtain a key embedded with the current position number, which is the target key of the text fragment. The above operation is equivalent to generating the target key of the text fragment based on the target history text content and the current position number. The generation process of the target key will be explained later, and will not be elaborated here.

[0071] The target historical value is generated based on the target historical text content. Since the target historical text content is consistent with the current text content, the target historical value is directly used as the target value of the text segment.

[0072] As can be seen, when determining whether reusable target historical reasoning information for text fragments exists in the historical cache, this application only uses the sameness of the current identifier and the target historical identifier as the reuse condition, that is, only the sameness of the current text content and the target historical text content as the reuse condition, without considering whether the current position number and the target historical position number are consistent. It is only necessary to adjust the position number embedded in the target historical key to the current position number to achieve position alignment, thus obtaining the target key of the text fragment without performing forward computation.

[0073] Therefore, when there are text fragments with the same content but different positions between the text to be reasoned and the historical reasoning text, the historical keys and historical values ​​of the text fragments in the historical reasoning text can be reused as the target keys and target values ​​of the text fragments in the current reasoning text. This fully utilizes the reusable historical key values, reduces redundant calculations, and improves reasoning efficiency.

[0074] Furthermore, after obtaining the target key and target value, the target inference information containing the target key and target value is appended to the history cache as new historical inference information, based on the current position number, for reuse in subsequent inferences. For example, if the current position number is the last number, the target inference information is directly added to the end of the history cache; if the current position number is not the last number, the target inference information is inserted into the corresponding position in the history cache.

[0075] S105. Based on the target key and target value, call the specified large model to perform reasoning on the text to be reasoned, and obtain the reasoning result.

[0076] After obtaining the target key and target value of the text fragment, a large model is specified to perform inference on the text to be inferred. The inference process can be divided into a pre-filling stage and a decoding stage. In both the pre-filling and decoding stages, the above inference method can be used to determine the target key and target value of the text fragment, reducing redundant calculations.

[0077] This application does not impose any special restrictions on the specific type of the specified large model or the specific process by which the specified large model performs reasoning operations on the text to be reasoned.

[0078] In summary, this application provides a control method for large-scale model inference. Based on the text content of a text fragment, reusable target historical inference information is identified from the historical cache without requiring the current position number to match the target historical position number. The consistency between the target historical identifier and the current identifier indicates that the target historical text content matches the text content of the text fragment. Therefore, the target key of the text fragment can be obtained simply by adjusting the target historical key based on the current position number, without needing to determine the target key through forward computation. Historical values ​​are generated based on the target historical text content, which matches the text content of the text fragment. Therefore, the historical value can be directly used as the target value of the text fragment, again without needing to determine the target value through forward computation. It is evident that this application can reuse historical key values ​​with the same text content but different position numbers, greatly expanding the reusable range of historical inference information, reducing redundant calculations in the inference process, and improving inference efficiency.

[0079] Based on the above embodiments:

[0080] As an optional embodiment, a target key for a text fragment is generated based on the target historical text content and the current location number, including:

[0081] Determine the position offset based on the current position number and the target's historical position number;

[0082] The rotation angle is determined based on the position offset and the preset fundamental frequency;

[0083] Construct a rotation matrix based on the rotation angle;

[0084] The target key is the product of the target history key and the rotation matrix.

[0085] As mentioned above, this application uses RoPE to embed historical position numbers into historical keys, enabling the historical keys to carry the relative position information of historical text fragments. Based on this, considering that the target historical key is generated according to the target historical position number and the target historical text content, and that the target historical text content is consistent with the current text content, the target key corresponding to the text fragment can be obtained by adjusting the position information carried by the target historical key.

[0086] Specifically, first, the position offset is determined based on the current position number and the target's historical position number. For example, the difference between the current position number and the target's historical position number is used as the position offset. The rotation angle is then determined based on the position offset and the fundamental frequency, and a rotation matrix is ​​constructed based on this angle. For example, the product of the position offset and the fundamental frequency is used as the rotation angle. Next, the target historical key is multiplied by the rotation matrix to obtain the target key, and the position information carried by the target key is the current position number.

[0087] In summary, based on the positional offset between the current position number and the target historical position number, a rotation matrix is ​​determined, and the target historical key is relocated to obtain the target key carrying the current position number. This allows text fragments to reuse historical reasoning information corresponding to historical position numbers different from the current position number, expands the key-value cache reuse range, reduces redundant calculations, and improves reasoning efficiency.

[0088] As an optional embodiment, when the historical cache includes multiple target historical inference information, extracting the target historical inference information corresponding to the target historical identifier that is the same as the current identifier from the historical cache includes:

[0089] The method for obtaining historical reasoning information for each target is determined; the method is either forward computation or cache reuse; cache reuse is used to represent the extraction of target historical reasoning information from the historical cache.

[0090] Determine whether there exists only one target historical reasoning information determined by forward computation;

[0091] If so, then the target historical reasoning information will be used as the final target historical reasoning information;

[0092] If not, obtain the generation time point of each target's historical reasoning information; and use the target's historical reasoning information corresponding to the latest generation time point as the final target's historical reasoning information.

[0093] In this embodiment, when the historical cache includes multiple target historical reasoning information, the method for determining each target historical reasoning information is determined; the method is either forward computation or cache reuse. Target historical reasoning information obtained through forward computation has higher reliability in terms of location information consistency and context adaptability. Therefore, target historical reasoning information determined through forward computation has a higher priority than target historical reasoning information determined through cache reuse. Based on this, if only one target historical reasoning information exists among all target historical reasoning information, that target reasoning information is directly used as the final target historical reasoning information.

[0094] If there are multiple target historical reasoning information determined by forward computation, or if there are no target historical reasoning information determined by forward computation (i.e., all target historical reasoning information is determined by cache reuse), then obtain the generation time point of each target historical reasoning information; take the target historical reasoning information corresponding to the latest generation time point as the final target historical reasoning information.

[0095] For example, if a user sequentially inputs three inference requests A, B, and C in a reasoning session, and has already completed the inference operations for inference requests A and B, the historical inference information for each historical text fragment in inference requests A and B is stored in the historical cache. Specifically, inference request A states that Xiaoming loves apples, inference request B states that Xiaohong loves apples and bananas, and inference request C states that Xiaoli not only loves apples and bananas, but also likes…

[0096] When reasoning about inference request C, for the text fragment "apple" in inference request C, the first historical inference information corresponding to "apple" in inference request A and the second historical inference information corresponding to "apple" in inference request B are retrieved from the historical cache using the current identifier corresponding to "apple". The generation time of the second inference information is closer to the current time, so the key value in the second historical inference information is reused as the key value of the text fragment "apple" in inference request C.

[0097] Furthermore, in actual use, the system resources used for real-time forward computation and cache lookup may be different. If for the same text segment, there is a key value obtained from forward computation and a key value retrieved from the historical cache, then the key value obtained from forward computation shall be used first.

[0098] In summary, when the historical cache includes multiple target historical inference information, the target historical inference information determined by forward computation is used first as the final target historical inference information, and the target historical inference information with the latest generation time is used as the final target historical inference information to ensure the reliability of the inference results.

[0099] As an optional embodiment, the history cache includes a low-level cache and a high-level cache; the low-level cache is used to store the historical identifiers and historical inference information corresponding to each historical text fragment in the inference session triggered by the current user; the high-level cache is used to store the identifiers and inference information corresponding to common text fragments that are consistent with predefined text in the inference sessions triggered by different users.

[0100] Extract target historical reasoning information from the historical cache that corresponds to the target historical identifier with the same identifier as the current identifier, including:

[0101] Determine whether the target historical reasoning information exists in the low-level cache;

[0102] If it exists, retrieve the target historical reasoning information from the low-level cache;

[0103] If it does not exist, perform the following steps:

[0104] Determine if the target historical reasoning information exists in the advanced cache;

[0105] If it exists, retrieve the target historical reasoning information from the advanced cache.

[0106] In practical applications, key-value caches for each inference session are typically maintained independently. When the same information exists between different inference sessions, a full forward computation of the request to be inferred still needs to be performed separately, resulting in relatively high inference latency.

[0107] Please refer to Figure 2 and Figure 3 , Figure 2 This is a schematic diagram illustrating the first principle of the control method for large-scale model inference provided in this application. Figure 3 This is a schematic diagram illustrating the principle of using a multi-level cache to retrieve target historical reasoning information, as provided in this application. Figure 2 and Figure 3 As shown, a multi-level caching system is constructed in this embodiment, and the historical cache includes low-level cache and high-level cache.

[0108] The low-level cache stores the historical identifiers and historical reasoning information corresponding to each historical text fragment in the current user-triggered inference session. When the current user first triggers an inference session, a separate low-level cache is allocated for that session. The low-level cache stores historical reasoning information for all text fragments from all historical inference requests entered by the current user, as well as historical reasoning information for text fragments obtained by reasoning from historical reasoning information. It can be understood that the current user can enter multiple inference requests within a single inference session, and when reasoning for each request, the target historical reasoning information can be retrieved from the low-level cache, enabling information sharing between multiple inference attempts.

[0109] Furthermore, the low-level cache can employ an LRU (Least Recently Used) management strategy to manage the historical inference information stored within it. The low-level cache stores the last time historical inference information was reused. When the low-level cache is full, the least recently reused historical inference information is removed first to maintain efficient utilization of cache space.

[0110] The system also maintains a globally shared high-level cache in the background. This high-level cache stores identifiers and inference information corresponding to common text fragments consistent with predefined text across different user-triggered inference sessions, enabling key-value cache reuse across sessions. For example, system prompts, task-specific prompt templates (including but not limited to RAG templates (Retrieval-Augmented Generation) and function call templates), unified opening prompts from the enterprise knowledge base, and frequently occurring function call description texts are used as common text fragments. Historical inference information stored in the high-level cache is shared across different inference sessions in a read-only manner.

[0111] Based on the above, such as Figure 3 As shown, when retrieving target historical inference information from the historical cache, the key-value cache scheduler is first used to check if the target historical inference information exists in the low-level cache. If the target historical inference information is not found in the low-level cache, it is then searched in the high-level cache. As an optional embodiment, a historical inference information index table can be pre-established. This index table uses the identifier of the historical text fragment as the key to point to the storage location of the historical inference information of the historical text fragment in the low-level cache, so as to check if the current identifier exists in the historical inference information index table, and determine the storage location of the target historical inference information in the low-level cache if the current identifier exists.

[0112] In addition, please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating the second principle of the control method for large model inference provided in this application. If the target historical inference information is not found in the high-level cache, forward computation can be performed on the text fragment to determine its target key and target value. Then, the target key, target value, and current position number are stored as new historical inference information in the low-level cache. If the text fragment is a general text fragment, the new historical inference information composed of the target key, target value, and current position number is further asynchronously stored in the high-level cache for reuse in subsequent inference sessions.

[0113] In summary, this embodiment constructs a multi-level caching system. The low-level cache enables information sharing among multiple inference requests within the same inference session. The high-level cache, while ensuring isolation between different users' inference sessions, allows for cross-session-level key-value cache reuse of common text fragments, significantly improving key-value cache hit rate and further reducing redundant computation.

[0114] Furthermore, since historical inference information from different inference sessions can be reused through the advanced cache, the target historical inference information can also be reused from the advanced cache when the current user triggers the inference session for the first time, without having to perform a full pre-filling calculation, thus reducing the inference latency of the first text segment.

[0115] As an optional embodiment, extracting target historical reasoning information corresponding to the target historical identifier that is the same as the current identifier from the historical cache includes:

[0116] Divide the historical cache into multiple cache pages;

[0117] The priority of a cache page is determined based on the number of times the historical inference information stored in the cache page is retrieved within a preset time period; and the cache pages are sorted according to their priority; the higher the number of times, the higher the priority.

[0118] The top-ranked cache pages are selected as the current cache pages, and the current cache pages are loaded onto the GPU to determine whether the target historical inference information exists in the current cache pages.

[0119] If it exists, extract the target historical reasoning information from the current cache page;

[0120] If it does not exist, a preset number of cache pages following the current cache page will be used as the current cache page; and the process will proceed to loading the current cache page onto the GPU to determine whether the target historical inference information exists in the current cache page, until all cache pages are loaded onto the GPU.

[0121] In practical applications, the GPU memory occupied by the key-value cache is linearly related to the length of the text to be inferred and the number of model layers. In long-context tasks or high-concurrency scenarios, the GPU memory occupied by the key-value cache can reach or even exceed the GPU memory occupied by the model parameters, resulting in a significant compression of available GPU memory space, limiting parallel inference capabilities, and potentially leading to frequent GPU memory scheduling and performance degradation.

[0122] Therefore, this embodiment divides the historical cache into multiple cache pages. Each cache page can be considered as the basic unit of on-demand loading during inference. This embodiment does not specifically limit the method of dividing the historical cache. For example, the historical cache can be evenly divided into multiple cache pages of a fixed size; or the cache pages can be dynamically configured based on hardware resources, inference task requirements, and system context length. For each cache page, its priority is determined by the number of times the historical inference information stored in the cache page is retrieved within a preset time period. The more times it is retrieved, the higher the activity level of the cache page, and therefore the higher its priority.

[0123] During inference, a predetermined number of cache pages at the top of the list are first loaded into the GPU as the current cache page. This allows for the retrieval of target historical inference information from the GPU during inference. If the target historical inference information exists in the current cache page, it is extracted from the current cache page. If the target historical inference information does not exist in the current cache page, a predetermined number of cache pages following the current cache page are used as the current cache page, and the steps of loading the current cache page into the GPU are repeated until all cache pages are loaded into the GPU.

[0124] Please refer to Figure 5 , Figure 5 This diagram illustrates a dynamic loading of cached pages provided in this application. As can be seen, during inference, only the cached pages currently needed are loaded into GPU memory via the key-value cache scheduler. Cache pages not currently needed remain in CPU memory or disk, thus avoiding the pre-allocation of all GPU memory, reducing memory fragmentation, and improving memory utilization. Even in contextual tasks or high-concurrency scenarios, good parallel inference capabilities are maintained.

[0125] Furthermore, when the historical cache is divided into low-level and high-level caches, each cache is further divided into multiple cache pages. Each cache page in the low-level cache is dynamically loaded into the GPU based on its activity level—that is, the number of times the historical inference information stored in the cache page has been retrieved within a preset time period—according to the aforementioned rules. Similarly, each cache page in the high-level cache is dynamically loaded into the GPU according to the same rules, prioritizing cache pages containing historical inference information that have been reused in multiple inference sessions.

[0126] In summary, this embodiment utilizes the concept of virtual memory paging management to divide a large-scale historical cache into multiple cache pages and schedule them on demand according to the activity of the cache pages, thereby improving the flexibility of historical cache management, reducing video memory usage, and ensuring inference capabilities.

[0127] As an optional embodiment, before performing reasoning operations on the text to be reasoned based on the target key and target value, the following steps are also included:

[0128] Determine whether each text fragment corresponds to the generated target inference information, and whether the target inference information passes the preset verification; the target inference information includes the target key and the target value;

[0129] If so, proceed to the step of inferring the text to be inferred based on the target key and target value by calling the specified large model;

[0130] If not, the fragments that did not generate target inference information and the fragments that failed the preset verification are taken as key text fragments; forward calculation is performed on the key text fragments to obtain the target inference information of the key text fragments; and the step of determining whether each text fragment corresponds to the generation of target inference information and whether the target inference information passes the preset verification is returned.

[0131] In this embodiment, before performing reasoning operations on the text to be reasoned based on the target key and target value, a consistency check is performed on all text fragments in the text to be reasoned to determine whether there are any text fragments that need to be independently computed to determine the target reasoning information.

[0132] First, it is determined whether each text fragment corresponds to the generated target inference information, and whether the target inference information passes the preset verification. The specific implementation method for the preset verification of the target inference information will be described in subsequent embodiments, and will not be elaborated here.

[0133] If all text fragments in the request to be reasoned correspond to target reasoning information, and all target reasoning information passes the preset verification, then the reasoning operation is performed on the text to be reasoned based on the target key and target value of each text fragment. This embodiment does not specifically limit the specific process of performing the reasoning operation on the text to be reasoned.

[0134] like Figure 5 As shown, if there are text fragments in the request to be inferred that have not generated target inference information, or text fragments whose target inference information has failed the preset verification, then forward computation is required for these text fragments. Specifically, the fragments that have not generated target inference information and the fragments that have failed the preset verification are taken as key text fragments; forward computation is performed on the key text fragments to obtain the target inference information of the key text fragments. The specific process of performing forward computation on the key text fragments to obtain the target inference information of the key text fragments is not described in this embodiment. After obtaining the target inference information of the key text fragments, the process returns to the step of determining whether each text fragment corresponds to the generation of target inference information and whether the target inference information passes the preset verification, to ensure that all text fragments in the request to be inferred correspond to the generation of target inference information that passes the preset verification, thus ensuring the reliability of the inference result.

[0135] In summary, in this embodiment, key text fragments are identified by preset verification, and forward calculations are performed on the key text fragments to fill in the missing target reasoning information of the key text fragments, thereby ensuring the correctness of the reasoning.

[0136] As an optional embodiment, determining that the target inference information passes a preset verification includes:

[0137] Determine whether the target inference information satisfies the first verification condition for the target value and the second verification condition for the target key; the first verification condition includes that the text content is consistent with the target historical text content and that the current identifier is consistent with the target historical identifier; the second verification condition includes that the rotation matrix used to generate the target key is successfully constructed and that the dimension of the target key is aligned with the dimension of the rotation matrix.

[0138] If so, then the target reasoning information is confirmed to have passed the preset verification.

[0139] In this application, a unique current identifier is generated for each text fragment based on its text content. Therefore, when the current identifier matches the target historical identifier, it indicates that the text content matches the target historical text content. However, in some extreme cases, it is possible for text fragments to have different text content but the same identifier. For example, when using a hash algorithm to generate an identifier based on the text content of a text fragment, if a hash collision occurs, the same identifier may be generated for text fragments with different text content. In this case, the text fragment will reuse incorrect target historical reasoning information, thereby affecting the reasoning result.

[0140] To avoid the aforementioned problems, this embodiment sets a first verification condition for the target value. The first verification condition includes the consistency between the current text content and the target historical text content, and the consistency between the current identifier and the target historical identifier. If the current text content is inconsistent with the target historical text content, or the current identifier is inconsistent with the target historical identifier, it indicates that the text content carried in the target value is inconsistent with the current text content, and therefore, inference operations cannot be performed based on this target value. If the current text content is consistent with the target historical text content, and the current identifier is consistent with the target historical identifier, it indicates that the text content carried in the target value is consistent with the current text content, and in this case, inference operations can be performed based on the target value.

[0141] Since the target value and the target key belong to the same target historical reasoning information, the target value satisfies the first verification condition mentioned above, indicating that the text content carried by the target key is consistent with the current text content. Subsequently, the position information carried by the target key can be verified using the second verification condition.

[0142] As mentioned earlier, the target history key needs to be adjusted based on the target historical text content and the current position number to generate the target key for the text fragment. To ensure the reliability of the target key, a second verification condition is set for it; the second verification condition includes dimension alignment and successful construction of the rotation matrix used to generate the target key. Dimension alignment includes aligning the dimensions of the target key with the dimensions of the rotation matrix, and may also include aligning the dimensions of the target key with the dimensions of the target value, etc., the specific alignment condition can be set according to actual needs. For the rotation matrix, the orthogonality of the rotation matrix can be verified to determine whether the rotation matrix has been successfully constructed; this embodiment does not impose any particular limitation on this.

[0143] In summary, this embodiment ensures that the text fragment reuses the correct target historical reasoning information by adding additional verification conditions, thereby ensuring the reliability of the reasoning results.

[0144] Please refer to Figure 6 , Figure 6 This is a schematic diagram of the control device for large-scale model inference provided in this application. The device includes:

[0145] The reasoning text acquisition module 601 is used to acquire the reasoning text input by the current user;

[0146] The segmentation module 602 is used to divide the text to be reasoned into text segments; generate a unique identifier for the text content of the text segment to obtain the current identifier; and generate a current position number based on the position of the text segment in the text to be reasoned.

[0147] The information extraction module 603 is used to extract target historical inference information corresponding to the target historical identifier that is the same as the current identifier from the historical cache; the target historical inference information includes the target historical location number, the target historical key, and the target historical value; the target historical key is generated based on the target historical location number and the target historical text content;

[0148] The information determination module 604 is used to generate a target key for a text fragment based on the target's historical text content and the current location number; and to use the historical value as the target value for the text fragment.

[0149] The reasoning module 605 is used to call a specified large model to reason about the text to be reasoned based on the target key and target value, and obtain the reasoning result.

[0150] For a detailed description of the control device for large model inference, please refer to the embodiments of the control method for large model inference described above; this application will not repeat the details here.

[0151] Based on the above embodiments:

[0152] As an optional embodiment, the information determination module 604 includes:

[0153] The offset determination module is used to determine the position offset based on the current position number and the target's historical position number;

[0154] The rotation angle determination module is used to determine the rotation angle based on the position offset and the preset base frequency;

[0155] The rotation matrix determination module is used to construct a rotation matrix based on the rotation angle.

[0156] The target key determination module is used to take the product of the target historical key and the rotation matrix as the target key;

[0157] The target value determination module is used to use historical values ​​as target values ​​for text segments.

[0158] As an optional embodiment, when the historical cache includes multiple target historical reasoning information, the information extraction module 603 includes:

[0159] The method determination module is used to determine the method for obtaining the historical reasoning information of each target; the determination method is either forward computation or cache reuse; cache reuse is used to represent the extraction of the target's historical reasoning information from the historical cache.

[0160] The target information discrimination module is used to determine whether there is only one target historical reasoning information determined by forward calculation; if so, the first determination module is triggered; if not, the second determination module is triggered.

[0161] The first determining module is used to use the target historical reasoning information as the final target historical reasoning information.

[0162] The second determining module is used to obtain the generation time point of the historical reasoning information of each target; and to take the target historical reasoning information corresponding to the latest generation time point as the final target historical reasoning information.

[0163] As an optional embodiment, the history cache includes a low-level cache and a high-level cache; the low-level cache is used to store the historical identifiers and historical inference information corresponding to each historical text fragment in the inference session triggered by the current user; the high-level cache is used to store the identifiers and inference information corresponding to common text fragments that are consistent with predefined text in the inference sessions triggered by different users.

[0164] Correspondingly, the information extraction module 603 includes:

[0165] The low-level cache discrimination module is used to determine whether the target historical reasoning information exists in the low-level cache; if it exists, the low-level extraction module is triggered; if it does not exist, the high-level extraction module is triggered.

[0166] The low-level extraction module is used to extract target historical reasoning information from the low-level cache;

[0167] The advanced extraction module is used to determine whether the target historical reasoning information exists in the advanced cache; if it exists, the target historical reasoning information is extracted from the advanced cache.

[0168] As an optional embodiment, the information extraction module 603 includes:

[0169] The partitioning module is used to divide the historical cache into multiple cache pages;

[0170] The sorting module is used to determine the priority of a cache page based on the number of times the historical inference information stored in the cache page is retrieved within a preset time period; and sort the cache pages according to the priority; the greater the number of times, the higher the priority.

[0171] The current cache page loading module is used to select a preset number of cache pages that are ranked first as the current cache page; and load the current cache page into the GPU to determine whether the target historical inference information exists in the current cache page; if it exists, the extraction submodule is triggered; if it does not exist, the update loading module is triggered.

[0172] The extraction submodule is used to extract target historical reasoning information from the current cache page;

[0173] The update loading module is used to select a preset number of cache pages following the current cache page as the current cache page; and to trigger the current cache page loading module until all cache pages are loaded onto the GPU.

[0174] As an optional embodiment, the device further includes:

[0175] The verification module is used to determine whether each text fragment corresponds to the generation of target reasoning information and whether the target reasoning information passes the preset verification before performing reasoning operations on the text to be reasoned based on the target key and target value. The target reasoning information includes the target key and target value. If yes, the reasoning module 605 is triggered; if no, the forward calculation module is triggered.

[0176] The forward calculation module is used to treat fragments that have not generated target inference information and fragments that have not passed the preset verification as key text fragments; perform forward calculation on the key text fragments to obtain the target inference information of the key text fragments; and trigger the verification module.

[0177] As an optional embodiment, the verification module includes:

[0178] The verification submodule is used to determine whether the target inference information meets the first verification condition for the target value and the second verification condition for the target key. The first verification condition includes that the text content is consistent with the target's historical text content and that the current identifier is consistent with the target's historical identifier. The second verification condition includes that the rotation matrix used to generate the target key is successfully constructed and that the dimensions of the target key are aligned with the dimensions of the rotation matrix. If so, the verification pass module is triggered.

[0179] The verification module is used to determine whether the target reasoning information has passed the preset verification.

[0180] This application also provides an apparatus, the apparatus comprising:

[0181] Memory, used to store computer programs;

[0182] A processor is used to implement the control methods for any of the above-mentioned large model inferences when executing computer programs.

[0183] For a detailed description of the device, please refer to the embodiments of the control method for large model inference described above; this application will not repeat them here.

[0184] This application also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described large model inference control methods.

[0185] The aforementioned storage media include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or integrated into dedicated logic circuitry.

[0186] For a detailed description of the storage medium, please refer to the embodiments of the control method for large model inference described above; this application will not repeat them here.

[0187] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0188] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Claims

1. A control method for large-scale model inference, characterized in that, The method includes: Obtain the text to be inferred from the current user input; The text to be reasoned is divided into text segments; a unique identifier is generated for the text content of each text segment to obtain the current identifier; and a current position number is generated based on the position of each text segment in the text to be reasoned. Extract target historical reasoning information corresponding to the target historical identifier that is the same as the current identifier from the historical cache; the target historical reasoning information includes target historical location number, target historical key, and target historical value; the target historical key is generated based on the target historical location number and the target historical text content. Based on the target historical text content and the current position number, generate the target key of the text fragment; use the target historical value as the target value of the text fragment; Based on the target key and the target value, a specified large model is invoked to perform reasoning on the text to be reasoned, and the reasoning result is obtained. The step of generating the target key for the text fragment based on the target historical text content and the current location number includes: The position offset is determined based on the current position number and the target historical position number; The rotation angle is determined based on the position offset and the preset fundamental frequency; Based on the rotation angle, construct a rotation matrix; The product of the target history key and the rotation matrix is ​​taken as the target key.

2. The control method for large model inference as described in claim 1, characterized in that, When the historical cache includes multiple target historical reasoning information, the step of extracting the target historical reasoning information corresponding to the target historical identifier that is the same as the current identifier from the historical cache includes: A method for obtaining the historical reasoning information of each target; the method is forward computation or cache reuse; the cache reuse is used to indicate that the historical reasoning information of the target is extracted from the historical cache; Determine whether there exists only one target historical reasoning information determined by the forward calculation method described above; If so, the target historical reasoning information shall be taken as the final target historical reasoning information; If not, then obtain the generation time point of each target historical reasoning information; and take the target historical reasoning information corresponding to the latest generation time point as the final target historical reasoning information.

3. The control method for large model inference as described in claim 1, characterized in that, The history cache includes a low-level cache and a high-level cache; the low-level cache is used to store the historical identifiers and historical inference information corresponding to each historical text fragment in the inference session triggered by the current user; the high-level cache is used to store the identifiers and inference information corresponding to common text fragments that are consistent with predefined text in the inference sessions triggered by different users. The step of retrieving target historical reasoning information from the historical cache corresponding to a target historical identifier that is the same as the current identifier includes: Determine whether the target historical reasoning information exists in the low-level cache; If it exists, the target historical reasoning information is extracted from the low-level cache; If it does not exist, perform the following steps: Determine whether the target historical reasoning information exists in the advanced cache; If it exists, the target historical reasoning information is extracted from the advanced cache.

4. The control method for large model inference as described in claim 1, characterized in that, The step of retrieving target historical reasoning information from the historical cache corresponding to a target historical identifier that is the same as the current identifier includes: The historical cache is divided into multiple cache pages; The priority of a cache page is determined based on the number of times the historical reasoning information stored in the cache page is retrieved within a preset time period; and the cache pages are sorted according to the priority; the higher the number of times the information is retrieved, the higher the priority. The top-ranked cache pages are selected as the current cache pages; and the current cache pages are loaded into the GPU to determine whether the target historical inference information exists in the current cache pages. If it exists, then extract the target historical reasoning information from the current cache page; If it does not exist, then a preset number of cache pages following the current cache page are taken as the current cache page; and the process proceeds to the step of loading the current cache page to the GPU to determine whether the target historical inference information exists in the current cache page, until all the cache pages are loaded to the GPU.

5. The control method for large model inference as described in claim 1, characterized in that, Before performing the reasoning operation on the text to be reasoned based on the target key and the target value, the method further includes: Determine whether each of the text fragments corresponds to the generation of target inference information, and whether the target inference information passes a preset verification; the target inference information includes the target key and the target value; If so, proceed to the step of inferring the text to be inferred by calling the specified large model based on the target key and the target value; If not, the fragments that did not generate target inference information and the fragments that failed the preset verification are taken as key text fragments; forward calculation is performed on the key text fragments to obtain the target inference information of the key text fragments; and the step of determining whether each of the text fragments corresponds to the generation of target inference information and whether the target inference information passes the preset verification is returned.

6. The control method for large model inference as described in claim 5, characterized in that, Determining that the target reasoning information passes a preset check includes: Determine whether the target inference information satisfies the first verification condition for the target value and the second verification condition for the target key; the first verification condition includes that the text content is consistent with the target historical text content and that the current identifier is consistent with the target historical identifier; the second verification condition includes that the rotation matrix for generating the target key is successfully constructed and that the dimension of the target key is aligned with the dimension of the rotation matrix. If so, then the target reasoning information is determined to have passed the preset verification.

7. A control device for large-scale model reasoning, characterized in that, The device includes: The reasoning text acquisition module is used to acquire the text to be reasoned input by the current user; The segmentation module is used to divide the text to be reasoned into text segments; generate a unique identifier for the text content of the text segment to obtain the current identifier; and generate a current position number based on the position of the text segment in the text to be reasoned. The information extraction module is used to extract target historical inference information corresponding to the target historical identifier that is the same as the current identifier from the historical cache; the target historical inference information includes a target historical location number, a target historical key, and a target historical value; the target historical key is generated based on the target historical location number and the target historical text content; The information determination module is used to generate a target key for the text fragment based on the target historical text content and the current location number; and to use the target historical value as the target value of the text fragment. The reasoning module is used to invoke a specified large model to reason about the text to be reasoned based on the target key and the target value, and obtain the reasoning result. The information determination module includes: The offset determination module is used to determine the position offset based on the current position number and the target's historical position number; The rotation angle determination module is used to determine the rotation angle based on the position offset and the preset base frequency; The rotation matrix determination module is used to construct a rotation matrix based on the rotation angle. The target key determination module is used to take the product of the target historical key and the rotation matrix as the target key; The target value determination module is used to use historical target values ​​as target values ​​for text segments.

8. A device, characterized in that, include: Memory, used to store computer programs; A processor for implementing the control method for large model inference as described in any one of claims 1 to 6 when executing the computer program.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the control method for large model inference as described in any one of claims 1 to 6.

Images