Model reasoning method and device and electronic equipment

Abstract

The invention provides a model reasoning method and device and electronic equipment. The method comprises the following steps: dividing a to-be-processed text into a plurality of text blocks; performing vectorization processing on each text block in the plurality of text blocks to obtain a plurality of semantic vectors; for each semantic vector in the plurality of semantic vectors, determining a plurality of target KV cache blocks of the to-be-processed text according to a similarity value between the semantic vector and a Key value in a pre-filled KV cache block; and reasoning the plurality of target KV cache blocks of the to-be-processed text through the pre-training language model to obtain a reasoning result. In the implementation process of the scheme, the target KV cache block similar to the to-be-processed text block in meaning is found according to the similarity value between the semantic vector and the Key value in the pre-filled KV cache block, and the requirement for strict and accurate matching of the semantic vector and the Key value is effectively relaxed, so that reuse of the KV cache block is realized with a higher probability, and the reusability of the KV cache block is improved. And the hit rate of the cache is effectively improved.

Description

Technical Field

[0001] The present application relates to the technical fields of deep learning, neural networks and large language models, and more specifically, to a model reasoning method, device and electronic device. Background Art

[0002] Currently, the inputs received by large language models are usually diverse, which means that each new request or conversation may contain text sequences of different lengths. In actual practice, it is found that the context lengths of different requests may be different, and longer contexts may overwrite the cache of shorter contexts, resulting in a decrease in the cache hit rate. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a model reasoning method, device and electronic device for improving the problem of decreased cache hit rate.

[0004] An embodiment of the present application provides a model reasoning method, including: dividing a text to be processed into multiple text blocks; performing vectorization processing on each of the multiple text blocks to obtain multiple semantic vectors; for each semantic vector in the multiple semantic vectors, determining multiple target KV cache blocks of the text to be processed according to a similarity value between the semantic vector and a Key value in a pre-filled KV cache block; and performing reasoning on the multiple target KV cache blocks of the text to be processed by using a pre-trained language model to obtain a reasoning result. In the implementation process of the above scheme, the text to be processed is divided into multiple text blocks, thereby reducing the size of each block. Since smaller text blocks are more likely to have appeared in previous requests, they are more likely to find a matching cache, thereby increasing the possibility of cache hit rate. Furthermore, each text block is vectorized to obtain a semantic vector that represents the intention of the text block. Even if the contents of the text blocks are slightly different, but the information conveyed is similar, the target KV cache block with a similar meaning to the text block to be processed can be found based on the similarity value between the semantic vector and the Key value in the pre-filled KV cache block, effectively relaxing the requirement for strict and precise matching of the semantic vector and the Key value, thereby having a greater probability of realizing the reuse of the KV cache block, effectively improving the cache hit rate.

[0005] Optionally, in an embodiment of the present application, multiple target KV cache blocks of the text to be processed are determined according to the similarity value between the semantic vector and the Key value in the pre-filled KV cache block, including: judging whether the similarity value between the semantic vector and the Key value in the KV cache block is greater than a similarity threshold; if so, adding the KV cache block to the multiple target KV cache blocks of the text to be processed, otherwise, generating a KV buffer block of the semantic vector, and adding the KV buffer block of the semantic vector to the multiple target KV cache blocks of the text to be processed. In the implementation process of the above scheme, by comparing the similarity between the newly generated semantic vector and the Key value in the existing KV cache block, it is decided whether to reuse the existing KV cache block or generate a new KV cache block, so that the target KV cache block with a similar meaning to the text block to be processed can be found through the similarity value between the semantic vector and the Key value in the pre-filled KV cache block, which effectively relaxes the requirement for strict and accurate matching of the semantic vector and the Key value, thereby reducing repeated calculations, and the model can generate output faster, improving the response speed of the entire system.

[0006] Optionally, in an embodiment of the present application, multiple target KV cache blocks of the text to be processed are determined based on the similarity value between the semantic vector and the Key value in the pre-filled KV cache block, including: fusing the semantic vector and the relative position of the semantic vector in multiple semantic vectors to obtain a feature vector; judging whether the similarity value between the feature vector and the Key value in the KV cache block is greater than a similarity threshold; if so, adding the KV cache block to the multiple target KV cache blocks of the text to be processed, otherwise, generating a KV buffer block of the semantic vector, and adding the KV buffer block of the semantic vector to the multiple target KV cache blocks of the text to be processed. In the implementation process of the above scheme, by matching the semantic vector and the feature vector fused with its relative position, the feature vector and the Key value in the cache block can be matched more finely, the cache that can be reused can be identified more accurately, and the accuracy of cache reuse is improved, thereby reducing unnecessary calculations and improving the response speed of the model.

[0007] Optionally, in an embodiment of the present application, before determining multiple target KV cache blocks of the text to be processed based on the similarity value between the semantic vector and the Key value in the pre-filled KV cache block, it also includes: obtaining a sample text and dividing the sample text into multiple sample blocks; vectorizing each sample block in the multiple sample blocks to obtain multiple sample vectors; for each sample vector in the multiple sample vectors, generating a KV cache block based on the sample vector and the relative position of the sample vector in the multiple sample vectors. In the implementation process of the above scheme, by dividing the input text into blocks and vectorizing them separately, the system can process long text input more efficiently, especially in the case of processing a large amount of repeated content, such as multi-round conversations or data analysis tasks. This method can significantly reduce computing requirements because the system can reuse the KV cache that has been calculated before, instead of recalculating the entire input each time, thereby improving the retrieval hit rate of the local KV cache.

[0008] Optionally, in an embodiment of the present application, after generating a KV cache block according to the sample vector and the relative position of the sample vector in a plurality of sample vectors, it also includes: determining whether the usage frequency of the sample vector is greater than a preset frequency threshold; if so, pre-filling the KV cache block into the video memory of the graphics processor GPU, otherwise, pre-filling the KV cache block into the memory of the central processing unit CPU. In the implementation process of the above scheme, the KV cache block storage location technology is determined by the usage frequency of the sample vector, so that the KV cache block used frequently is placed in the GPU video memory, which can significantly reduce the access delay, because the GPU video memory usually provides faster access speed than the CPU memory. Further, by distinguishing KV cache blocks with different usage frequencies and selecting a suitable storage location according to their access frequency, the cache used frequently is placed near the computing unit, and the cache used infrequently can be placed in the relatively cheap CPU memory, saving costs without sacrificing performance, thereby optimizing the utilization efficiency of hardware resources. Further, storing the KV cache blocks used frequently in the GPU reduces the need for frequent transmission of these cache blocks between the CPU and the GPU, which helps to reduce the pressure on network bandwidth. This is especially important for large-scale distributed systems, as it can reduce the overhead caused by cross-node communication.

[0009] Optionally, in an embodiment of the present application, the pre-trained language model includes: multiple neural network nodes; before the pre-trained language model is used to infer multiple target KV cache blocks of the text to be processed, it also includes: for each of the multiple neural network nodes, determining whether the activation frequency of the neural network node is greater than a preset frequency threshold; if so, pre-filling the neural network node into the GPU's video memory, otherwise, pre-filling the neural network node into the CPU's memory. In the implementation of the above scheme, by utilizing the characteristics of the hot and cold power law distribution of the neural network nodes, the frequently activated and called hot nodes are pre-filled into the GPU's video memory, and the infrequently called cold nodes are pre-filled into the CPU's memory, so that the CPU in the inference decoding process implements cold node inference, and the GPU in the inference decoding process implements hot node inference, effectively completing the heterogeneous computing function. This heterogeneous computing function can greatly reduce the computing power cost and storage cost of large model inference services, thereby reducing the overall cost of large model inference services.

[0010] Optionally, in an embodiment of the present application, a plurality of target KV cache blocks of the text to be processed are inferred by a pre-trained language model, including: obtaining a neural network node and a target KV cache block from the video memory of the GPU or the memory of the CPU; determining whether the activation frequency of the neural network node is greater than a preset frequency threshold; if so, the target KV cache block is inferred by the GPU, otherwise, the target KV cache block is inferred by the CPU. In the implementation process of the above scheme, the activation frequency of the neural network node is determined to determine whether to perform inference on the GPU or the CPU. High-frequency activated nodes mean that these nodes are frequently called during the inference process, so placing these nodes and their related KV cache blocks on the GPU for inference can significantly improve performance. Furthermore, by assigning nodes activated at different frequencies to the most suitable hardware resources (GPU or CPU), computing resources can be used more efficiently. The execution of low-frequency activated nodes on the CPU can reduce the burden on the GPU and avoid the waste of GPU resources, thereby effectively optimizing the utilization efficiency of resources.

[0011] Optionally, in an embodiment of the present application, multiple target KV cache blocks of the text to be processed are inferred through a pre-trained language model, including: reassembling multiple target KV cache blocks of the text to be processed to obtain an assembled context KV cache block; reasoning the assembled context KV cache block through a pre-trained language model. In the implementation process of the above scheme, by reassembling multiple target KV cache blocks, the system can more efficiently utilize the information in the cache, which means that even if some parts of the input text are new or changed, as long as most of the content is repeated, the system can reuse the existing cache and reduce repeated calculations, thereby effectively improving the cache hit rate and utilization rate.

[0012] The embodiment of the present application also provides a model reasoning device, including: a text block division module, used to divide the text to be processed into multiple text blocks; a text vectorization module, used to perform vectorization processing on each of the multiple text blocks to obtain multiple semantic vectors; a cache block determination module, used to determine, for each semantic vector in the multiple semantic vectors, multiple target KV cache blocks of the text to be processed according to the similarity value between the semantic vector and the Key value in the pre-filled KV cache block; an inference result acquisition module, used to perform inference on the multiple target KV cache blocks of the text to be processed through a pre-trained language model to obtain an inference result.

[0013] Optionally, in an embodiment of the present application, a cache block determination module includes: a similarity value judgment submodule, used to determine whether the similarity value between the semantic vector and the Key value in the KV cache block is greater than a similarity threshold; a target cache block determination submodule, used to add the KV cache block to multiple target KV cache blocks of the text to be processed if the similarity value between the semantic vector and the Key value in the KV cache block is greater than the similarity threshold; if the similarity value between the semantic vector and the Key value in the KV cache block is less than or equal to the similarity threshold, generate a KV buffer block of the semantic vector, and add the KV buffer block of the semantic vector to the multiple target KV cache blocks of the text to be processed.

[0014] Optionally, in an embodiment of the present application, a cache block determination module includes: a feature vector acquisition submodule, used to fuse the semantic vector and the relative position of the semantic vector in multiple semantic vectors to obtain a feature vector; a similarity value judgment submodule, used to judge whether the similarity value between the feature vector and the Key value in the KV cache block is greater than a similarity threshold; a target cache determination submodule, used to add the KV cache block to multiple target KV cache blocks of the text to be processed if the similarity value between the feature vector and the Key value in the KV cache block is greater than the similarity threshold; if the similarity value between the feature vector and the Key value in the KV cache block is less than or equal to the similarity threshold, generate a KV buffer block of the semantic vector, and add the KV buffer block of the semantic vector to the multiple target KV cache blocks of the text to be processed.

[0015] Optionally, in an embodiment of the present application, the model inference device also includes: a text acquisition and division module, used to acquire sample text and divide the sample text into multiple sample blocks; a sample vector acquisition module, used to vectorize each sample block in the multiple sample blocks to obtain multiple sample vectors; a KV cache block generation module, used to generate a KV cache block for each sample vector in the multiple sample vectors according to the sample vector and the relative position of the sample vector in the multiple sample vectors.

[0016] Optionally, in an embodiment of the present application, the model inference device also includes: a usage frequency judgment module, used to judge whether the usage frequency of the sample vector is greater than a preset frequency threshold; a cache library pre-filling module, used to pre-fill the KV cache block into the video memory of the graphics processing unit GPU if the usage frequency of the sample vector is greater than the preset frequency threshold, and pre-fill the KV cache block into the memory of the central processing unit CPU if the usage frequency of the sample vector is less than or equal to the preset frequency threshold.

[0017] Optionally, in an embodiment of the present application, the pre-trained language model includes: multiple neural network nodes; the model reasoning device also includes: an activation frequency judgment module, used to judge whether the activation frequency of the neural network node is greater than a preset frequency threshold for each neural network node in the multiple neural network nodes; a network node filling module, used to pre-fill the neural network node into the video memory of the GPU if the activation frequency of the neural network node is greater than the preset frequency threshold, and pre-fill the neural network node into the memory of the CPU if the activation frequency of the neural network node is less than or equal to the preset frequency threshold.

[0018] Optionally, in an embodiment of the present application, the inference result acquisition module includes: a node cache block acquisition submodule, used to obtain a neural network node and a target KV cache block from the video memory of the GPU or the memory of the CPU; an activation frequency judgment submodule, used to judge whether the activation frequency of the neural network node is greater than a preset frequency threshold; a cache block inference submodule, used to infer the target KV cache block through the GPU if the activation frequency of the neural network node is greater than the preset frequency threshold, and to infer the target KV cache block through the CPU if the activation frequency of the neural network node is less than or equal to the preset frequency threshold.

[0019] Optionally, in an embodiment of the present application, the inference result acquisition module includes: a cache block reassembly submodule, used to reassemble multiple target KV cache blocks of the text to be processed to obtain an assembled context KV cache block; a pre-trained model inference submodule, used to infer the assembled context KV cache block through a pre-trained language model.

[0020] An embodiment of the present application further provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions execute the method described above when executed by the processor.

[0021] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is executed.

[0022] The embodiment of the present application also provides a computer program product, including: a computer program or a computer instruction, and the computer program or the computer instruction executes the method described above when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A schematic diagram of the flow of the model reasoning method provided in the embodiment of the present application is shown;

[0025] Figure 2 A schematic diagram of a process for obtaining a target KV cache block provided by an embodiment of the present application is shown;

[0026] Figure 3 A schematic diagram of the flow of the model reasoning device provided in an embodiment of the present application is shown;

[0027] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. It should be understood that the drawings in the embodiment of the present application only serve the purpose of explanation and description, and are not used to limit the protection scope of the embodiment of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in the embodiment of the present application shows the operation implemented according to some embodiments of the embodiment of the present application. It should be understood that the operation of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the embodiment of the present application, or remove one or more operations from the flowchart.

[0029] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application described and shown in the drawings generally herein can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the embodiments of the present application claimed for protection, but merely represents the selected embodiments of the embodiments of the present application.

[0030] It is understandable that the "first" and "second" in the embodiments of the present application are used to distinguish similar objects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit the difference. In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that there may be three relationships, such as A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship. The term "multiple" refers to more than two (including two), and similarly, "multiple groups" refers to more than two groups (including two groups).

[0031] It should be noted that the model reasoning method provided in the embodiment of the present application can be executed by an electronic device, where the electronic device refers to a device terminal having the function of executing a computer program or the above-mentioned server, such as a smart phone, a personal computer, a tablet computer, a personal digital assistant or a mobile Internet device. A server refers to a device that provides computing services through a network, such as an x86 server and a non-x86 server, and non-x86 servers include mainframes, minicomputers and UNIX servers.

[0032] See also Figure 1 The flow chart of the model reasoning method provided by the embodiment of the present application is shown; the main idea of ​​the model reasoning method is that, under the premise of KV cache (KV cache) block scheduling as the center, the similarity value between the semantic vector and the Key value in the pre-filled KV cache block is introduced as the matching standard, rather than the strict exact match as the standard, so that there is a greater probability of realizing the reuse of KV cache blocks, effectively improving the cache hit rate. The implementation method of the above model reasoning method may include:

[0033] Step S110: Divide the text to be processed into a plurality of text blocks.

[0034] The implementation method of the above step S110 is, for example: in the above process of dividing the text to be processed into multiple text blocks, the input text to be processed can be divided into multiple text blocks in a semantic segmentation manner or a separator (such as a paragraph mark, a period, a question mark or a semicolon or other punctuation marks). Among them, the semantic segmentation manner refers to cutting the text based on the boundary of the logical structure or thematic meaning after understanding the logical structure and thematic meaning of the text, for example, using a third-party small language model to divide an article into different chapters or paragraphs, each chapter or paragraph contains a complete argument or information unit.

[0035] Step S120: performing vectorization processing on each of the multiple text blocks to obtain multiple semantic vectors.

[0036] Step S130: for each semantic vector in the multiple semantic vectors, determine multiple target KV cache blocks of the to-be-processed text according to the similarity value between the semantic vector and the Key value in the pre-filled KV cache block.

[0037] It is understandable that when large-scale inference services process sequence data, especially long sequences, in order to improve inference efficiency and reduce memory consumption, they usually use KV cache to store previously calculated Key and Value matrices. In this way, when processing subsequent data, these cached Key and Value can be reused instead of recalculating, which speeds up inference and saves resources.

[0038] The Key value in the above-mentioned KV Cache block stores the semantic vector stored in the pre-filling stage. The above-mentioned KV cache block can be the Key and Value cache block used in the Attention mechanism of the Transformer architecture, so the KV cache block is also called a semantic block. This semantic block relaxes the strict and precise matching requirements during matching, and can improve the reuse probability of the KV cache block. Therefore, it can be applied in large-scale, highly overloaded large-model reasoning scenarios, and effectively reduces the large-scale large-model reasoning service cost while meeting the delay requirements. Among them, the above-mentioned Transformer architecture is divided into a self-attention module and an FFN module, of which the Feed Forward Networks (FFN) module occupies about 66% of the computational workload, while the computational delay of the self-attention module accounts for about 60%.

[0039] Step S140: Inferring multiple target KV cache blocks of the text to be processed through the pre-trained language model to obtain an inference result.

[0040] It can be understood that the reasoning process of the above-mentioned pre-trained language model is divided into two stages to respond to user queries, which include: a prefill stage and a decoding stage, wherein the prefill stage processes a user prompt consisting of a series of tokens to generate the first token of the user response in one step, and the decoding stage generates subsequent tokens sequentially in multiple steps, and each decoding step generates a new token based on the tokens generated in the previous step until the termination token is reached.

[0041] In the implementation process of the above scheme, the text to be processed is divided into multiple text blocks, thereby reducing the size of each block. Since smaller text blocks are more likely to have appeared in previous requests, they are more likely to find a matching cache, thereby increasing the possibility of cache hit rate. Furthermore, each text block is vectorized to obtain a semantic vector that represents the intention of the text block. Even if the contents of the text blocks are slightly different, but the information conveyed is similar, the target KV cache block with a similar meaning to the text block to be processed can be found based on the similarity value between the semantic vector and the Key value in the pre-filled KV cache block, effectively relaxing the requirement for strict and precise matching of the semantic vector and the Key value, thereby having a greater probability of realizing the reuse of the KV cache block, effectively improving the cache hit rate.

[0042] As an optional implementation of the above step S120, the implementation of vectorizing each of the multiple text blocks may include: using a Byte Pair Encoding (BPE) tool to tokenize each of the multiple text blocks, thereby obtaining words or subwords, and using a pre-trained word embedding model to map the words or subwords to a high-dimensional space to obtain a mapped semantic vector. The pre-trained word embedding model may adopt an embedding layer of a Word2Vec model, a GloVe model, a BERT (Bidirectional Encoder Representation from Transformers) model, or an embedding layer of a BART (Bidirectional and Auto-Regressive Transformers).

[0043] Optionally, if it is necessary to store a feature vector that is a fusion of the semantic vector and the relative position in the Key value, the semantic vector and the relative position can also be fused to obtain the feature vector that needs to be stored in the Key value. The fusion here can be directly adding the relative position to the semantic vector, or splicing the relative position in front of or behind the semantic vector.

[0044] As an optional implementation of the above step S130, the Key value in the above KV cache block may store a semantic vector, and then multiple target KV cache blocks of the to-be-processed text may be determined according to the similarity value between the semantic vector and the Key value in the pre-filled KV cache block. This implementation may include:

[0045] Step S131: determine whether the similarity value between the semantic vector and the Key value in the KV cache block is greater than a similarity threshold.

[0046] An implementation method of the above-mentioned step S131, for example, may use an efficient vector retrieval algorithm to determine whether the similarity value between the semantic vector as the Key and the Key value in the KV cache block is greater than a similarity threshold; wherein the vector retrieval algorithm may adopt the Faiss retrieval algorithm, which may accelerate the local KV cache (KVcache) block that is most similar to the current text block, and arrange the corresponding KV cache block into the KV cache of the long context, which may be stored when pre-filled in the memory of the CPU, or may be stored when pre-filled in the video memory of the GPU.

[0047] Step S132: If the similarity value between the semantic vector and the Key value in the KV cache block is greater than the similarity threshold, the KV cache block is added to the multiple target KV cache blocks of the text to be processed.

[0048] See also Figure 2 The schematic diagram of the acquisition process of the target KV cache block provided by the embodiment of the present application is shown; the implementation method of the above step S132 is, for example: if the similarity value between the semantic vector and the Key value in the KV cache block is greater than the similarity threshold, the KV cache library greater than the similarity threshold can be determined as a matching KV cache block, and the matching KV cache block is added to the multiple target KV cache blocks of the text to be processed, that is, the matching KV cache block is synchronously sent to the decoding instance, so that the decoding instance determines the matching KV cache block as the added target KV cache block, and uses the GPU or CPU (determined by the activation frequency of the neural network node) to infer the target KV cache block, thereby reducing the overall large model inference service delay. Among them, the above-mentioned KV cache block for comparing similarity can be loaded from the KV cache pool of the GPU, or it can be loaded from the KV cache pool of the CPU.

[0049] Optionally, in the specific practice, the frequency or frequency of orchestration of each KV cache block can also be analyzed, and the KV cache block can be load balanced and resource scheduled according to the frequency or frequency, which accelerates the process of determining the target KV cache block, thereby reducing the delay time of the large model reasoning process, especially the long context pre-filled reasoning. Among them, orchestration refers to the process of coordinating and managing multiple KV cache blocks, components and services to ensure that KV cache blocks, components and services can work together to complete tasks. In the context of large model reasoning services, the purpose of orchestrating KV Cache (Key-Value Cache) is to improve the efficiency and performance of the service.

[0050] Step S133: If the similarity value between the semantic vector and the Key value in the KV cache block is less than or equal to the similarity threshold, a KV buffer block of the semantic vector is generated, and the KV buffer block of the semantic vector is added to multiple target KV cache blocks of the text to be processed.

[0051] For example, the implementation method of the above step S133 is as follows: Figure 2 , if the similarity value between the semantic vector and the Key value in the KV cache block is less than or equal to the similarity threshold, the incremental KV buffer block of the semantic vector is generated using a distributed pipeline parallel method, and the incremental KV buffer block of the semantic vector is added to multiple target KV cache blocks of the text to be processed, that is, the incremental KV cache block is synchronously sent to the decoding instance, so that the decoding instance determines the incremental KV cache block as the target KV cache block to be added, and uses the GPU or CPU (determined by the activation frequency of the neural network node) to reason about the target KV cache block, thereby reducing the overall large model reasoning service delay. Among them, the above-mentioned KV cache block for comparing similarity can be loaded from the KV cache pool of the GPU, or from the KV cache pool of the CPU. Specifically, the incremental KV buffer block of the semantic vector can be generated by a distributed pipeline parallel method of pre-filling the instance, and then the incremental KV buffer block of the semantic vector is added to multiple target KV cache blocks of the text to be processed by the pre-filling instance. Optionally, when the large model inference service in the data center is at a low point, the distribution results of the semantic vector and the KV buffer block corresponding to the semantic vector generated in a distributed pipeline parallel manner can be statistically calculated, and then the semantic vector and the KV buffer block corresponding to the semantic vector in the GPU video memory and CPU memory can be adjusted and arranged according to the distribution results.

[0052] In the implementation process of the above scheme, by comparing the similarity between the newly generated semantic vector and the Key value in the existing KV cache block, it is decided whether to reuse the existing KV cache block or generate a new KV cache block. Therefore, the target KV cache block with similar meaning to the text block to be processed can be found through the similarity value between the semantic vector and the Key value in the pre-filled KV cache block, which effectively relaxes the requirement of strict and precise matching of the semantic vector and the Key value, thereby reducing repeated calculations, enabling the model to generate output faster, and improving the response speed of the entire system.

[0053] As another optional implementation of the above step S130, the Key value in the above KV cache block may also store a feature vector fused with a semantic vector and a relative position. Then, the implementation of determining multiple target KV cache blocks of the to-be-processed text according to the similarity value between the feature vector fused with the semantic vector and the relative position and the Key value in the pre-filled KV cache block may include:

[0054] Step S134: The semantic vector and the relative position of the semantic vector in multiple semantic vectors are merged to obtain a feature vector, and it is determined whether the similarity value between the feature vector and the Key value in the KV cache block is greater than a similarity threshold.

[0055] The implementation methods of the above step S134 include: a first addition fusion method, for example: adding the semantic vector in the relative position of multiple semantic vectors to the semantic vector, thereby obtaining the added feature vector, and judging whether the similarity value between the added feature vector and the Key value in the KV cache block is greater than the similarity threshold. A second splicing fusion method, for example: splicing the semantic vector in the relative position of multiple semantic vectors in front of or behind the semantic vector, thereby obtaining the spliced ​​feature vector, and judging whether the similarity value between the spliced ​​feature vector and the Key value in the KV cache block is greater than the similarity threshold.

[0056] Step S135: If the similarity value between the feature vector and the Key value in the KV cache block is greater than the similarity threshold, the KV cache block is added to the multiple target KV cache blocks of the text to be processed.

[0057] The implementation method of the above step S135 includes: if the similarity value between the feature vector and the Key value in the KV cache block is greater than the similarity threshold, the KV cache library greater than the similarity threshold can be determined as a matching KV cache block, and the matching KV cache block is added to the multiple target KV cache blocks of the text to be processed. In other words, the matching KV cache block is synchronously sent to the decoding instance so that the decoding instance determines the matching KV cache block as the target KV cache block to be added, and uses the GPU or CPU (determined by the activation frequency of the neural network node) to reason about the target KV cache block, thereby reducing the overall large model reasoning service delay. Among them, the above-mentioned KV cache block for comparing similarity can be loaded from the KV cache pool of the GPU, or it can be loaded from the KV cache pool of the CPU.

[0058] Step S136: If the similarity value between the feature vector and the Key value in the KV cache block is less than or equal to the similarity threshold, a KV buffer block of the semantic vector is generated, and the KV buffer block of the semantic vector is added to multiple target KV cache blocks of the text to be processed.

[0059] The implementation method of the above step S136 includes: if the similarity value between the feature vector and the Key value in the KV cache block is less than or equal to the similarity threshold, a distributed pipeline parallel method is used to generate a KV buffer block of the semantic vector, and the KV buffer block of the semantic vector is added to multiple target KV cache blocks of the text to be processed. In other words, the incremental KV cache block is synchronously sent to the decoding instance so that the decoding instance determines the incremental KV cache block as the target KV cache block to be added, and uses the GPU or CPU (determined by the activation frequency of the neural network node) to infer the target KV cache block, thereby reducing the overall large model inference service delay. Among them, the above-mentioned KV cache block for comparing similarity can be loaded from the KV cache pool of the GPU, or it can be loaded from the KV cache pool of the CPU.

[0060] As an optional implementation of the above model inference method, before determining multiple target KV cache blocks of the text to be processed according to the similarity value between the semantic vector and the Key value in the pre-filled KV cache block, the above KV cache block can also be generated. This implementation may include:

[0061] Step S137: Acquire sample text, and divide the sample text into multiple sample blocks.

[0062] The implementation method of the above step S137 is, for example: obtaining a sample text, and dividing the sample text into multiple sample blocks in a semantic segmentation manner (such as chapter breaks or section breaks) or a separator (such as a paragraph break, a period). The semantic segmentation manner refers to cutting the text based on the boundary of the logical structure or the subject meaning after understanding the logical structure and subject meaning of the text. For example, using chapter breaks or section breaks can divide an article into different chapters or paragraphs, each of which contains a complete argument or information unit.

[0063] Step S138: performing vectorization processing on each sample block in the multiple sample blocks to obtain multiple sample vectors.

[0064] The implementation of the above step S138 is, for example: using a byte pair encoding (BPE) tool to segment each sample in the multiple sample blocks, thereby obtaining words or sub-words, and using a pre-trained word embedding model to map the words or sub-words to a high-dimensional space to obtain a mapped semantic vector. The pre-trained word embedding model can use a Word2Vec model, a GloVe model, an embedding layer of a BERT model, or an embedding layer of a BART.

[0065] Step S139: for each sample vector in the multiple sample vectors, generate a KV cache block according to the sample vector and the relative position of the sample vector in the multiple sample vectors.

[0066] The implementation method of the above step S139 is, for example: for each sample vector in a plurality of sample vectors, the relative position of the sample vector and the sample vector in a plurality of sample vectors is fused to obtain a fused feature vector, and the fused feature vector is used as a key (Key), and the intermediate result generated by the feature vector in the pre-filling stage is used as a value (Value), to generate the above-mentioned KV cache block. It can be understood that the intermediate results generated in these pre-filling stages can be reused in the decoding stage (decoding), avoiding repeated calculation of semantically similar intermediate results (i.e., the similarity value between the vector and the Key value in the KV cache block is greater than the similarity threshold), thereby improving the reasoning efficiency. Optionally, after generating the KV cache block, the distribution update strategy of the KV cache block can also be set to update the locally stored KV cache block in real time according to the retrieval result of the KV cache block, so as to reduce the arrangement time of the KV cache block, thereby improving the retrieval hit rate of the local KV cache.

[0067] As an optional implementation of the above model inference method, after generating a KV cache block according to the sample vector and the relative position of the sample vector in multiple sample vectors, the KV cache block may also be pre-filled into a video memory or a memory. This implementation may include:

[0068] Step S139a: Determine whether the usage frequency of the sample vector is greater than a preset frequency threshold.

[0069] Step S139b: If the usage frequency of the sample vector is greater than a preset frequency threshold, the KV cache block is pre-filled into the video memory of the graphics processor GPU.

[0070] For example, the implementation method of the above step S139b is as follows: if the usage frequency of the sample vector is greater than the preset frequency threshold, it means that these sample vectors are frequently used, and the sample vector and the KV cache block can be pre-filled into the video memory of the graphics processor GPU through the pre-filling instance, that is, the sample vector and the KV cache block are sent to the video memory of the graphics processor GPU. The above-mentioned pre-filling instance refers to a program instance that completes the operation in the pre-filling stage, and the instance can be run as a service or as a background program.

[0071] Step S139c: If the usage frequency of the sample vector is less than or equal to the preset frequency threshold, the KV cache block is pre-filled into the memory of the central processing unit CPU.

[0072] An implementation example of the above step S139c is as follows: if the usage frequency of the sample vector is less than or equal to the preset frequency threshold, it means that these sample vectors are not frequently used vectors. At this time, the sample vector and the KV cache block can be pre-filled into the memory of the central processing unit CPU through a pre-filling instance, that is, the sample vector and the KV cache block are sent to the memory of the central processing unit CPU.

[0073] As an optional implementation of the above step S140, the above pre-trained language model may include: a plurality of neural network nodes; before the pre-trained language model is used to infer the plurality of target KV cache blocks of the to-be-processed text, the neural network nodes may also be pre-filled into a video memory or a memory. This implementation may include:

[0074] Step S141: For each neural network node among the multiple neural network nodes, determine whether the activation frequency of the neural network node is greater than a preset frequency threshold.

[0075] The implementation method of the above step S141 is, for example: It can be understood that the neural network node above can be a neural network node in a feedforward neural network (FFN) module of large model reasoning. Before reasoning, such neural network node can count the activation frequency of each neural network node through a large amount of training data in the offline analysis stage. The activation evaluation here can use the activation frequency of different neural network nodes in the FFN layer in the large model reasoning service in the last time period, or the frequency of the neural network node being used for calculation after activation. After counting the activation frequency of each neural network node, it can also be determined whether the activation frequency of the neural network node is greater than a preset frequency threshold. Optionally, after counting the activation frequency of each neural network node, the activation frequency of each neural network node (for example, in the FFN layer) can also be stored in the configuration file corresponding to each neural network node, so that the configuration file is used by the neural network node in the decoding stage, thereby accelerating the model reasoning process. For example: Use the configuration file to load the corresponding neural network nodes in the GPU and CPU respectively, so that after the different neural network nodes of the FFN layer are loaded, they can be in the corresponding position in the large model reasoning process. Among them, the above configuration file can store the position information of different neural network nodes of the FFN layer during the large model inference process.

[0076] Step S142: If the activation frequency of the neural network node is greater than a preset frequency threshold, the neural network node is pre-filled into the video memory of the GPU.

[0077] An implementation example of the above-mentioned step S142 is as follows: if the activation frequency of a neural network node is greater than a preset frequency threshold, it means that the neural network node is a hot node, that is, the neural network node will be frequently activated. Therefore, the neural network node can be pre-filled into the video memory of a GPU with faster matrix calculation through a pre-filled instance, so that the GPU can load and use the neural network node for analysis and reasoning, thereby speeding up the rate of analysis and reasoning.

[0078] Step S143: If the activation frequency of the neural network node is less than or equal to the preset frequency threshold, the neural network node is pre-filled into the memory of the CPU.

[0079] An implementation example of the above-mentioned step S143 is: if the activation frequency of the neural network node is less than or equal to the preset frequency threshold, it means that the neural network node is a cold node, that is, the neural network node will not be activated frequently. Therefore, the neural network node can be pre-filled into the memory of the CPU with slower matrix calculation.

[0080] In the implementation process of the above scheme, by utilizing the characteristics of hot and cold power-law distribution of neural network nodes, hot nodes that are frequently activated and called are pre-filled into the GPU's video memory, and cold nodes that are not frequently called are pre-filled into the CPU's memory, so that the CPU in the inference decoding process implements cold node reasoning, and the GPU in the inference decoding process implements hot node reasoning, effectively completing the heterogeneous computing function. This heterogeneous computing function can greatly reduce the computing power cost and storage cost of large model reasoning services, thereby reducing the overall cost of large model reasoning services.

[0081] As another optional implementation of the above step S140, the above implementation of reasoning on multiple target KV cache blocks of the to-be-processed text by using the pre-trained language model may include:

[0082] Step S144: Obtain the neural network node and target KV cache block from the video memory of the GPU or the memory of the CPU.

[0083] The implementation method of the above step S144 is, for example: obtaining a neural network node from the video memory of the GPU or the memory of the CPU (determined by the activation frequency of the neural network node recorded in the configuration information), specifically, if the activation frequency of the neural network node is greater than the preset frequency threshold, then the neural network node is obtained from the video memory of the GPU, and correspondingly, if the activation frequency of the neural network node is less than or equal to the preset frequency threshold, then the neural network node is obtained from the video memory of the CPU. Then, the target KV cache block is obtained from the KV cache pool in the video memory of the GPU or the KV cache pool in the memory of the CPU (determined by the usage frequency of the target KV cache block recorded in the configuration information), specifically, if the usage frequency of the target KV cache block is greater than the preset frequency threshold, then the target KV cache block is obtained from the KV cache pool in the video memory of the GPU, and if the usage frequency of the target KV cache block is less than or equal to the preset frequency threshold, then the target KV cache block is obtained from the KV cache pool in the memory of the CPU.

[0084] Step S145: Determine whether the activation frequency of the neural network node is greater than a preset frequency threshold.

[0085] Step S146: If the activation frequency of the neural network node is greater than the preset frequency threshold, the target KV cache block is inferred through the GPU.

[0086] Step S147: If the activation frequency of the neural network node is less than or equal to the preset frequency threshold, the target KV cache block is inferred through the CPU.

[0087] The implementation method of the above steps S145 to S147 is, for example: judging whether the activation frequency of the neural network node is greater than the preset frequency threshold through the executable program in the operating system. If the activation frequency of the neural network node is greater than the preset frequency threshold, it means that the neural network node is a hot node, and the target KV cache block can be inferred by the neural network node through the GPU. Similarly, if the activation frequency of the neural network node is less than or equal to the preset frequency threshold, it means that the neural network node is a cold node, and the target KV cache block can be inferred by the neural network node through the CPU.

[0088] As another optional implementation of the above step S140, the above implementation of reasoning on multiple target KV cache blocks of the to-be-processed text by using the pre-trained language model may include:

[0089] Step S148: reassemble multiple target KV cache blocks of the text to be processed to obtain an assembled context KV cache block.

[0090] Step S149: Reasoning the assembled context KV cache block through the pre-trained language model.

[0091] An implementation method of the above steps S148 to S149 is, for example: in the decoding stage of the large model inference service, multiple target KV cache blocks of the text to be processed are reassembled to obtain an assembled context KV cache block; wherein the above multiple target KV cache blocks include: a KV cache block retrieved from the GPU video memory or the CPU memory, and a KV buffer block generated according to the semantic vector. In the implementation process of the above scheme, by reassembling multiple target KV cache blocks, the system can more efficiently utilize the information in the cache, which means that even if some parts of the input text are new or changed, as long as most of the content is repeated, the system can reuse the existing cache and reduce repeated calculations, thereby effectively improving the cache hit rate and utilization rate.

[0092] See also Figure 3 The flowchart of the model reasoning device provided in the embodiment of the present application is shown; the embodiment of the present application provides a model reasoning device 200, including:

[0093] The text block division module 210 is used to divide the text to be processed into multiple text blocks.

[0094] The text vectorization module 220 is used to perform vectorization processing on each text block in the multiple text blocks to obtain multiple semantic vectors.

[0095] The cache block determination module 230 is used to determine, for each semantic vector in the multiple semantic vectors, multiple target KV cache blocks of the to-be-processed text according to the similarity value between the semantic vector and the Key value in the pre-filled KV cache block.

[0096] The inference result obtaining module 240 is used to perform inference on multiple target KV cache blocks of the text to be processed through a pre-trained language model to obtain an inference result.

[0097] As an optional implementation manner of the above device, the cache block determination module includes:

[0098] The similarity value judgment submodule is used to judge whether the similarity value between the semantic vector and the Key value in the KV cache block is greater than a similarity threshold.

[0099] The target cache block determination submodule is used to add the KV cache block to multiple target KV cache blocks of the text to be processed if the similarity value between the semantic vector and the Key value in the KV cache block is greater than the similarity threshold; if the similarity value between the semantic vector and the Key value in the KV cache block is less than or equal to the similarity threshold, generate a KV buffer block of the semantic vector, and add the KV buffer block of the semantic vector to multiple target KV cache blocks of the text to be processed.

[0100] As an optional implementation manner of the above device, the cache block determination module includes:

[0101] The feature vector acquisition submodule is used to fuse the semantic vector and the relative position of the semantic vector in multiple semantic vectors to obtain a feature vector.

[0102] The similarity value judgment submodule is used to judge whether the similarity value between the feature vector and the Key value in the KV cache block is greater than the similarity threshold.

[0103] The target cache determination submodule is used to add the KV cache block to multiple target KV cache blocks of the text to be processed if the similarity value between the feature vector and the Key value in the KV cache block is greater than the similarity threshold; if the similarity value between the feature vector and the Key value in the KV cache block is less than or equal to the similarity threshold, generate a KV buffer block of the semantic vector, and add the KV buffer block of the semantic vector to multiple target KV cache blocks of the text to be processed.

[0104] As an optional implementation of the above device, the model reasoning device further includes:

[0105] The text acquisition and division module is used to acquire sample text and divide the sample text into multiple sample blocks.

[0106] The sample vector obtaining module is used to perform vectorization processing on each sample block in the multiple sample blocks to obtain multiple sample vectors.

[0107] The KV cache block generation module is used to generate a KV cache block for each sample vector in the multiple sample vectors according to the sample vector and the relative position of the sample vector in the multiple sample vectors.

[0108] As an optional implementation of the above device, the model reasoning device further includes:

[0109] The usage frequency determination module is used to determine whether the usage frequency of the sample vector is greater than a preset frequency threshold.

[0110] The cache library pre-filling module is used to pre-fill the KV cache block into the video memory of the graphics processing unit GPU if the usage frequency of the sample vector is greater than the preset frequency threshold, and to pre-fill the KV cache block into the memory of the central processing unit CPU if the usage frequency of the sample vector is less than or equal to the preset frequency threshold.

[0111] As an optional implementation of the above device, the pre-trained language model includes: a plurality of neural network nodes; the model reasoning device also includes:

[0112] The activation frequency judgment module is used to judge whether the activation frequency of each neural network node among multiple neural network nodes is greater than a preset frequency threshold.

[0113] The network node filling module is used to pre-fill the neural network node into the GPU's video memory if the activation frequency of the neural network node is greater than a preset frequency threshold, and to pre-fill the neural network node into the CPU's memory if the activation frequency of the neural network node is less than or equal to the preset frequency threshold.

[0114] As an optional implementation of the above device, the inference result obtaining module includes:

[0115] The node cache block acquisition submodule is used to obtain the neural network nodes and target KV cache blocks from the GPU video memory or the CPU memory.

[0116] The activation frequency judgment submodule is used to judge whether the activation frequency of the neural network node is greater than a preset frequency threshold.

[0117] The cache block inference submodule is used to infer the target KV cache block through the GPU if the activation frequency of the neural network node is greater than the preset frequency threshold, and to infer the target KV cache block through the CPU if the activation frequency of the neural network node is less than or equal to the preset frequency threshold.

[0118] As an optional implementation of the above device, the inference result obtaining module includes:

[0119] The cache block reassembly submodule is used to reassemble multiple target KV cache blocks of the text to be processed to obtain an assembled context KV cache block.

[0120] The pre-trained model reasoning submodule is used to reason about the assembled context KV cache block through the pre-trained language model.

[0121] It should be understood that the device corresponds to the above-mentioned model reasoning method embodiment and can execute the various steps involved in the above-mentioned method embodiment. The specific functions of the device can be found in the description above, and the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system (OS) of the device.

[0122] See also Figure 4 The electronic device 300 provided in the embodiment of the present application includes: a processor 310 and a memory 320, wherein the memory 320 stores machine-readable instructions executable by the processor 310, and when the machine-readable instructions are executed by the processor 310, the above method is executed.

[0123] The embodiment of the present application also provides a computer-readable storage medium 330, on which a computer program is stored, and the computer program is executed by the processor 310 to execute the above method. The computer-readable storage medium 330 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.

[0124] The embodiment of the present application also provides a computer program product, including: a computer program or a computer instruction, and the computer program or the computer instruction executes the method described above when executed by a processor.

[0125] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0126] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed device and method can also be implemented in other ways. The device embodiments described above are only schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module, a program segment or a code, and a part of a module, a program segment or a code includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also be different from the order of occurrence marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which is mainly determined by the functions involved.

[0127] In addition, each functional module of each embodiment in the embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. In addition, in the description of this specification, the description of reference terms "one embodiment", "some embodiments", "example", "specific example", "some examples", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present application. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples without contradiction.

[0128] The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. A model reasoning method, characterized in that: include: Divide the text to be processed into multiple text blocks; Performing vectorization processing on each of the multiple text blocks to obtain multiple semantic vectors; For each semantic vector in the plurality of semantic vectors, determining a plurality of target KV cache blocks of the to-be-processed text according to a similarity value between the semantic vector and a Key value in a pre-filled KV cache block; Inference is performed on multiple target KV cache blocks of the text to be processed through a pre-trained language model to obtain an inference result.

2. The method according to claim 1, characterized in that Determining a plurality of target KV cache blocks of the to-be-processed text according to the similarity value between the semantic vector and the Key value in the pre-filled KV cache block includes: Determine whether the similarity value between the semantic vector and the Key value in the KV cache block is greater than the similarity threshold; If so, the KV cache block is added to the multiple target KV cache blocks of the text to be processed; otherwise, a KV buffer block of the semantic vector is generated and the KV buffer block of the semantic vector is added to the multiple target KV cache blocks of the text to be processed.

3. The method according to claim 1, characterized in that Determining a plurality of target KV cache blocks of the to-be-processed text according to the similarity value between the semantic vector and the Key value in the pre-filled KV cache block includes: fusing the semantic vector and the relative position of the semantic vector at the plurality of semantic vectors to obtain a feature vector; Determine whether the similarity value between the feature vector and the Key value in the KV cache block is greater than a similarity threshold; If so, the KV cache block is added to the multiple target KV cache blocks of the text to be processed; otherwise, a KV buffer block of the semantic vector is generated and the KV buffer block of the semantic vector is added to the multiple target KV cache blocks of the text to be processed.

4. The method according to claim 1, characterized in that Before determining the multiple target KV cache blocks of the to-be-processed text according to the similarity value between the semantic vector and the Key value in the pre-filled KV cache block, the method further includes: Obtaining a sample text, and dividing the sample text into a plurality of sample blocks; Performing vectorization processing on each sample block of the multiple sample blocks to obtain multiple sample vectors; For each sample vector in the multiple sample vectors, the KV cache block is generated according to the sample vector and the relative position of the sample vector in the multiple sample vectors.

5. The method according to claim 4, characterized in that After generating the KV cache block according to the sample vector and the relative position of the sample vector among the plurality of sample vectors, the method further includes: Determine whether the usage frequency of the sample vector is greater than a preset frequency threshold; If yes, the KV cache block is pre-filled into the video memory of the graphics processing unit GPU; otherwise, the KV cache block is pre-filled into the memory of the central processing unit CPU.

6. The method according to claim 5, characterized in that The pre-trained language model includes: a plurality of neural network nodes; before the pre-trained language model is used to infer a plurality of target KV cache blocks of the text to be processed, it also includes: For each neural network node among the plurality of neural network nodes, determining whether an activation frequency of the neural network node is greater than a preset frequency threshold; If so, the neural network node is pre-filled into the video memory of the GPU; otherwise, the neural network node is pre-filled into the memory of the CPU.

7. The method according to claim 6, characterized in that The reasoning of the multiple target KV cache blocks of the to-be-processed text by using the pre-trained language model includes: Obtain the neural network node and the target KV cache block from the video memory of the GPU or the memory of the CPU; Determining whether the activation frequency of the neural network node is greater than a preset frequency threshold; If so, the target KV cache block is inferred through the GPU; otherwise, the target KV cache block is inferred through the CPU.

8. The method according to any one of claims 1 to 6, characterized in that: The reasoning of the multiple target KV cache blocks of the to-be-processed text by using the pre-trained language model includes: Reassembling multiple target KV cache blocks of the text to be processed to obtain an assembled context KV cache block; The assembled context KV cache block is inferred through a pre-trained language model.

9. A model reasoning device, characterized in that: include: A text block division module is used to divide the text to be processed into multiple text blocks; A text vectorization module, used for performing vectorization processing on each of the multiple text blocks to obtain multiple semantic vectors; A cache block determination module, used for determining, for each semantic vector in the plurality of semantic vectors, a plurality of target KV cache blocks of the text to be processed according to a similarity value between the semantic vector and a key value in a pre-filled KV cache block; The inference result acquisition module is used to infer the multiple target KV cache blocks of the text to be processed through a pre-trained language model to obtain an inference result.

10. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions are executed by the processor to perform any method according to claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is executed.

12. A computer program product, characterized in that include: A computer program or a computer instruction, wherein when the computer program or the computer instruction is executed by a processor, the method according to any one of claims 1 to 8 is executed.

Images