A reasoning task processing method, device, computer program product and storage medium

Abstract

Embodiments of the present application provide a reasoning task processing method and device, a computer program product and a storage medium. On the one hand, key-value information generated by a target model for a reasoning task can be stored in an auxiliary storage device to reduce memory usage. On the other hand, a general codebook shared by multiple reasoning tasks is preset, and after receiving a reasoning task, the general codebook can be directly used to determine the corresponding code word for the stored key-value information. In the decoding operation process, the general codebook can be used to find the target code word related to decoding, and the key-value information corresponding to the target code word can be read back from the auxiliary storage device. Accordingly, when different reasoning tasks are received, the general codebook can be reused, and there is no need to construct a special codebook for the reasoning task in real time, thereby effectively avoiding the occupation of reasoning time caused by constructing a special codebook.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a reasoning task processing method, device, computer program product, and storage medium. Background Technology

[0002] In the field of Natural Language Processing (NLP), large-scale language models have become a hot topic in research and application. As the model size continues to increase, the computational and storage overhead in the autoregressive inference process also increases significantly.

[0003] During autoregressive inference, the model generates a large number of keys and corresponding values, which can be referred to as key-value information. As large language models are applied to more complex inference scenarios, the amount of key-value information generated within the model increases, and the memory space occupied by the model also increases, sometimes exceeding the memory capacity of the inference device. Moreover, the memory space allowed for autoregressive inference is limited, thus leading to excessively high inference latency.

[0004] The current solution to this problem is to directly delete some key-value information, thereby saving memory space in the inference device. While this improves the problem of excessive inference latency, key-value information is a crucial basis for autoregressive inference. Deleting key-value information may lead to insufficient information during autoregressive inference, affecting the quality of inference. Summary of the Invention

[0005] This application provides a reasoning task processing method, device, computer program product, and storage medium to improve the reasoning performance of a model.

[0006] This application provides a reasoning task processing method, including: In response to receiving an inference task, a pre-set general codebook is used to determine the corresponding codeword for the key-value information generated by the target model in the pre-filling stage for the inference task; and the key-value information corresponding to the inference task is stored in an auxiliary storage device. During the decoding operation performed by the target model for the inference task, the target codeword related to the current decoding operation is searched in the general codebook. The key-value information corresponding to the target codeword is read back from the auxiliary storage device for the target model, and used as the reasoning basis for this decoding operation.

[0007] Furthermore, using a pre-built universal codebook, the codewords corresponding to the key-value information generated by the target model for the inference task are determined, including: For any key-value information generated by the target model for the inference task, vector quantization is performed to determine the codeword corresponding to the key-value information in the general codebook; The general codebook contains codewords generated based on vector quantization technology.

[0008] Furthermore, the process of constructing the general codebook includes: Obtain a sample set, which contains text samples and / or visual samples capable of covering multiple reasoning tasks, as training samples; Using the target model, key-value information is generated for different training samples in the sample set; The key-value information generated for different training samples is converted into feature vectors, and the converted feature vectors are clustered to obtain multiple clusters. Indexes are created for representative vectors in the multiple clusters, which are then used as codewords to construct the general codebook.

[0009] Further, searching for the target codeword related to this decoding operation in the general codebook includes: Determine the query vector constructed by the target model for this decoding operation; Calculate the correlation between the query vector and the representative vectors corresponding to each codeword in the general codebook; Based on the calculated correlation, the target codeword related to this decoding operation is searched in the general codebook.

[0010] Further, the correlation between the query vector and the representative vectors corresponding to each codeword in the general codebook is calculated, including: The inner product between the query vector and the representative vector corresponding to each codeword in the general codebook is calculated to characterize the correlation.

[0011] Furthermore, based on the calculated correlation, the target codeword related to this decoding operation is searched in the general codebook, including: The K codewords in the general codebook that have the highest correlation with the query vector are determined as the target codewords; Where K is a positive integer.

[0012] Furthermore, the method also includes: An association is established between the word position information corresponding to the key value information and the codeword determined for the key value information, so as to construct the association between word position information and codeword for the reasoning task.

[0013] Further, reading back key-value information corresponding to the target codeword from the auxiliary storage device for the target model includes: Based on the association between the word position information and the codeword corresponding to the reasoning task, the target word position information associated with the target codeword is determined; Based on the target word location information, a read-back instruction is sent to the auxiliary storage device so that the auxiliary storage device can hit the key value information required for this decoding operation according to the target word location information. The auxiliary storage device stores key-value information for the reasoning task using word position information as an index.

[0014] This application also provides a computing device, including a memory, a processor, and a communication component; The memory is used to store one or more computer instructions; The processor is coupled to the memory and the communication component and is used to execute one or more computer instructions to perform the aforementioned inference task processing method.

[0015] This application also provides a computer-readable storage medium for storing a computer program, which, when executed by one or more processors, causes the one or more processors to perform the aforementioned inference task processing method.

[0016] This application also provides a computer program product, including a computer program that, when executed by one or more processors, causes the one or more processors to perform the aforementioned inference task processing method.

[0017] This application proposes a method for processing inference tasks. On one hand, the key-value information generated by the target model for the inference task can be transferred to an auxiliary storage device, effectively reducing the memory footprint on the inference device where the target model resides, thus avoiding problems such as excessive inference latency. On the other hand, a universal codebook that can be shared by multiple inference tasks is pre-configured. Based on this universal codebook, upon receiving an inference task, the codebook can be directly used to determine the corresponding codeword for the transferred key-value information. Furthermore, during any decoding operation performed on the inference task, the universal codebook can be used to find the target codeword related to decoding and read back the key-value information corresponding to the target codeword from the auxiliary storage device for the target model. Therefore, in this application embodiment, when receiving different inference tasks, the universal codebook can be reused without needing to build a dedicated codebook for the inference task in real time, effectively avoiding the time consumed by building a dedicated codebook, ensuring inference efficiency, and improving inference performance. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a reasoning task processing method provided for an exemplary embodiment of this application; Figure 2 A logical schematic diagram of a reasoning task processing method provided for an exemplary embodiment of this application; Figure 3 A schematic diagram of a model reasoning process based on a universal codebook is provided for an exemplary embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computing device provided for another exemplary embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] It should be noted that, in the cases involving user information in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) comply with relevant laws and standards.

[0021] Before proceeding with a detailed description of the technical solutions provided in the various embodiments of this application, the following is a brief explanation of several technical concepts involved in this application.

[0022] Large-scale language models are artificial intelligence models based on deep learning technology, possessing powerful language understanding and generation capabilities.

[0023] Autoregressive inference is a common inference method used by large language models when handling inference tasks. Its basic principle is to predict the next output word based on the generated word sequence. The autoregressive inference process includes two stages: the prefill stage and the decoder stage. These two stages are closely related and cooperate with each other to generate a high-quality output sequence.

[0024] The pre-filling stage is mainly to prepare for the entire autoregressive inference process. It can segment the original input of the inference task into tokens to obtain the token sequence corresponding to the inference task. The token sequence can be encoded by the encoder in the model to generate the generation key and the corresponding value of each token in the token sequence, which can be called key-value information.

[0025] Key-value cache (KV cache) is the key-value information generated in the model for the inference task. It can be filled in the memory space of the inference device where the model is located. For example, the inference device can be a graphics processing unit (GPU), and the key-value information can be filled in the GPU's video memory.

[0026] The decoding phase, building upon the pre-filling phase, uses the pre-filled key-value cache and model parameters to progressively generate the output sequence. This phase typically involves multiple rounds of decoding. Each decoding operation relies on the key-value information provided in the pre-filling phase and the key-value information corresponding to previously output terms. By continuously querying the key-value cache and performing model calculations, output terms are predicted. Thus, by sequentially performing decoding operations in the decoding phase, output terms are generated sequentially, resulting in the output sequence corresponding to the inference task.

[0027] A token is a fundamental and crucial concept in Natural Language Processing (NLP). It refers to the smallest unit or basic element used to represent text during NLP processing, and can be a character, word, letter, number, or punctuation mark. In some large-scale language models that support multimodal processing, data from modalities such as video and images can also be represented as visual tokens.

[0028] As described in the background section, large language models are being used in increasingly complex inference scenarios, leading to a rapid increase in the demand for context length. This embodiment supports large language models for handling inference tasks such as long document question answering, codebase-level code understanding, and complex logical reasoning. While continuous training or extrapolation can enable models to support larger context lengths, inference with long contexts faces significant efficiency challenges. This is because the size of the key-value cache (KVCache) increases linearly with the context length, potentially exceeding the capacity of GPU memory. Furthermore, the decoding phase in autoregressive inference is often the performance bottleneck and is memory-constrained; therefore, the continuous increase in the key-value cache affects the inference efficiency during the decoding phase, resulting in excessively high inference latency.

[0029] To address the issue of excessive memory consumption caused by key-value cache (KV) in long-context inference tasks, some solutions propose permanently removing the KV cache corresponding to unimportant terms during autoregressive inference. Once removed, this key-value information will no longer be used in subsequent decoding operations. However, the inventors discovered that the importance of terms changes dynamically at different time steps during decoding: key-value information initially deemed unimportant may become crucial in subsequent decoding operations, and vice versa. Therefore, this solution of directly removing key-value information actually negatively impacts the model's inference quality.

[0030] To address the problems existing in current solutions, this application provides an inference task processing method. The basic idea is to propose a technical solution that transfers key-value information and supports on-demand retrieval, in order to ensure the inference efficiency of the model by saving memory space in the inference device and to ensure the inference quality of the model by supporting on-demand retrieval of key-value information, thereby effectively improving the model performance.

[0031] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0032] Figure 1 The following is a flowchart illustrating a reasoning task processing method provided as an exemplary embodiment of this application. Figure 2 This is a logical schematic diagram of an inference task processing method provided for an exemplary embodiment of this application. The method can be executed by an inference task processing device, which can be implemented as software, hardware, or a combination of both. The inference task processing device can be integrated into a computing device, which can be the inference device where the model to be optimized resides, or it can be another device independent of the inference device. This embodiment does not limit the deployment location of the inference task processing device.

[0033] refer to Figure 1The reasoning task processing method provided in this embodiment may include: Step 100: In response to receiving the inference task, use a pre-set general codebook to determine the corresponding codeword for the key-value information generated by the target model in the pre-filling stage for the inference task; and store the key-value information corresponding to the inference task in the auxiliary storage device. Step 101: During the decoding operation performed by the target model for the inference task, search for the target codeword related to this decoding operation in the general codebook; Step 102: Read the key-value information corresponding to the target codeword from the auxiliary storage device for the target model, as the reasoning basis for this decoding operation.

[0034] This embodiment describes the reasoning task processing method from the perspective of a reasoning task, demonstrating the performance improvement in processing reasoning tasks. It is understood that upon receiving a new reasoning task, the reasoning task processing method provided in this embodiment can be re-executed.

[0035] The target model in this embodiment can be any inference model to be optimized, such as the large language model introduced earlier. This embodiment does not limit the type of the target model; any model that needs to be decoded based on key-value information is acceptable. No further examples of target model types will be provided here.

[0036] In step 100, after receiving the inference task, the target model can begin performing autoregressive inference for the task. Referring to the preceding technical concept introduction, key-value information can be generated for the inference task during autoregressive inference. This embodiment proposes that the key-value information generated by the target model at pre-filled nodes for the inference task can be stored in an auxiliary storage device. That is, referring to... Figure 2 In this embodiment, the key-value information generated by the target model for the inference task is no longer stored in the inference device where the target model is located, but in the auxiliary storage device. This can effectively reduce the memory space occupied by the inference device and avoid affecting the inference efficiency of the decoding stage due to insufficient memory space of the inference device.

[0037] refer to Figure 2 In this embodiment, a general codebook is also pre-configured. This general codebook can be shared by multiple inference tasks. It is understood that there is no upper limit to the number of "multiple" tasks; in this embodiment's technical concept, the general codebook can support a massive number of inference tasks. Furthermore, the general codebook in this embodiment can be continuously expanded to support even more inference tasks.

[0038] The general codebook in this embodiment is a data structure used for information quantization; it is a collection containing multiple codewords. A codeword is an element in the general codebook, representing an index corresponding to a representative vector derived through extensive training and inductive summarization. A codeword can characterize the position or state of information in the quantization space. In this embodiment, the essence of a codeword is index information; therefore, the amount of information is much lower than the feature representation corresponding to the information.

[0039] Based on this universal codebook, in step 100 of this embodiment, the key-value information generated by the target model for the reasoning task can be quantified to determine the corresponding codeword in the universal codebook.

[0040] The quantization operation here can be understood as selecting appropriate codewords from a general codebook according to certain rules and measurement standards to replace the feature representations corresponding to the key-value information, thereby achieving the goal of representing key-value information with less information. Based on this, after storing the aforementioned key-value information to an auxiliary storage device, the inference device only needs to retain the codewords corresponding to the key-value information (i.e., the aforementioned index). This significantly reduces the memory space occupied in the inference device.

[0041] As can be seen, the pre-set universal codebook in this embodiment contains a large number of codewords, thus covering a wide variety of inference tasks. A vast number of inference tasks can find suitable codewords within the universal codebook. Therefore, in step 100, when various inference tasks are received, the quantization of key-value information can be directly completed based on the pre-set universal codebook in this embodiment, thereby more efficiently transferring key-value information. The construction process of the universal codebook does not consume the inference time of the inference task; therefore, it effectively saves the inference time required for the inference task.

[0042] Based on this, this embodiment proposes a key-value information recall mechanism.

[0043] refer to Figure 1 In step 101, during any decoding operation performed by the target model for the inference task, the target codeword related to the current decoding operation can be searched in the general codebook.

[0044] As described in the preceding technical concept introduction, the target model's autoregressive inference process for the inference task includes a decoding phase, which may contain multiple time steps, each corresponding to a decoding operation. For ease of description, step 101 explains the key-value information retrieval mechanism proposed in this embodiment from the perspective of a single decoding operation.

[0045] The general codebook provided in this embodiment is also used during the decoding process. It is understood that key-value information is an important basis for reasoning during decoding, and the distribution of the target model's attention to key-value information changes dynamically during different decoding operations.

[0046] Based on this, the key-value information retrieval mechanism proposed in this embodiment allows for the retrieval of key-value information on demand for decoding operations. That is, key-value information related to the decoding operation can be filtered out and retrieved, without the need for a full retrieval.

[0047] As mentioned earlier, in this embodiment, when the key-value information is transferred, the codeword corresponding to the key-value information is recorded. In this way, in step 101, the target codeword related to this decoding operation is found in the general codebook. That is, the target codeword can be used as a clue to find the key-value information related to this decoding operation.

[0048] Continue to refer to Figure 1 In step 102, it is proposed that the key value information corresponding to the target codeword can be read back from the auxiliary storage device for the target model as the reasoning basis for this decoding operation.

[0049] refer to Figure 2 The key-value information generated for the reasoning task is stored in an auxiliary storage device. In step 102, the required key-value information can be read back from the auxiliary storage device as needed during the decoding operation. As mentioned above, during the readback, the target codeword can be used as a clue to gradually find the storage location of the key-value information, thereby accurately reading back the required key-value information for this decoding operation.

[0050] This embodiment also proposes that the key-value information generated for the reasoning task mentioned in step 101 may include the key-value information corresponding to the words output by the decoding operation in the decoding stage. That is, in the decoding stage, after any decoding operation is completed, the key-value information generated for the newly output words will also be transferred to the auxiliary storage device.

[0051] Taking any preceding decoding operation as an example, based on the universality of the general codebook in this embodiment, after the preceding decoding operation is completed, the key-value information corresponding to the new word element can also be matched with a suitable codeword in the general codebook. Moreover, in this embodiment, the key-value information corresponding to the new word element can also be transferred to the auxiliary storage device. This means that the key-value information corresponding to the new word element generated during the decoding stage does not need to occupy memory space in the inference device, further saving memory space in the inference device. On this basis, during this decoding operation, the matched target codeword may contain a codeword corresponding to the key-value information of the new word element.

[0052] This ensures that key-value information related to the decoding operation can be more comprehensively filtered during the decoding process, avoiding the omission of key-value information corresponding to new words output in the decoding stage. This effectively improves the comprehensiveness of the reasoning basis used in the decoding operation, thereby improving the reasoning accuracy of the decoding operation.

[0053] In summary, this embodiment proposes a method for processing inference tasks. On one hand, the key-value information generated by the target model for the inference task can be transferred to an auxiliary storage device, thereby effectively reducing the memory usage on the inference device where the target model resides and avoiding problems such as excessive inference latency. On the other hand, a universal codebook that can be shared by multiple inference tasks is pre-configured. Based on this universal codebook, upon receiving an inference task, the universal codebook can be directly used to determine the corresponding codeword for the transferred key-value information; moreover, during any decoding operation performed for the inference task, the universal codebook can be used to find the target codeword related to decoding and read back the key-value information corresponding to the target codeword from the auxiliary storage device for the target model. Accordingly, in this embodiment, when different inference tasks are received, the universal codebook can be reused without the need to build a dedicated codebook for the inference task in real time, thereby effectively avoiding the time consumed by building a dedicated codebook and ensuring inference efficiency.

[0054] In the above or below embodiments, various quantization methods can be used to support the transfer of key-value information to auxiliary storage devices.

[0055] One alternative implementation proposes that any key-value information generated by the target model for the inference task can be vector-quantized to determine the codeword corresponding to the key-value information in a general codebook; wherein the general codebook contains codewords generated based on vector quantization technology.

[0056] In this optional implementation, vector quantization technology is used to quantize the key-value information to be transferred. Accordingly, the codewords in the general codebook of this embodiment are also generated based on vector quantization technology. Based on this, after vector quantization of the key-value information to be transferred, the codeword corresponding to the key-value information can be found in the general codebook of this embodiment.

[0057] The following is a brief introduction to vector quantization techniques to help explain the principle and logic of how vector quantization can be used to match key-value information with codewords in a general codebook in this optional implementation.

[0058] The core idea of ​​Vector Quantization (VQ) is to map a set of vectors in a high-dimensional space to a finite, predefined set of vectors (i.e., a codebook). The vectors in the codebook are then used to approximate the original vectors, thereby achieving data compression. For the key-value information in this embodiment, the corresponding feature vector is first calculated. Then, in the representative vectors corresponding to each codeword in the general codebook, a representative vector similar to the feature vector is searched. The index of the found representative vector in the codebook can then be used as the codeword corresponding to the key-value information.

[0059] In this optional implementation, an exemplary scheme for constructing a universal codebook is also provided. This exemplary scheme proposes: An achievable sample set, containing text and / or visual samples capable of covering multiple inference tasks, can be used as training samples; Using the target model, key-value information is generated for different training samples in the sample set. The key-value information generated for different training samples is converted into feature vectors, and the converted feature vectors are clustered to obtain multiple clusters. Indexes are constructed for representative vectors in multiple clusters, serving as codewords, to build a universal codebook.

[0060] As can be seen from the above description of vector quantization techniques, the representative vector mentioned here can be the central feature vector in the cluster.

[0061] In this exemplary solution, training samples can be text, images, and videos, etc. The data format of the training samples is not limited here and can be selected as needed. It is understood that inference tasks typically involve text, images, and videos to be processed. In this exemplary solution, multiple training inference tasks can be constructed, or multiple real historical inference tasks can be obtained, and the text, images, and videos to be processed carried in these inference tasks can be included in the sample set. In this embodiment, the modality of the training samples in the sample set is not limited, as long as it matches the modality supported by the target model.

[0062] In this exemplary scheme, key-value information is generated for different training samples in the sample set by leveraging the key-value information generation capability of the target model during the pre-filling stage. This generated key-value information serves as the data foundation for constructing a universal codebook, effectively ensuring the adaptability between the constructed codebook and the target model. Especially during inference, when determining the corresponding codewords for the key-value information generated by the target model, the accuracy of the determined codewords is effectively guaranteed.

[0063] In this exemplary scheme, the sample set used can contain a rich variety of training samples, thus covering a wide range of inference tasks, rather than being limited to the finite input samples corresponding to a single inference task. No upper limit is placed on the size of the sample set here, which allows for a clearer understanding of the universality of the general codebook, which is completely different from the dedicated codebooks mentioned earlier that are based on inference tasks. Moreover, due to the larger size of the sample set, the clusters formed in the above clustering process will be more refined and reasonable, and consequently, the representative vectors obtained based on these clusters will be more comprehensive and refined.

[0064] Based on this, when transferring key-value information, the pre-defined universal codebook ensures that the key-value information can be mapped to more accurate codewords; during the decoding operation, the universal codebook ensures that the relevant target codewords can be found more accurately for the decoding operation. With accuracy guaranteed in both aspects, the comprehensiveness and accuracy of the key-value information read back for the decoding operation can be effectively improved, thereby enhancing the inference accuracy of the decoding operation.

[0065] Figure 3 This is a schematic diagram of a model reasoning process based on a universal codebook, provided as an exemplary embodiment of this application.

[0066] refer to Figure 3 In this embodiment, it is also proposed that the constructed general codebook can be stored in the inference device where the target model is located. Of course, it can also be stored in other storage locations outside the inference device. In this embodiment, the storage location of the general codebook is not limited.

[0067] In summary, this embodiment proposes an optional implementation method for quantizing key-value information and an exemplary scheme for pre-setting a general codebook. By using vector quantization technology to quantize key-value information and using vector quantization technology to construct a general codebook based on a massive sample set, the versatility of the general codebook can be effectively improved, as well as the quantization accuracy of key-value information can be effectively improved, thereby improving the inference accuracy of decoding operations.

[0068] Of course, other quantization methods can also be used to quantize the key-value information to be transferred in this embodiment, such as product quantization (PQ). Correspondingly, other types of quantization techniques can also be used to support the construction of the general codebook in this embodiment, such as product quantization. In this embodiment, the quantization technique for the key-value information can be consistent with the quantization technique used in the pre-set general codebook. No limitation is made on the quantization technique here, and no further examples are given.

[0069] In the above or following embodiments, various implementation methods can be used to find the relevant target codeword for this decoding operation. One optional implementation method is provided below.

[0070] refer to Figure 3 The following is proposed in this optional implementation: The query vector constructed by the target model for this decoding operation can be determined; Calculate the correlation between the query vector and the representative vectors corresponding to each codeword in the general codebook; Based on the calculated correlation, the target codeword related to this decoding operation is searched in the general codebook.

[0071] In the decoding phase, the query vector is automatically generated by the target model itself based on the current decoding state and requirements, without requiring manual input from the user.

[0072] In this optional implementation, the query vector constructed by the target model for this decoding operation is used as the basis for searching for the target codeword in the general codebook. The query vector can fully reflect the reasoning direction of this decoding operation. Therefore, using the query vector as the basis for searching for the target codeword can effectively improve the accuracy and comprehensiveness of the found target codeword.

[0073] Based on this, after the target model automatically generates a query vector for this decoding operation according to the information already processed and the current generation progress and other internal states, the inference task processing device in this embodiment can obtain the query vector and find the target codeword in the general codebook based on the query vector. By reading back the key value information corresponding to the target codeword, it can help the target model decide what the next new word to be generated is.

[0074] In this optional implementation, it is proposed that multiple schemes can be used to calculate the correlation between the query vector and the representative vectors corresponding to each codeword in the general codebook. One exemplary scheme proposes: The inner product between the query vector and the representative vectors corresponding to each codeword in the general codebook can be calculated to characterize the correlation.

[0075] The inner product between vectors reflects the degree of similarity between them. The specific calculation logic of the inner product will not be elaborated here.

[0076] Of course, in this optional implementation, besides using the inner product to characterize the correlation, other methods can also be used. For example, the Euclidean distance between the query vector and the representative vectors corresponding to each codeword in the general codebook can be calculated to characterize the correlation between them. Here, the method of calculating the correlation is not limited, nor are further examples provided.

[0077] Furthermore, the selection of target codewords based on relevance supports flexible design of preset standards. Under one exemplary preset standard, the target codeword selection scheme can be: The K codewords in the general codebook that have the highest correlation with the query vector are identified as the target codewords; where K is a positive integer and can be flexibly adjusted as needed.

[0078] In this selection scheme, an upper limit value K is used to define the aforementioned preset standard. The value of K can be flexibly adjusted according to different inference accuracy requirements. For example, for models with high inference accuracy requirements, the value of K can be increased to read back more key-value information in the decoding stage, thereby improving the comprehensiveness of the read key-value information; while for models with high inference efficiency requirements, the value of K can be decreased to read back less key-value information in the decoding node, but enough to ensure inference accuracy, thereby ensuring inference efficiency in the decoding stage.

[0079] Besides using a maximum quantity value to define the preset standard, other forms can also be used. For example, a percentage maximum value of m% can be used to define the preset standard, taking the top m% of codewords in the general codebook that have the highest correlation with the query vector as the target codewords. No further limitations are set here, nor will any more examples be provided.

[0080] In summary, this embodiment proposes an optional implementation for selecting target codewords from a general codebook. By using the query vector constructed by the target model for the decoding operation as the search basis, it can be ensured that the representative vector corresponding to the found target codeword is sufficiently correlated with the query vector. Furthermore, the key value information pointed to by the target codeword is sufficiently correlated with the representative vector corresponding to the target codeword. Therefore, it can be ensured that the key value information pointed to by the target codeword is sufficiently correlated with the query vector provided by the target model. In this way, appropriate and comprehensive key value information can be read back for the decoding operation, thereby fully ensuring the inference accuracy of the decoding operation.

[0081] Of course, in this embodiment, besides searching for relevant target codewords for the current decoding operation based on query vectors, other implementation methods can also be used to find relevant target codewords for the current decoding operation. For example, the context of the current decoding operation, such as the semantic and syntactic structure of the preceding text, can be used to help determine relevant target codewords. For instance, in text generation in natural language processing, key-value information that reflects the theme or context can be selected from the key-value information used in the previous decoding operation as the query basis; in a general codebook, codewords with sufficiently high relevance to these selected key-value information can be searched as target codewords. The implementation method for finding relevant target codewords for the current decoding operation is not limited here, nor are further examples provided.

[0082] In the above or below embodiments, various implementation methods can be used to ensure that the auxiliary storage device can accurately provide key value information corresponding to the target codeword.

[0083] refer to Figure 3 One optional implementation proposes that, for any key-value information transferred for the reasoning task, the corresponding word can be determined, along with its position information within the word sequence of the reasoning task. A correlation is then established between the word position information corresponding to the key-value information and the codeword determined for that key-value information. This allows for the establishment of a correlation between word position information and codewords for the reasoning task.

[0084] Referring to the preceding technical concept introduction, in the autoregressive inference process, the target model maintains a token sequence for the inference task. In this embodiment, during the pre-filling stage, this token sequence may contain tokens segmented by the target model from the original input data. During the decoding stage, the target model can continuously add new tokens output after the decoding operation to this token sequence. From the perspective of this decoding operation, this token sequence already contains tokens segmented from the original input data during the pre-filling stage, as well as new tokens output by the decoding operation prior to this decoding operation.

[0085] Furthermore, the target model also records the positional information of each lexical in the lexical sequence. Lexical positional information can be understood as describing the position of a lexical in the lexical sequence, for example, being the 5th position in the lexical sequence. Here, lexical positional information can be understood as a kind of identifier for lexical units.

[0086] In this optional implementation, the word position information maintained by the target model for each word corresponding to the inference model is retained, and the word position information is used as an intermediate medium to achieve accurate transmission of key-value information between the inference device where the target model is located and the auxiliary storage device used to store key-value information for the inference task.

[0087] In this optional implementation, the association between the word position information and codewords generated for the inference task can be stored in the local storage space of the inference device, so as to serve as one of the query bases during the decoding stage. Of course, the association can also be stored in other storage locations outside the inference device, and the association can be accessed from the corresponding storage location during the decoding stage. Here, there is no limitation on the storage location of the association.

[0088] Building on this, in this optional implementation, it is also proposed that key-value information be stored in an auxiliary storage device using word position information as an index for the reasoning task.

[0089] Thus, during this decoding operation, an exemplary readback scheme for key-value information could be as follows: after determining the target codeword for this decoding operation, the target word position information associated with the target codeword can be determined based on the "association relationship between word position information and codeword" corresponding to the inference task; based on the target word position information, a readback instruction is sent to the auxiliary storage device so that the auxiliary storage device can hit the key-value information required for this decoding operation according to the target word position information.

[0090] In this exemplary readback scheme, the readback instruction sent to the auxiliary storage device carries the target lexical position information corresponding to the target codeword. It can be understood that, from the perspective of different inference tasks, the inference task processing device in this embodiment will maintain the aforementioned "association relationship between lexical position information and codewords" for different inference tasks, and the auxiliary storage device will also establish indexes for the key-value information corresponding to different inference tasks. This effectively ensures the isolation of key-value information between different inference tasks and effectively avoids readback errors.

[0091] In summary, this embodiment provides an optional implementation method for the read-back operation during the decoding process. By using the word position information maintained for each word in the word sequence in the target model as an intermediate medium, the key-value information required for the decoding operation can be correctly transmitted between the inference device and the auxiliary storage device, thereby improving the accuracy and comprehensiveness of the read-back key-value information and thus improving the inference accuracy of the decoding operation.

[0092] Of course, in this embodiment, other implementation methods can also be used to ensure that the auxiliary storage device can accurately provide key-value information corresponding to the target codeword. For example, dedicated identification information can be configured for the key-value information, and in this embodiment, the identification information configured for the key-value information can be consistently followed among the inference task processing device, the inference device where the target model is located, and the auxiliary storage device. Further details will not be elaborated here, nor will more examples of implementation methods be provided.

[0093] It should be noted that some processes described in the above embodiments and accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel.

[0094] Figure 4This is a schematic diagram of a computing device provided for another exemplary embodiment of this application. The aforementioned inference task processing apparatus for executing the inference task processing method can be integrated into this computing device. This computing device can be the inference device where the target model resides; of course, it can also be other independent devices besides the inference device, which is not limited here. (Reference) Figure 4 The computing device may include: memory 40 and processor 41.

[0095] The memory 40 is used to store the computer program corresponding to the above-mentioned reasoning task processing device.

[0096] Processor 41, coupled to memory 40, is used to execute computer programs in memory 40 for: In response to receiving an inference task, a pre-set general codebook is used to determine the corresponding codeword for the key-value information generated by the target model in the pre-filling stage for the inference task; and the key-value information corresponding to the inference task is stored in an auxiliary storage device. During the process of the target model performing a decoding operation for the inference task, the target codeword related to the current decoding operation is searched in the general codebook; The key-value information corresponding to the target codeword is read back from the auxiliary storage device for the target model, and used as the reasoning basis for this decoding operation.

[0097] In an optional embodiment, when the processor 41 uses a preset general codebook to determine the corresponding codeword for the key-value information generated by the target model for the inference task, it can specifically be used for: For any key-value information generated by the target model for the inference task, vector quantization is performed to determine the codeword corresponding to the key-value information in the general codebook; The general codebook contains codewords generated based on vector quantization technology.

[0098] In an optional embodiment, when constructing the general codebook, the processor 41 may specifically be used to: Obtain a sample set containing training samples that can cover multiple inference tasks; Using the target model, key-value information is generated for different training samples in the sample set; The key-value information generated for different training samples is converted into feature vectors, and the converted feature vectors are clustered to obtain multiple clusters. Indexes are created for representative vectors in the multiple clusters, which are then used as codewords to construct the general codebook.

[0099] In an optional embodiment, when the processor 41 searches for the target codeword related to the current decoding operation in the general codebook, it may specifically be used to: Determine the query vector constructed by the target model for this decoding operation; Calculate the correlation between the query vector and the representative vectors corresponding to each codeword in the general codebook; Based on the calculated correlation, the target codeword related to this decoding operation is searched in the general codebook.

[0100] In an optional embodiment, when the processor 41 searches for a target codeword related to the current decoding operation in the general codebook based on the calculated relevance, it may specifically perform the following: The inner product between the query vector and the representative vector corresponding to each codeword in the general codebook is calculated to characterize the correlation.

[0101] In an optional embodiment, when the processor 41 determines a codeword in the general codebook that has a correlation that meets a preset standard as the target codeword, it may specifically be used to: The K codewords in the general codebook that have the highest correlation with the query vector are determined as the target codewords; Where K is a positive integer.

[0102] In an alternative embodiment, processor 41 may also be used for: For any key-value information generated for the reasoning task, determine the word position information of the corresponding word in the word sequence of the reasoning task; An association is established between the word position information corresponding to the key value information and the codeword determined for the key value information, so as to construct the association between word position information and codeword for the reasoning task.

[0103] In an alternative embodiment, processor 41 may also be used for: The association between the lexical position information and codewords established for the reasoning task is stored in the local storage space.

[0104] In an optional embodiment, when the processor 41 reads back the key-value information corresponding to the target codeword from the auxiliary storage device, it may specifically be used to: Based on the association between the word position information and codewords corresponding to the reasoning task, the target word position information associated with the target codeword is determined; Based on the target word location information, a read-back instruction is sent to the auxiliary storage device so that the auxiliary storage device can hit the key value information required for this decoding operation according to the target word location information. The auxiliary storage device stores key-value information for the reasoning task using word position information as an index.

[0105] Furthermore, such as Figure 4 As shown, the computing device also includes other components such as a communication component 42 and a power supply component 43. Figure 4 The diagram only shows some components and does not mean that the computing device includes only these components. Figure 4 The components shown.

[0106] It is worth noting that the technical details of the above embodiments of the computer device can be referred to the relevant descriptions in the foregoing method embodiments. To save space, they will not be repeated here, but this should not cause any loss to the scope of protection of this application.

[0107] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0108] The aforementioned communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.

[0109] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0110] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium may be volatile, non-volatile, or a combination thereof, and may be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, digital video disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium.

[0111] Accordingly, this application also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above method embodiments. It should be understood that each step or combination of steps in the above method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above method embodiments.

[0112] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0113] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for processing reasoning tasks, characterized in that, include: In response to receiving an inference task, the codewords are determined for the key-value information generated by the target model in the pre-filling stage for the inference task using a pre-set general codebook. And store the key-value information corresponding to the reasoning task in an auxiliary storage device; During the decoding operation performed by the target model for the inference task, the target codeword related to the current decoding operation is searched in the general codebook. The key-value information corresponding to the target codeword is read back from the auxiliary storage device for the target model, and used as the reasoning basis for this decoding operation.

2. The method according to claim 1, characterized in that, Using a pre-built universal codebook, determine the corresponding codewords for the key-value information generated by the target model for the inference task, including: For any key-value information generated by the target model for the inference task, vector quantization is performed to determine the codeword corresponding to the key-value information in the general codebook; The general codebook contains codewords generated based on vector quantization technology.

3. The method according to any one of claims 1 or 2, characterized in that, The process of constructing the general codebook includes: Obtain a sample set, which contains text samples and / or visual samples capable of covering multiple reasoning tasks, as training samples; Using the target model, key-value information is generated for different training samples in the sample set; The key-value information generated for different training samples is converted into feature vectors, and the converted feature vectors are clustered to obtain multiple clusters. Indexes are created for representative vectors in the multiple clusters, which are then used as codewords to construct the general codebook.

4. The method according to claim 1, characterized in that, Searching for the target codeword related to this decoding operation in the general codebook includes: Determine the query vector constructed by the target model for this decoding operation; Calculate the correlation between the query vector and the representative vectors corresponding to each codeword in the general codebook; Based on the calculated correlation, the target codeword related to this decoding operation is searched in the general codebook.

5. The method according to claim 4, characterized in that, Calculating the correlation between the query vector and the representative vectors corresponding to each codeword in the general codebook includes: The inner product between the query vector and the representative vector corresponding to each codeword in the general codebook is calculated to characterize the correlation.

6. The method according to any one of claims 4 or 5, characterized in that, Based on the calculated correlation, the target codeword related to this decoding operation is searched in the general codebook, including: The K codewords in the general codebook that have the highest correlation with the query vector are determined as the target codewords; Where K is a positive integer.

7. The method according to claim 1, characterized in that, Also includes: For any key-value information generated for the reasoning task, determine the word position information of the corresponding word in the word sequence of the reasoning task; An association is established between the word position information corresponding to the key value information and the codeword determined for the key value information, so as to construct the association between word position information and codeword for the reasoning task.

8. The method according to claim 7, characterized in that, Also includes: The association between the lexical position information and codewords established for the inference task is stored in the local storage space.

9. The method according to claim 7 or 8, characterized in that, Reading key-value information corresponding to the target codeword from the auxiliary storage device for the target model includes: Based on the association between the word position information and the codeword corresponding to the reasoning task, the target word position information associated with the target codeword is determined; Based on the target word location information, a read-back instruction is sent to the auxiliary storage device so that the auxiliary storage device can hit the key value information required for this decoding operation according to the target word location information. The auxiliary storage device stores key-value information for the reasoning task using word position information as an index.

10. A computing device, characterized in that, Includes memory, processor, and communication components; The memory is used to store one or more computer instructions; The processor is coupled to the memory and the communication component and is used to execute one or more computer instructions for performing the inference task processing method according to any one of claims 1-9.

11. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is executed by one or more processors, the one or more processors perform the reasoning task processing method according to any one of claims 1-9.

12. A computer program product, characterized in that, The invention includes a computer program that, when executed by one or more processors, causes the one or more processors to perform the reasoning task processing method according to any one of claims 1-9.

Images