Key-value data processing method and text processing method

Abstract

The application discloses a key-value data processing method and a text processing method, and relates to the technical field of large models and artificial intelligence. The method comprises the following steps: processing a plurality of word units in an input text sequence to obtain query vectors, key vectors and value vectors corresponding to the plurality of word units, and obtaining a plurality of original key-value data based on the key vectors and the value vectors corresponding to the plurality of word units; performing statistics on the plurality of original key-value data to obtain target statistical features, and determining a target quantization mode of the plurality of original key-value data from a plurality of quantization modes based on the target statistical features, wherein the quantization accuracy of different quantization modes is different; quantizing the plurality of original key-value data based on the target quantization mode to obtain a plurality of target key-value data, and storing the plurality of target key-value data in a key-value cache. The application solves the technical problem of how to reduce the error caused by quantization while reducing the storage resources occupied by the key-value cache.

Description

Technical Field

[0001] This application relates to large model technology and artificial intelligence, specifically to a method for processing key-value data and a method for processing text. Background Technology

[0002] Key-value caching plays a crucial role in improving the efficiency and performance of Large Language Models (LLMs). While it can increase the inference speed of large language models, it also consumes significant amounts of storage space. As the application scenarios for large language model inference become increasingly diverse, many inference tasks require longer input texts. Since the size of the key-value cache is positively correlated with the length of the input text, the memory usage of the key-value cache continues to increase, making it more challenging to deploy large language models on memory-constrained devices.

[0003] In related technologies, quantization is used to compress key-value caches and reduce the storage space occupied by key-value caches. However, the precision conversion during the quantization process can introduce errors, especially when there are outliers in the data distribution. The errors introduced by quantization will further increase and affect the performance of large language models.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a method for processing key-value data and a method for processing text, so as to at least solve the technical problem of how to reduce the storage resources occupied by key-value cache while reducing the error caused by quantization.

[0006] According to one aspect of the embodiments of this application, a method for processing key-value data is provided, comprising: processing multiple tokens in an input text sequence to obtain query vectors, key vectors, and value vectors corresponding to the multiple tokens, and obtaining multiple original key-value data based on the key vectors and value vectors corresponding to the multiple tokens; performing statistics on the multiple original key-value data to obtain target statistical features, and determining a target quantization method for the multiple original key-value data from multiple quantization methods based on the target statistical features, wherein different quantization methods have different quantization accuracies; quantizing the multiple original key-value data based on the target quantization method to obtain multiple target key-value data, and storing the multiple target key-value data in a key-value cache.

[0007] According to another aspect of the embodiments of this application, a text processing method is also provided, comprising: responding to an input command applied to an operation interface, displaying an input text sequence on the operation interface, wherein the input text sequence contains multiple tokens; processing the multiple tokens to obtain query vectors, key vectors, and value vectors corresponding to the multiple tokens, and obtaining multiple original key-value data based on the key vectors and value vectors corresponding to the multiple tokens; determining a target quantization method for the multiple original key-value data based on the target statistical features of the multiple original key-value data; quantizing the multiple original key-value data based on the target quantization method to obtain multiple target key-value data, and storing the multiple target key-value data in a key-value cache; performing data recovery on the multiple target key-value data to obtain multiple recovered key-value data; generating an output text sequence corresponding to the input text sequence based on the multiple recovered key-value data and the query vectors corresponding to the multiple tokens; and displaying the output text sequence on the operation interface.

[0008] According to another aspect of the embodiments of this application, a method for processing key-value data is also provided, comprising: obtaining an input text sequence by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the input text sequence, and the input text sequence contains multiple tokens; processing the multiple tokens in the input text sequence to obtain query vectors, key vectors, and value vectors corresponding to the multiple tokens, and obtaining multiple original key-value data based on the key vectors and value vectors corresponding to the multiple tokens; determining a target quantization method for the multiple original key-value data based on the target statistical features of the multiple original key-value data; quantizing the multiple original key-value data based on the target quantization method to obtain multiple target key-value data; and storing the multiple target key-value data in a key-value cache by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter includes the multiple target key-value data.

[0009] According to another aspect of the embodiments of this application, a key-value data processing system is also provided, comprising: a client for sending an input text sequence, wherein the input text sequence contains multiple tokens; a server connected to the client for processing the multiple tokens to obtain query vectors, key vectors, and value vectors corresponding to the multiple tokens, and obtaining multiple raw key-value data based on the key vectors and value vectors corresponding to the multiple tokens; determining a target quantization method for the multiple raw key-value data based on the target statistical features of the multiple raw key-value data; quantizing the multiple raw key-value data based on the target quantization method to obtain multiple target key-value data, and storing the multiple target key-value data in a key-value cache.

[0010] According to another aspect of the embodiments of this application, a computing device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0011] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor connected to the memory via a bus for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0012] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0013] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0014] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0015] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0016] In this embodiment, multiple tokens in the input text sequence are processed to obtain query vectors, key vectors, and value vectors corresponding to the tokens. Based on the key vectors and value vectors, multiple raw key-value data are obtained. Based on the target statistical features of the raw key-value data, a target quantization method is determined. The raw key-value data is then quantized according to the target quantization method to obtain multiple target key-value data, which are then stored in a key-value cache. It is noteworthy that the target statistical features of the raw key-value data allow for the pre-determined different precision requirements. The target quantization method is dynamically adjusted based on these precision requirements, preserving high-precision key-value data during quantization and reducing errors caused by precision conversion. Storing the quantized target key-value data in the key-value cache reduces storage resources occupied by the cache, achieving the technical effect of reducing quantization errors while simultaneously reducing storage resources occupied by the cache. This solves the technical problem of how to reduce quantization errors while reducing storage resources occupied by the cache.

[0017] It is worth noting that the general description above and the detailed description that follow are merely for illustrative purposes and do not constitute a limitation on this application. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a schematic diagram illustrating an application scenario of a key-value data processing method according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of a key-value data processing method according to an embodiment of this application;

[0021] Figure 3 This is a flowchart of an optional key-value data processing method according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of an optional dynamic range quantization according to an embodiment of this application;

[0023] Figure 5 This is a flowchart of a text processing method according to an embodiment of this application;

[0024] Figure 6 This is a flowchart of another key-value data processing method according to an embodiment of this application;

[0025] Figure 7 This is a schematic diagram of a key-value data processing system according to an embodiment of this application;

[0026] Figure 8 This is a structural block diagram of a computing device according to an embodiment of this application;

[0027] Figure 9 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] The technical solution provided in this application is mainly implemented using large-scale model technology. Here, "large-scale model" refers to a deep learning model with a massive number of parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of parameters. Large-scale models can also be called foundational models. They are pre-trained using large-scale unlabeled corpora to produce pre-trained models with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multimodal pre-training models.

[0031] It should be noted that, in practical applications, large models can be fine-tuned using a small number of samples to adapt them for different tasks. For example, large models can be widely used in Natural Language Processing (NLP), computer vision, and speech processing. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and image generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. Therefore, the main application scenarios for large models include, but are not limited to, digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design. In this embodiment, the example of continuing to cache key-value pairs during text processing of a large language model is used for explanation.

[0032] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0033] Subtensor: A subtensor is a partial tensor obtained by partitioning a tensor. A tensor can be viewed as a multidimensional array, and a subtensor is one or more smaller chunks of data extracted from this multidimensional array. Subtensors can be generated by selecting some dimensions of the tensor or by slicing it. In machine learning and deep learning, especially when dealing with large-scale data, using subtensors can process data more efficiently.

[0034] Key-Value Cache: Key and value vectors are crucial components of the self-attention mechanism in the Transformer architecture (a deep learning model architecture). The key-value cache temporarily stores computed key-value pairs for quick retrieval and reuse in subsequent processing. This caching strategy significantly reduces computational resource consumption and improves model inference speed.

[0035] Quantization: Quantization is an important model compression technique in machine learning, aiming to improve the versatility and deployment efficiency of models by reducing the storage requirements and computational resource consumption of model parameters. The core idea of ​​quantization is to convert high-precision floating-point parameters (such as 32-bit or 16-bit floating-point numbers) into low-precision integers (such as 8-bit integers), thereby significantly reducing the model's storage space and computational complexity.

[0036] According to an embodiment of this application, a method for processing key-value data is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0037] Considering the large number of model parameters in large models and the limited computing resources of mobile terminals, the method provided in this application embodiment can be applied to, for example, Figure 1 The application scenarios shown are not limited to these. In, for example... Figure 1 In the application scenario shown, the large model is deployed on server 10. Server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. These client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to access the large model, thereby implementing the method provided in this embodiment.

[0038] In this embodiment, the system consisting of a client device and a server can perform the following steps: the client device sends an input text sequence, wherein the input text sequence contains multiple tokens; the server processes the multiple tokens to obtain query vectors, key vectors, and value vectors corresponding to the multiple tokens, and obtains multiple raw key-value data based on the key vectors and value vectors corresponding to the multiple tokens; based on the target statistical features of the multiple raw key-value data, the target quantization method of the multiple raw key-value data is determined; the multiple raw key-value data is quantized based on the target quantization method to obtain multiple target key-value data, and the multiple target key-value data is stored in the key-value cache.

[0039] It should be noted that with the rapid development of high-performance computing units, the methods provided in this application embodiment can also be applied to model-in-machine systems in other application scenarios. In one optional embodiment, the model-in-machine system has multiple built-in models, and users can select one model to adjust as needed to obtain their own model. The high-performance computing unit built into the model-in-machine system can then directly call the adjusted model to execute the methods provided in this application embodiment. In another optional embodiment, the large model-in-machine system has a pre-trained model built-in, and the high-performance computing unit built into the model-in-machine system can then directly call that model to execute the methods provided in this application embodiment.

[0040] Furthermore, when users need to train their own models, they can upload their own datasets via the client. These datasets are then sent to the server, allowing the server to adjust the pre-trained model using the dataset to obtain the user's customized model, which can then be deployed to the production environment. To facilitate users' model adjustment needs, the server provides complete adjustment tools, development frameworks, and processes, supporting multiple adjustment strategies. This allows the adjusted model to better adapt to different application domains and achieve a high degree of customization.

[0041] Under the aforementioned operating environment, this application provides the following: Figure 2 The method for processing key-value data is shown. Figure 2 This is a flowchart of a key-value data processing method according to an embodiment of this application. Figure 2 As shown, the method may include the following steps:

[0042] Step S202: Process multiple tokens in the input text sequence to obtain query vectors, key vectors and value vectors corresponding to multiple tokens, and obtain multiple original key-value data based on the key vectors and value vectors corresponding to multiple tokens.

[0043] The input text sequence in the above steps is a data stream consisting of multiple consecutive characters or words. It can be a piece of text or a sentence, and it is the object processed by the large language model.

[0044] The tokens in the above steps are the basic units in the input text sequence. They can be words, characters, or subwords, and are the smallest units of information processed by the large language model.

[0045] The query vector in the above steps represents the information that the current word needs to obtain from other words. The degree of association between the current word and other words in the sequence can be evaluated by calculating the dot product of the query vector and the key vector.

[0046] The key vector in the above steps is used to characterize the features of the lexical units. These features include the lexical unit's position and function in the input text sequence, as well as its potential relationship with other lexical units. The key vector is used to compare with the query vector to determine the attention weight.

[0047] The value vector in the above steps is used to represent the detailed information and semantic content of the lexical. Once the attention weights are determined, the query vector is weighted and summed with the value vectors of other lexical terms to obtain context-related information.

[0048] The raw key-value data in the above steps is high-precision data generated from key vectors and value vectors without quantization.

[0049] In one optional embodiment, the input text sequence is encoded into a series of word embeddings, where each word corresponds to an embedding vector. Positional encoding is then added to the word embeddings to preserve the relative position information of the words within the sequence. The word embeddings with added positional encoding are then processed using a linear transformation method, converting the embedding vector of each word into a query vector, a key vector, and a value vector, respectively. Subsequently, the key vector and value vector are combined to obtain the original key-value data.

[0050] In another alternative embodiment, multiple terms in the input text sequence are converted into high-dimensional feature vectors. These high-dimensional feature vectors are then processed using an attention mechanism to obtain query vectors, key vectors, and value vectors. Finally, the key vectors and value vectors are combined to form the original key-value data.

[0051] Step S204: Statistical analysis is performed on multiple raw key-value data to obtain target statistical features. Based on the target statistical features, the target quantization method for multiple raw key-value data is determined from multiple quantization methods. The quantization precision of different quantization methods is different.

[0052] The target statistical features in the above steps refer to the mathematical features of the original key-value data, such as mean, variance, maximum value, and data distribution, which are used to determine the appropriate quantization method for the original key-value data.

[0053] The target quantization method in the above steps is a quantization strategy for the original key-value data determined based on the target statistical characteristics. It may include, but is not limited to, the selection of quantization precision and the selection of quantization method. The quantization precision may be 4 bits or 8 bits, but is not limited to these. The quantization method may be a linear quantization method or a nonlinear quantization method.

[0054] The quantization precision in the above steps is used to characterize the accuracy of the floating-point values ​​that the quantized data can represent. Quantization precision can be determined by the number of quantization bits; since more bits can express more different values, a higher number of quantization bits results in higher quantization precision. In an optional embodiment, the target statistical features of multiple original key-value data are then compared with a preset quantization threshold. If the target statistical features of the multiple original key-value data are less than or equal to the preset quantization threshold, since the data variation in the multiple original key-value data is small, a low-precision quantization method can be used to reduce storage requirements, and the target quantization method for the multiple original key-value data is determined to be a low-precision quantization method. If the target statistical features of the multiple original key-value data are greater than the preset quantization threshold, since the data variation in the multiple original key-value data is large, a high-precision quantization method can be used to adapt to higher data complexity and precision requirements, and the target quantization method for the multiple original key-value data is determined to be a high-precision quantization method.

[0055] Target statistical features can reflect the variation range of multiple raw key-value data. Based on the target statistical features, lower precision is used for raw key-value data with a small variation range, while higher precision is used for raw key-value data with a large variation range. Determining the quantization precision based on target statistical features, and then determining the quantization method, not only reduces the overall storage and computation requirements, but also allows for the selection of appropriate quantization methods for different data characteristics, effectively reducing quantization errors.

[0056] In another alternative embodiment, a quantization prediction model can be trained to learn the relationship between target statistical features and quantization bit depth, enabling it to predict the quantization bit depth based on the target statistical features of multiple raw key-value data. The quantization prediction model can be a decision tree model or a random forest model, but is not limited to these. Multiple target statistical features are calculated based on multiple raw key-value data. These target statistical features may include, but are not limited to, mean, variance, range, skewness, and kurtosis. The target statistical features are input into the quantization prediction model to obtain the quantization bit depth, thereby determining the target quantization method.

[0057] Step S206: Quantize multiple original key-value data based on the target quantization method to obtain multiple target key-value data, and store the multiple target key-value data in the key-value cache.

[0058] The target key-value data in the above steps refers to the key-value data processed by applying the target quantization method. Compared with the original key-value data, the precision of the target key-value data has been reduced, but the content of the original data is preserved as much as possible in order to reduce quantization error.

[0059] The key-value cache in the above steps is used to store quantized key-value pairs to avoid repeated calculations in subsequent inference processes, thereby improving the inference speed and efficiency of the model. Here, key-value pairs are pairs of data consisting of key vectors and value vectors.

[0060] In one optional embodiment, quantization parameters can be determined based on the target quantization method. These parameters include quantization bit depth, quantization interval, and quantization step size. The quantization interval needs to cover the dynamic range of multiple original key-value data points, and the quantization step size can be determined based on the quantization bit depth. Then, the original key-value data is mapped to the integer precision corresponding to the quantization bit depth according to the quantization parameters to obtain the target key-value data. Finally, the obtained multiple target key-value data are stored in a key-value cache.

[0061] In this embodiment, multiple tokens in the input text sequence are processed to obtain query vectors, key vectors, and value vectors corresponding to the tokens. Based on the key vectors and value vectors, multiple raw key-value data are obtained. Based on the target statistical features of the raw key-value data, a target quantization method is determined. The raw key-value data is then quantized according to the target quantization method to obtain multiple target key-value data, which are then stored in a key-value cache. It is noteworthy that the target statistical features of the raw key-value data allow for the pre-determined different precision requirements. The target quantization method is dynamically adjusted based on these precision requirements, preserving high-precision key-value data during quantization and reducing errors caused by precision conversion. Storing the quantized target key-value data in the key-value cache reduces storage resources occupied by the cache, achieving the technical effect of reducing quantization errors while simultaneously reducing storage resources occupied by the cache. This solves the technical problem of how to reduce quantization errors while reducing storage resources occupied by the cache.

[0062] In the above embodiments of this application, statistical analysis of the plurality of original key-value data to obtain target statistical features includes: performing square root transformation on the plurality of original key-value data to obtain a plurality of transformed key-value data; and performing statistical analysis on the plurality of transformed key-value data to obtain target statistical features.

[0063] The square root transformation, also known as the square root operation, in the above steps is a mathematical transformation technique that can be used to smooth data distribution.

[0064] In one optional embodiment, a square root transformation, or square root operation, is performed on multiple original key-value data to obtain multiple transformed key-value data. By performing a square root transformation on multiple original key-value data, the influence of extreme values ​​is reduced, and the data distribution of the original key-value data is optimized, thereby achieving the goal of reducing quantization error.

[0065] Then, target statistical features of the transformed key-value data are calculated, including mean, variance, and range. These target statistical features provide data distribution information, which can be used to assess the dynamic range of the transformed key-value data. The dynamic range of the transformed key-value data can then be used to determine the accuracy requirements of the transformed key-value data.

[0066] In the above embodiments of this application, determining the target quantization method for multiple original key-value data from multiple quantization methods based on target statistical characteristics includes: determining the evaluation index corresponding to different quantization methods based on target statistical characteristics; and determining the quantization method as the target quantization method when the evaluation index corresponding to any quantization method meets the evaluation conditions corresponding to the quantization method.

[0067] The quantization methods mentioned above are strategies used in the quantization compression process. Different quantization methods correspond to different data transformation rules, and the choice of quantization method directly affects the accuracy and storage efficiency of the quantization results.

[0068] The evaluation metrics in the above steps are used to determine whether multiple raw key-value data are suitable for the current quantization method. Evaluation metrics may include, but are not limited to, the variance of multiple raw key-value data, the maximum value of multiple raw key-value data, the range of values ​​that the quantized data can cover, and the distribution density of the quantized data in a specific value range.

[0069] The evaluation conditions in the above steps are used to determine the target quantization method. They can be preset by the user or calculated based on the quantization accuracy of the target quantization method.

[0070] In one optional embodiment, different quantization methods can be determined. For example, different quantization methods can be divided into two types: low-precision quantization and high-precision quantization. Low-precision quantization uses int8 (8-bit integer) quantization, while high-precision quantization uses fp16 (16-bit floating-point) quantization. Different quantization methods can also be divided into three types: low-precision quantization, medium-precision quantization, and high-precision quantization. Low-precision quantization uses the lower 4 bits, medium-precision quantization uses the middle 4 bits, and high-precision quantization uses the higher 4 bits. However, the quantization method settings are not limited to these. The first example is used below for illustration.

[0071] When determining the quantization conditions for different quantization methods, for example, the quantization conditions can be: low-precision quantization is used when the maximum value of multiple original key-value data is within the value range that the quantized data can cover; high-precision quantization is used when the maximum value of multiple original key-value data is not within the value range that the quantized data can cover.

[0072] Then, based on the target statistical characteristics, the evaluation metrics corresponding to low-precision quantization and high-precision quantization are determined, namely, the maximum value of multiple original key-value data and the value range that the quantized data can cover. It should be noted that the evaluation metrics corresponding to different quantization methods can be the same or different. This example illustrates the case where the evaluation metrics corresponding to the quantization methods are the same. In practical applications, the evaluation metrics corresponding to different quantization methods can be different.

[0073] After determining the evaluation index, it is judged whether the evaluation index meets the evaluation conditions. When the maximum value of multiple original key-value data is within the value range that the quantized data can cover, the evaluation index is determined to meet the evaluation conditions of low-precision quantization, and the target quantization method is determined to be low-precision quantization. When the maximum value of multiple original key-value data is not within the value range that the quantized data can cover, the evaluation index is determined to meet the evaluation conditions of high-precision quantization, and the target quantization method is determined to be high-precision quantization.

[0074] In the above embodiments of this application, the target statistical features include: the variance of multiple original key-value data and the maximum value among the multiple original key-value data; based on the target statistical features, determining the evaluation indicators corresponding to different quantization methods includes: based on the maximum value, determining the data range of multiple original key-value data and using the data range as the evaluation indicator corresponding to the first quantization method; based on the variance and the maximum value, determining the evaluation indicator corresponding to the second quantization method, wherein the quantization precision of the second quantization method is greater than that of the first quantization method, and the evaluation indicator corresponding to the second quantization method includes the data range, representation density, and dispersion of multiple original key-value data; based on the variance, determining the evaluation indicator corresponding to the third quantization method, wherein the quantization precision of the third quantization method is greater than that of the second quantization method, and the evaluation indicator corresponding to the third quantization method includes the representation density and dispersion of multiple original key-value data.

[0075] The first quantization method in the above steps is a low-precision quantization method. For example, it can be a method that quantizes the lower 4 bits, which has the lowest quantization precision among the three quantization methods.

[0076] The second quantization method in the above steps is a medium-precision quantization method, for example, it can be a medium 4-bit quantization method, with quantization precision between the first quantization method and the third quantization method.

[0077] The third quantization method in the above steps is a high-precision quantization method. For example, it can be a method that quantizes the high 4 bits, and it has the highest quantization precision among the three quantization methods.

[0078] In one optional embodiment, for the first quantization method with low quantization accuracy, it is sufficient to ensure that the quantized data can cover the dynamic range of the original data. When the data range of multiple original key-value data is within the value range that the quantized data can cover, it indicates that the quantized data can cover the dynamic range of the original data. Therefore, the data range of multiple original key-value data can be determined based on the maximum value, and the data range can be used as the evaluation index corresponding to the first quantization method.

[0079] The second quantization method offers higher quantization accuracy compared to the first. For the second quantization method, while ensuring the quantized data covers the dynamic range of the original data, the concentration of data distribution must also be considered, i.e., the variance of the multiple original key-value pairs. If the variance is small, it means the distribution of the multiple original key-value pairs is relatively concentrated, and the current quantization bit depth is sufficient. If the variance is large, it means the distribution of the multiple original key-value pairs is relatively dispersed, requiring more quantization bits to ensure data details are preserved and quantization errors are reduced. Therefore, the evaluation metrics for the second quantization method can be determined based on the variance and maximum value. The evaluation metrics for the second quantization method include the data range, representation density, and dispersion of the multiple original key-value pairs. Data range, representation density, and dispersion collectively reflect the dynamic characteristics and distribution uniformity of the data. Data range ensures that quantization can cover all values, avoiding information loss; representation density and dispersion are used to determine the quantization details of the data, i.e., whether high precision is needed to capture subtle changes in the distribution.

[0080] The third quantization method retains more bits and has a stronger ability to represent numerical values. It requires focusing on the variance of the distribution details of multiple original key-value pairs. By analyzing the variance of these original key-value pairs, we can determine whether the data distribution is sufficiently complex and whether higher quantization precision is needed to preserve the data's detailed information. Therefore, evaluation metrics for the third quantization method can be determined based on variance. These evaluation metrics include the representation density and dispersion of the multiple original key-value pairs. The third quantization method is suitable for situations with a wide data range but well-defined distribution characteristics. By analyzing the data density and dispersion in greater detail, we can accurately determine which regions require higher-precision quantization, thereby further optimizing the use of storage and computing resources while maintaining data processing accuracy.

[0081] In the above embodiments of this application, the evaluation index corresponding to the second quantization method is determined based on variance and maximum value, including: obtaining the dispersion of multiple original key-value data based on variance; obtaining the representation density of multiple original key-value data based on variance and a first scaling factor; obtaining the representation range of multiple original key-value data based on maximum value; and obtaining the evaluation index corresponding to the second quantization method based on data region, representation density, and representation range.

[0082] The degree of dispersion in the above steps can be characterized by variance. Variance measures the dispersion of data; a larger variance indicates greater differences between data points and a more uneven distribution, while a smaller variance indicates more concentrated data points and a more uniform distribution. In addition to variance, dispersion can also be characterized by standard deviation, range, interquartile range, etc., but is not limited to these.

[0083] In the above steps, the first scaling factor, used in the second quantization method, is used to scale the variance of multiple original key-value data. The first scaling factor is a preset value that can be preset according to the actual application; no specific value corresponding to the first scaling factor is limited here.

[0084] The representation density in the above steps refers to the density of numerical points that can be represented by multiple original key-value data.

[0085] The range of values ​​represented in the above steps is the range of values ​​that the original key-value data can cover.

[0086] In one optional embodiment, the variance of multiple raw key-value data can be calculated to characterize the dispersion of the multiple raw key-value data. Then, the product of the variance and a first scaling factor is calculated to obtain the representation density of the multiple raw key-value data. Subsequently, the maximum value among the multiple raw key-value data is obtained, and the maximum value is used to characterize the representation range of the multiple raw key-value data. The data region, representation density, and representation range are used as evaluation metrics corresponding to the second quantization method.

[0087] In the above embodiments of this application, the evaluation index corresponding to the third quantization method is determined based on variance, including: obtaining the representation density of multiple original key-value data based on variance and the second scaling factor; obtaining the dispersion of multiple original key-value data based on variance; and obtaining the evaluation index corresponding to the third quantization method based on the dispersion and representation density.

[0088] In the above steps, the second scaling factor is used in the third quantization method to scale the variance of multiple original key-value data. The second scaling factor is a preset value that can be preset according to the actual application. Here, we do not impose any restrictions on the specific value of the second scaling factor.

[0089] In one optional embodiment, the representation density of multiple original key-value data can be obtained by multiplying the variance of multiple original key-value data by a second scaling factor, and the variance can be used to characterize the dispersion of the multiple original key-value data. Then, the dispersion and representation density are used as evaluation metrics corresponding to the third quantization method.

[0090] In the above embodiments of this application, statistical analysis is performed on multiple original key-value data to obtain target statistical features, including: dividing multiple original key-value data to obtain multiple sub-key-value data; performing statistics on any one sub-key-value data to obtain sub-statistical features of any one sub-key-value data; and summarizing the sub-statistical features of multiple sub-key-value data to obtain target statistical features.

[0091] The subkey data in the above steps is a portion of the data that is separated from the original key data.

[0092] The sub-statistical features of the sub-key value data in the above steps are the statistical features corresponding to the sub-key value data, which may include, but are not limited to, mean, variance, maximum value, minimum value, etc.

[0093] In one optional embodiment, multiple original key-value data can be divided to obtain multiple sub-key-value data. The division criteria can be time series, spatial location, or data characteristics, but are not limited to these. The sub-key-value data must follow a Laplace distribution with a mean of 0, meaning the absolute values ​​of the sub-key-value data must conform to an exponential distribution. The expression for the absolute values ​​of sub-key-value data conforming to an exponential distribution is as follows:

[0094] |Y|~Exponential(b -1 );

[0095] Where |Y| is the absolute value of the subkey value data, Exponential represents the exponential distribution, and b is the average value of the subkey value data.

[0096] Then, statistical analysis is performed on any subkey value data to calculate substatistical features such as variance and maximum value. These substatistical features can be used to assess the distribution range and density of the data and guide the selection of subquantization methods. The substatistical features of multiple subkey value data are then summarized to obtain the target statistical features.

[0097] In the above embodiments of this application, determining the target quantization method of multiple original key-value data from multiple quantization methods based on target statistical features includes: determining the sub-quantization method of any sub-key-value data from multiple quantization methods based on the sub-statistical features of any sub-key-value data; determining the initial mask corresponding to any sub-key-value data based on the sub-quantization method of any sub-key-value data, wherein the initial mask is used to characterize whether the sub-quantization method of any sub-key-value data is the corresponding quantization method; performing interpolation processing on the initial masks corresponding to multiple quantization methods to obtain the target mask corresponding to multiple original key-value data; and determining the target quantization method based on the target mask, wherein the spatial size of the target mask is consistent with the spatial size of multiple original key-value data.

[0098] The sub-quantization method in the above steps is the quantization method selected for sub-key value data based on sub-statistical characteristics.

[0099] The initial mask in the above steps is used to mark the sub-quantization method of the sub-key value data. Different sub-quantization methods correspond to different initial masks.

[0100] The target mask in the above steps is the mask obtained by interpolating the initial mask and adjusting the spatial size of the mask tensor.

[0101] The spatial dimensions in the above steps can characterize the geometric properties of tensor data, including the length, width, and depth of the tensor data.

[0102] In one optional embodiment, based on the sub-statistical characteristics of any subkey value data, the numerical range of the subkey value data is evaluated. If the numerical range of the subkey value data is relatively small, the sub-quantization method of the subkey value data is determined to be to convert high-precision data into low-precision data, thereby reducing computational costs. If the numerical range of the subkey value data is relatively large, the sub-quantization method of the subkey value data is determined to retain high precision, thereby reducing the precision loss caused by quantization.

[0103] A mapping relationship between sub-quantization methods and initial masks can be pre-defined. After determining the sub-quantization method for the sub-key value data, the initial mask corresponding to any sub-key value data is determined through the mapping relationship. The initial mask is used to mark the sub-quantization methods corresponding to different sub-key value data. Then, the initial mask can be interpolated using an interpolation function to adjust the spatial dimensions of the mask tensor, including its height and width, so that the spatial dimensions of the mask tensor are consistent with the spatial dimensions of the input tensor of the large language model, resulting in the target mask. The spatial dimensions of the target mask are consistent with the spatial dimensions of multiple original key value data. Subsequently, based on the target mask, the target quantization method for multiple original key value data is determined.

[0104] In the above embodiments of this application, the method further includes: performing data recovery on multiple target key-value data based on the target mask to obtain multiple recovered key-value data; and generating an output text sequence corresponding to the input text sequence based on the multiple recovered key-value data and the query vectors corresponding to multiple tokens.

[0105] The restored key-value data in the above steps is the original data corresponding to the target key-value data. When the large language model uses the target key-value data stored in the key-value cache, the target key-value data can be restored to obtain the original data that can be directly used by the large language model.

[0106] The output text sequence in the above steps is the output text sequence obtained by the large language model after processing the input text sequence.

[0107] In one optional embodiment, when storing target key-value data in a key-value cache, the target mask corresponding to the target key-value data is also stored in the key-value cache. When the large language model needs to use the target key-value data stored in the key-value cache, the multiple target key-value data are restored to their values ​​before quantization using the target mask, resulting in multiple restored key-value data. These multiple restored key-value data are then input into the large language model, enabling the large language model to generate an output text sequence based on the multiple restored key-value data and the query vectors corresponding to the multiple tokens.

[0108] The following description uses an optional embodiment, with the first, second, and third quantization methods corresponding to the low 4-bit quantization, middle 4-bit quantization, and high 4-bit quantization methods, respectively. The quantization bit depth of each quantization method has the following relationship:

[0109] hp = hc + lp + lc;

[0110] hp≥0;

[0111] lp≥0;

[0112] lc≥0;

[0113] Where hp is the number of high-precision bits, hc is the number of bits truncated from the high-precision bits, lp is the number of low-precision bits, and lc is the number of bits truncated from the low-precision bits.

[0114] Because the quantization precision of the first, second, and third quantization methods is different, the specific values ​​of hp, hc, lp, and lc are different for each quantization method. Therefore, the representation range and representation density are also different for each quantization method.

[0115] The formulas for calculating the representation range corresponding to different quantization methods are as follows:

[0116]

[0117] Where RR represents the representation range corresponding to different quantization methods, hp represents the number of bits for high precision corresponding to the quantization method, hc represents the number of bits clipped from the high-order bits corresponding to the quantization method, and Δ is the scaling factor.

[0118] The formulas for calculating the representation density corresponding to different quantization methods are as follows:

[0119] RC=2 lc ·Δ;

[0120] Where RC represents the representation density corresponding to different quantization methods, lc represents the number of bits clipped from the low-order bits corresponding to the quantization method, and Δ is the scaling factor.

[0121] The first quantification method uses the maximum value as its evaluation metric, and the evaluation conditions for the first quantification method are as follows:

[0122] max(Y)≤RR1;

[0123] Where max(Y) is the maximum value of multiple original key-value data, and RR1 is the upper limit of the representation range of the first quantization method.

[0124] That is, the maximum value of multiple raw key-value data is compared with the representation range of the first quantization method. If the maximum value of multiple raw key-value data is less than the upper limit of the representation range of the first quantization method, it is determined that the maximum value of multiple raw key-value data is within the representation range of the first quantization method, and the quantization method of multiple raw key-value data is determined to be the first quantization method.

[0125] The second quantification method uses maximum value and variance as evaluation indicators, and the evaluation conditions for the second quantification method are as follows:

[0126] max(Y) > RR2;

[0127]

[0128] Where max(Y) is the maximum value of multiple original key-value data, var(Y) is the variance of multiple original key-value data, RR2 is the upper limit of the representation range of the second quantization method, RC2 is the representation density corresponding to the second quantization method, and δ is a pre-set constant.

[0129] That is, the maximum value of multiple original key-value data is compared with the upper limit of the representation range of the second quantization method, and the ratio of the variance of multiple original key-value data to the representation density corresponding to the second quantization method is compared with a pre-set constant δ. If the above evaluation conditions are met, the quantization method of multiple original key-value data is determined to be the second quantization method.

[0130] The third quantitative method uses variance as its evaluation metric, and the evaluation criteria for the third quantitative method are as follows:

[0131]

[0132] Where var(Y) is the variance of multiple original key-value data, RC3 is the representation density corresponding to the third quantization method, and δ is a pre-set constant.

[0133] That is, the ratio of the variance of multiple original key-value data to the representation density corresponding to the third quantization method is compared with a pre-set constant δ. If the above evaluation conditions are met, the quantization method of multiple original key-value data is determined to be the third quantization method.

[0134] Figure 3 This is a flowchart of an optional key-value data processing method according to an embodiment of this application. An optional key-value data processing method is as follows: Figure 3 As shown:

[0135] Step S302: Obtain the original key-value data from the key-value cache.

[0136] Step S304: Perform square root transformation.

[0137] Step S306: Perform dynamic range quantization.

[0138] Step S308: Obtain the target key data and the target mask.

[0139] Step S310: Perform inverse quantization.

[0140] Perform dequantization on the target key-value data based on the target mask.

[0141] Step S312: Obtain the recovered data.

[0142] The recovered data is based on the original data before quantization corresponding to the target key-value data.

[0143] Step S314: Reuse the recovered data.

[0144] That is, the recovered data will continue to be used for data processing of large language models.

[0145] Figure 4 This is a schematic diagram of an optional dynamic range quantization according to an embodiment of this application, such as... Figure 4 As shown, the variance, mean, representation range, and representation density of the original key-value data are calculated. Then, based on these statistical characteristics, a suitable target quantization method is determined from the low 4-bit quantization method, the middle 4-bit quantization method, and the high 4-bit quantization method. The original key-value data is quantized based on the target quantization method to obtain the target key-value data and the target mask.

[0146] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0147] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0149] According to embodiments of this application, a text processing method is also provided. Figure 5 This is a flowchart of a text processing method according to an embodiment of this application, such as... Figure 5 As shown, the method includes:

[0150] Step S502: In response to the input command applied to the operation interface, the input text sequence is displayed on the operation interface, wherein the input text sequence contains multiple words;

[0151] The input commands in the above steps are applied to the user interface and can be used to input text sequences.

[0152] Step S504: Process multiple tokens to obtain query vectors, key vectors and value vectors corresponding to multiple tokens, and obtain multiple original key-value data based on the key vectors and value vectors corresponding to multiple tokens;

[0153] Step S506: Based on the target statistical features of multiple raw key-value data, determine the target quantization method of multiple raw key-value data; quantize multiple raw key-value data based on the target quantization method to obtain multiple target key-value data, and store the multiple target key-value data in the key-value cache;

[0154] Step S508: Perform data recovery on multiple target key-value data to obtain multiple recovered key-value data;

[0155] Step S510: Based on multiple recovery key-value data and query vectors corresponding to multiple tokens, generate the output text sequence corresponding to the input text sequence;

[0156] Step S512: Display the output text sequence on the operation interface.

[0157] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0158] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0159] According to embodiments of this application, a method for processing key-value data is also provided. Figure 6 This is a flowchart of another key-value data processing method according to an embodiment of this application, such as... Figure 6 As shown, the method includes:

[0160] Step S602: Obtain the input text sequence by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the input text sequence, and the input text sequence contains multiple words;

[0161] The first interface in the above steps is an interface used to obtain the input text sequence, and the parameter of the first interface can be the input text sequence.

[0162] Step S604: Process multiple tokens in the input text sequence to obtain query vectors, key vectors and value vectors corresponding to multiple tokens, and obtain multiple original key-value data based on the key vectors and value vectors corresponding to multiple tokens;

[0163] Step S606: Based on the target statistical characteristics of multiple raw key-value data, determine the target quantization method of multiple raw key-value data;

[0164] Step S608: Quantize multiple raw key-value data based on the target quantization method to obtain multiple target key-value data;

[0165] Step S610: Store multiple target key-value data into a key-value cache by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes multiple target key-value data.

[0166] The second interface in the above steps is an interface used to store multiple target key-value data. The parameters of the second interface can be multiple target key-value data.

[0167] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0169] According to embodiments of this application, a key-value data processing system is also provided. Figure 7 This is a schematic diagram of a key-value data processing system according to an embodiment of this application, such as... Figure 7 As shown, the system includes:

[0170] Client 70 is used to send an input text sequence, wherein the input text sequence contains multiple tokens;

[0171] Server 72 connects to the client and is used to process multiple tokens to obtain query vectors, key vectors, and value vectors corresponding to the multiple tokens. Based on the key vectors and value vectors corresponding to the multiple tokens, multiple raw key-value data are obtained. Based on the target statistical features of the multiple raw key-value data, the target quantization method of the multiple raw key-value data is determined. Based on the target quantization method, the multiple raw key-value data are quantized to obtain multiple target key-value data, and the multiple target key-value data are stored in the key-value cache.

[0172] It should be noted that the client 70 and server 72 mentioned above correspond to steps S602 to S610 in the above embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also be part of the device and run in the server 10 provided in the above embodiments.

[0173] It should be noted that the preferred embodiments involved in the above embodiments of this application are the same as the solutions, application scenarios and implementation processes provided in the above embodiments, but are not limited to the solutions provided in the above embodiments.

[0174] Embodiments of this application may provide a computing device. Figure 8 This is a structural block diagram of a computing device according to an embodiment of this application. Figure 8 As shown, the computing device 800 may include: one or more (only one is shown in the figure) processors 802, memory 804, memory controller, and peripheral interfaces.

[0175] The aforementioned computing device can be understood as an integrated smart terminal, including but not limited to servers, desktop computers, PCs (Personal Computers), all-in-one model machines, etc., and the computing device may have the model described in the above embodiments of this application pre-installed.

[0176] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other model types), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API calling capabilities, allowing models to be called into created applications through API interfaces, and providing application management tools to achieve application control.

[0177] Furthermore, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0178] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0179] The processor can invoke an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.

[0180] Embodiments of this application may provide an electronic device. Figure 9 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 9 As shown, the electronic device may include: an input / output device 92; a memory 94; and a processor 96, wherein the processor 96 is connected to the input / output device 92 and the memory 94 via a bus 98.

[0181] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0182] The processor can invoke an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.

[0183] It will be understood by those skilled in the art that the structure shown in the figure is merely illustrative, and the computing device may also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, or other terminal device. This figure does not limit the structure of the aforementioned computing device. For example, the computing device 100 may also include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or may have a different configuration than that shown in the figure.

[0184] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0185] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store program code executed by the method provided in the above embodiments.

[0186] Optionally, in this embodiment, the storage medium may be located in a computing device.

[0187] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program, which, when the executable program is running, controls the device where the computer-readable storage medium is located to execute the method described in any of the above embodiments.

[0188] Embodiments of this application also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.

[0189] Embodiments of this application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which can be used to store a computer program that, when executed by a processor, implements the method provided in the above embodiments.

[0190] Embodiments of this application also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0191] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0193] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0194] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0195] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0196] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for processing key-value data, characterized in that, include: Multiple tokens in the input text sequence are processed to obtain query vectors, key vectors, and value vectors corresponding to the multiple tokens. Based on the key vectors and value vectors corresponding to the multiple tokens, multiple original key-value data are obtained. Statistical analysis is performed on the multiple original key-value data to obtain target statistical features. Based on the target statistical features, the target quantization method for the multiple original key-value data is determined from multiple quantization methods, wherein the quantization precision of different quantization methods is different. The multiple original key-value data are quantized based on the target quantization method to obtain multiple target key-value data, and the multiple target key-value data are stored in the key-value cache.

2. The method according to claim 1, characterized in that, The step of statistically analyzing the multiple original key-value data to obtain target statistical features includes: Perform a square root transformation on the multiple original key-value data to obtain multiple transformed key-value data; The target statistical features are obtained by statistically analyzing the multiple transformed key-value data.

3. The method according to claim 1 or 2, characterized in that, The step of determining the target quantization method for the multiple original key-value data from multiple quantization methods based on the target statistical features includes: Based on the aforementioned target statistical characteristics, evaluation indicators corresponding to different quantification methods are determined; If any evaluation index of a quantification method satisfies the evaluation conditions corresponding to that quantification method, then the quantification method is determined to be the target quantification method.

4. The method according to claim 3, characterized in that, The target statistical features include: the variance of the plurality of original key-value data, and the maximum value among the plurality of original key-value data; the determination of evaluation indicators corresponding to different quantification methods based on the target statistical features includes: Based on the maximum value, the data range of the plurality of original key-value data is determined, and the data range is used as the evaluation index corresponding to the first quantification method; Based on the variance and the maximum value, an evaluation index corresponding to the second quantization method is determined, wherein the quantization accuracy of the second quantization method is greater than that of the first quantization method, and the evaluation index corresponding to the second quantization method includes the data range, representation density, and dispersion of the plurality of original key-value data. Based on the variance, an evaluation index corresponding to the third quantization method is determined, wherein the quantization accuracy of the third quantization method is greater than that of the second quantization method, and the evaluation index corresponding to the third quantization method includes the representation density and dispersion of the multiple original key-value data.

5. The method according to claim 4, characterized in that, The step of determining the evaluation index corresponding to the second quantification method based on the variance and the maximum value includes: Based on the variance, the degree of dispersion of the multiple original key-value data is obtained; Based on the variance and the first scaling factor, the representation density of the plurality of original key-value data is obtained; Based on the maximum value, the representation range of the plurality of original key-value data is obtained; Based on the data region, the representation density, and the representation range, the evaluation index corresponding to the second quantization method is obtained.

6. The method according to claim 4, characterized in that, The step of determining the evaluation index corresponding to the third quantification method based on the variance includes: Based on the variance and the second scaling factor, the representation density of the plurality of original key-value data is obtained; Based on the variance, the degree of dispersion of the multiple original key-value data is obtained; Based on the degree of dispersion and the representation density, the evaluation index corresponding to the third quantization method is obtained.

7. The method according to claim 1 or 2, characterized in that, The step of statistically analyzing the multiple original key-value data to obtain target statistical features includes: The original key-value data is divided to obtain multiple sub-key-value data; By performing statistics on any subkey value, we can obtain the sub-statistical features of that subkey value. The target statistical feature is obtained by summarizing the sub-statistical features of the multiple sub-key value data.

8. The method according to claim 7, characterized in that, The step of determining the target quantization method for the multiple original key-value data from multiple quantization methods based on the target statistical features includes: Based on the sub-statistical features of any one of the sub-key value data, determine the sub-quantization method of any one of the multiple quantization methods; Based on the subquantization method of any one of the subkey values, determine the initial mask corresponding to the any one of the subkey values, wherein the initial mask is used to characterize the subquantization method of any one of the subkey values; Interpolation processing is performed on the initial masks corresponding to the multiple quantization methods to obtain the target masks corresponding to the multiple original key-value data, wherein the spatial size of the target mask is consistent with the spatial size of the multiple original key-value data; Based on the target mask, the target quantization method is determined.

9. The method according to claim 8, characterized in that, The method further includes: Based on the target mask, data recovery is performed on the multiple target key-value data to obtain multiple recovered key-value data; Based on the multiple recovery key-value data and the query vectors corresponding to the multiple tokens, an output text sequence corresponding to the input text sequence is generated.

10. A text processing method, characterized in that, include: In response to an input command applied to the user interface, an input text sequence is displayed on the user interface, wherein the input text sequence contains multiple word elements; The multiple lexical units are processed to obtain the query vector, key vector, and value vector corresponding to the multiple lexical units, and multiple original key-value data are obtained based on the key vector and value vector corresponding to the multiple lexical units; Based on the target statistical characteristics of the multiple raw key-value data, the target quantization method of the multiple raw key-value data is determined; The multiple original key-value data are quantized based on the target quantization method to obtain multiple target key-value data, and the multiple target key-value data are stored in the key-value cache; Data recovery is performed on the multiple target key-value data to obtain multiple recovered key-value data; Based on the multiple recovery key-value data and the query vectors corresponding to the multiple lexical units, an output text sequence corresponding to the input text sequence is generated; The output text sequence is displayed on the user interface.

11. A method for processing key-value data, characterized in that, include: The input text sequence is obtained by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the input text sequence, and the input text sequence contains multiple words; Multiple tokens in the input text sequence are processed to obtain query vectors, key vectors, and value vectors corresponding to the multiple tokens. Based on the key vectors and value vectors corresponding to the multiple tokens, multiple original key-value data are obtained. Based on the target statistical characteristics of the multiple raw key-value data, the target quantization method of the multiple raw key-value data is determined; Based on the target quantization method, the multiple original key-value data are quantized to obtain multiple target key-value data; The multiple target key-value data are stored in a key-value cache by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the multiple target key-value data.

12. A key-value data processing system, characterized in that, include: A client is used to send an input text sequence, wherein the input text sequence contains multiple tokens; The server, connected to the client, processes the multiple lexical units to obtain query vectors, key vectors, and value vectors corresponding to the multiple lexical units, and obtains multiple raw key-value data based on the key vectors and value vectors corresponding to the multiple lexical units; determines the target quantization method of the multiple raw key-value data based on the target statistical features of the multiple raw key-value data; quantizes the multiple raw key-value data based on the target quantization method to obtain multiple target key-value data, and stores the multiple target key-value data in the key-value cache.

13. A computing device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 11.

14. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor, connected to the memory via a bus, is used to run the program, wherein the program, when running, executes the method according to any one of claims 1 to 11.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 11.

16. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 11.

Images