Information processing method, electronic device and computer readable storage medium

Abstract

The present disclosure relates to the technical field of large model technologies and data processing. Disclosed are an information processing method, an electronic device and a computer readable storage medium. The method comprises: in a forward inference process of a target machine learning model, acquiring parameter information corresponding to a query request, wherein the parameter information is obtained by encoding the query request; performing quantization compression on the parameter information according to a preset quantization mode, to obtain quantization compression information; using the quantization compression information to perform quantization compression processing on the parameter information, to obtain cache information; and performing inference on the basis of the cache information to obtain a query result corresponding to the query request. The present disclosure solves the technical problem in the related art of low model precision caused by offline compression key value cache, complex quantitative calibration steps and the influence of calibration data on the model precision.

Description

Information processing method, electronic device, and computer-readable storage medium

[0001] Cross-reference

[0002] This disclosure claims priority to the Chinese patent application filed with the China Patent Office on September 19, 2023, with application number 2023112160246 and invention name “Information processing method, electronic device and computer-readable storage medium”, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0003] The present disclosure relates to the fields of large model technology and data processing technology, and in particular to an information processing method, an electronic device, and a computer-readable storage medium. Background Art

[0004] As the amount of computing data increases, researchers have built artificial intelligence (AI) models with extremely large parameter counts, also known as large-scale language models (LLMs), by increasing the scale and complexity of the models. For example, the Chat Generative Pre-trained Transformer (ChatGPT), thereby improving the model's performance in language understanding, generation, and related tasks. In the inference process of large models, video memory is one of the bottlenecks. The key-value cache (K / V Cache) generated during the inference process will occupy a large amount of video memory, and some large models require multiple graphics processing units (GPUs) during the inference process. In addition, in the decoding calculation of large models, the bandwidth from GPU global memory to shared memory / registers is also a bottleneck. Every time a new token is generated, it needs to be read from the existing K / V Cache. Therefore, compressing the K / V Cache can reduce the memory usage during large model inference. It can also reduce the total input / output (IO) from GPU global memory to shared memory / registers during decoding calculations, thereby improving decoding efficiency.

[0005] Currently, the TRT-LLM model only supports offline quantization of the K / V cache. This quantization method requires pre-calibration using calibration data. Model accuracy is affected by this calibration data, and calibration data is unavailable in some scenarios. Furthermore, offline compression of the K / V cache in the TRT-LLM model results in lower model accuracy, rendering the proposed memory savings meaningless.

[0006] To address the above-mentioned problems, no effective solutions have been proposed so far.

[0007] Summary of the Invention

[0008] The embodiments of the present disclosure provide an information processing method, an electronic device, and a computer-readable storage medium to at least solve the technical problems in related technologies such as offline compressed key-value caching, complex quantization and calibration steps, and the influence of calibration data on model accuracy, resulting in low model accuracy.

[0009] According to one aspect of an embodiment of the present disclosure, there is provided an information processing method, including: in a forward reasoning process of a target machine learning model, obtaining parameter information corresponding to a query request, wherein the parameter information is obtained by encoding the query request; quantizing and compressing the parameter information according to a preset quantization method to obtain quantized compressed information, wherein the quantized compressed information is used to represent a ratio of quantized compression of the parameter information; performing quantized compression processing on the parameter information using the quantized compressed information to obtain cache information, wherein the cache information is used to represent cached information obtained after the parameter information is quantized and compressed; and obtaining a query result corresponding to the query request based on reasoning on the cache information.

[0010] According to another aspect of an embodiment of the present disclosure, a cloud server is also provided, including: a first cache, used to store parameter information corresponding to a query request, wherein the parameter information is obtained by encoding the query request; a processor, used to read the parameter information from the first cache during the forward reasoning process of the target machine learning model, quantize and compress the parameter information according to a preset quantization method to obtain quantized compressed information, and use the quantized compressed information to quantize and compress the parameter information to obtain cache information, wherein the quantized compressed information is used to represent the ratio of quantized compression of the parameter information, and the cache information is used to infer the query result corresponding to the query request; a second cache, used to store the cache information.

[0011] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes any one of the above-mentioned information processing methods when running.

[0012] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements any of the above methods when executed by a processor.

[0013] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0014] According to another aspect of an embodiment of the present disclosure, a computer program is further provided, which implements any of the above methods when executed by a processor.

[0015] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is further provided, which includes a stored executable program, wherein when the executable program runs, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned information processing methods.

[0016] In the disclosed embodiment, during the forward reasoning process of the target machine learning model, the query request is first encoded to obtain parameter information corresponding to the query request, and then the parameter information is quantized and compressed according to a preset quantization method to obtain quantized compressed information, and then the quantized compressed information is used to perform quantized compression processing on the parameter information to obtain cache information, and finally, the query result corresponding to the query request is obtained based on the cache information reasoning, thereby achieving the purpose of dynamically quantizing parameter information (i.e., key-value information) and compressing cache information of the parameter information (i.e., key-value cache) without the need for quantization calibration, thereby achieving the technical effect of effectively compressing the key-value cache while meeting the model accuracy, reducing the video memory occupancy during the inference process, and reducing the total input / output of the graphics processor global memory to the shared memory / register in the decoding calculation, thereby solving the technical problems in the related technology of offline compression of key-value cache, complex quantization calibration steps, and the model accuracy being affected by the calibration data, resulting in low model accuracy.

[0017] It is easy to note that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present disclosure, and do not constitute a limitation of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the disclosure of this application. In the drawings:

[0019] FIG1 is a schematic diagram of an application scenario of an information processing method according to an embodiment of the present disclosure;

[0020] FIG2 is a flow chart of an information processing method according to an embodiment of the present disclosure;

[0021] FIG3 is a flowchart of another information processing method according to an embodiment of the present disclosure;

[0022] FIG4 is a flowchart of another information processing method according to an embodiment of the present disclosure;

[0023] FIG5 is a schematic diagram of a cloud server according to an embodiment of the present disclosure;

[0024] FIG6 is a flowchart of an information processing method according to an embodiment of the present disclosure;

[0025] FIG7 is a structural diagram of an information processing method and apparatus according to an embodiment of the present disclosure;

[0026] FIG8 is a structural diagram of another information processing method and apparatus according to an embodiment of the present disclosure;

[0027] FIG9 is a structural block diagram of a computer terminal according to an embodiment of the present disclosure;

[0028] FIG10 is a structural block diagram of a processor according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present disclosure.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] The technical solution provided by the present disclosure is mainly implemented using large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. The large model can also be called a cornerstone model / foundation model. It is pre-trained by large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as LLM models and multi-modal pre-training models.

[0032] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned with a small number of samples, so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), and image generation. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0033] First, some nouns or terms that appear in the description of the embodiments of the present disclosure are subject to the following explanations:

[0034] Large-scale language model: A machine learning model with a large number of parameters and complex structure, which is an extended form of the neural network model.

[0035] K / V Cache: A key-value cache used to store and quickly access key-value pairs. K / V Cache is an efficient caching mechanism that can be used to accelerate data access and reduce data storage pressure.

[0036] Quantization: In neural networks, quantization is the process of converting floating-point parameters or activation values ​​into integer or fixed-point representations with lower bits. The purpose of quantization is to reduce the memory usage and computational complexity of the neural network model, thereby improving the model's efficiency and inference speed.

[0037] Dequantization: In neural networks, dequantization is the process of converting quantized integer or fixed-point representations back to floating-point representations. The goal of dequantization is to convert quantized integer or fixed-point representations back to floating-point representations for normal neural network calculations and inference, preserving the model's precision and accuracy.

[0038] Symmetric quantization divides the range of floating-point values ​​into a fixed number of equal intervals, each represented by an integer. In symmetric quantization, the value 0 is typically represented by a special integer, and the range of negative and positive values ​​is the same, thus reducing storage requirements.

[0039] Asymmetric Quantization: Dynamically adjusts the range of model weights so that the range of integer representations better matches the weight distribution. Asymmetric quantization typically introduces a scaling factor and a zero offset to map floating-point numbers to integer representations and performs the reverse mapping during the calculation process. By dynamically adjusting the scaling factor and zero offset, asymmetric quantization can better preserve the accuracy of weights and reduce storage requirements.

[0040] Per-token quantization: A technique used to reduce the storage and computational costs of natural language processing (NLP) models. In NLP tasks, models typically consist of many different parameters that require storage and computation, resulting in large model sizes and computational complexity. Per-token quantization aims to reduce the number of parameters per word or token.

[0041] Per-head quantization is a technique used in NLP tasks. Models typically use a multi-head self-attention mechanism to process input sequences. Each attention head has its own parameters that require storage and computation. Per-head quantization aims to reduce the number of parameters per attention head.

[0042] Forward inference refers to the process of passing input data through the model's computational pipeline, from the input layer to the output layer, to generate the model's predictions. In neural networks, forward inference involves passing input samples through the network's forward propagation process, computing and passing them layer by layer, ultimately obtaining the model's output. During forward inference, input data undergoes a series of linear transformations, activation functions, pooling operations, and other processes to gradually extract and transform features, ultimately generating the model's predictions. Forward inference is typically used after model training is complete to make predictions or inferences on new input data.

[0043] Perplexity (PPL) is a performance metric used to evaluate large models. It measures the level of confusion or uncertainty in predicting the next word. A smaller PPL is better.

[0044] K4 / K8: A quantization method that quantizes the K Cache in the multi-head attention block to 4 bits / 8 bits.

[0045] V4 / V8: A quantization method that quantizes the V Cache in the multi-head attention block to 4 bits / 8 bits.

[0046] Cache: Cache is designed to reduce data transmission delays between the processor and main memory. It is used to store recently used data so that it can be quickly accessed when needed.

[0047] Buffer: Buffer is designed to solve the problem of mismatched data transmission speeds. It is used to temporarily store data so that data can be sent and received at a more balanced rate.

[0048] Related technologies for offline quantization of K / V Cache based on the TRT-LLM model have the following defects.

[0049] Flaw 1: Offline quantization of the K / V cache in TRT-LLM requires pre-calibration data. The final model accuracy is affected by the choice of calibration data. Furthermore, calibration data is unavailable for some application scenarios. Offline calibration increases the complexity of LM inference deployment and is not user-friendly.

[0050] Defect 2: When TRT-LLM offline quantization compresses the key / v cache to INT4, the accuracy of the LLM model is significantly affected. For example, for the llama-13B model, when the key / v cache is compressed to INT4, the PPL index measured on the Wiki-text dataset is higher than the PPL index of the llama-7B model without key / v cache compression. This can be understood as the accuracy of a larger model optimized for key / v cache being lower than that of a smaller model of the same series. This makes the use of key / v cache compression to save video memory and reduce I / O bandwidth requirements from global memory to shared memory / registers meaningless.

[0051] With respect to the above-mentioned defects, no effective solution has been proposed before the present disclosure.

[0052] According to an embodiment of the present disclosure, an information processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0053] Considering the huge amount of model parameters of the large model and the limited computing resources of the mobile terminal, the above-mentioned information processing method provided by the embodiment of the present disclosure can be applied to the application scenario shown in Figure 1, but is not limited thereto. In the application scenario shown in Figure 1, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client devices 20 here can include but are not limited to: smart phones, tablet computers, laptops, PDAs, personal computers, smart home devices, car-mounted devices, etc. The client device 20 can interact with the user through a graphical user interface to implement the call to the large model, thereby implementing the method provided by the embodiment of the present disclosure.

[0054] Optionally, as shown in FIG1 , the large model can also be deployed in a cloud server 10 ′, and the cloud server 10 ′ communicates with the client device 20 through a communication protocol, for example, Hypertext Transfer Protocol / Hypertext Transfer Protocol Secure (HTTP / HTTPS), Transmission Control Protocol / Internet Protocol (TCP / IP), or Remote Procedure Call (RPC), etc., which are not limited here.

[0055] In an embodiment of the present disclosure, a system composed of a client device and a server can perform the following steps: the client device executes steps such as obtaining a query request input by a user in a graphical user interface and sending the query request to the server; the server executes steps such as outputting a query result based on the obtained query request and returning the query result to the client device. It should be noted that, when the operating resources of the client device can meet the deployment and operation conditions of the large model, the embodiment of the present disclosure can be carried out in the client device.

[0056] Optionally, the server may independently perform steps such as determining corresponding query results based on the obtained query request, thereby training a large model, etc., which is not limited here.

[0057] In the above operating environment, the present disclosure provides an information processing method as shown in Figure 2. Figure 2 is a flow chart of an information processing method according to an embodiment of the present disclosure. As shown in Figure 2, the method may include the following steps:

[0058] Step S21, during the forward reasoning process of the target machine learning model, obtaining parameter information corresponding to the query request, wherein the parameter information is obtained by encoding the query request;

[0059] Step S22, quantizing and compressing the parameter information according to a preset quantization method to obtain quantized compressed information;

[0060] Step S23, performing quantization compression processing on the parameter information using the quantization compression information to obtain cache information;

[0061] Step S24: obtaining a query result corresponding to the query request based on inference of the cache information.

[0062] The target machine learning model can be a large model pre-trained offline. For example, the pre-trained large model can use a basic neural network as the target machine learning model, including an input layer, an embedding layer, an encoding layer, an attention layer, a convolutional layer, a decoding layer, a linear layer, and a loss layer. Of course, the above-mentioned target machine learning model can also use an existing large model, such as GPT, GPT-3, BERT, Transformer-XL, etc., which is not limited here.

[0063] A query request can be understood as a "query" (Q) in the target machine learning model, or a query vector. Q is part of the input to the target machine learning model and describes the user's question or need. Q can be a sentence or a paragraph of text. The target machine learning model aims to generate an appropriate answer based on the query Q.

[0064] Parameter information can be understood as information about parameters associated with a query request, such as key-value information.

[0065] The quantization compression information is used to represent the information required for quantization compression of the parameter information. Exemplarily, the quantization compression information may include a scaling ratio (scale), which is used to adjust or control the size, capacity, or complexity of the model. In the embodiment of the present disclosure, the scale is used to represent the ratio of quantization compression of the parameter information.

[0066] Cache information can be understood as cache information obtained by quantizing and compressing parameter information, for example, key-value cache information obtained by quantizing and compressing key-value information, including key cache information (K Cache) and value cache information (V Cache).

[0067] In the disclosed embodiments, during the forward inference process of the target machine learning model, parameter information corresponding to the query request is obtained by encoding the obtained query request. After obtaining the parameter information, the parameter information is quantized and compressed according to a preset quantization method to obtain quantized compressed information. This can be understood as obtaining quantized compressed information in real time based on the parameter information corresponding to the query request, thereby achieving a dynamic scale determination effect.

[0068] After obtaining the quantized compressed information, the parameter information can be quantized and compressed based on the real-time quantized compressed information, achieving dynamic compression of the parameter information and achieving the effect of caching the information. By quantizing and compressing the parameter information, the data precision is reduced, thereby reducing video memory usage and computational complexity. In addition, by dynamically quantizing the parameter information, the model can be calibrated without the use of calibration data, avoiding the problem of model accuracy being affected by improper selection of calibration data, and there is no need to consider the acquisition of calibration data.

[0069] After obtaining the cache information, the query result corresponding to the query request can be inferred based on the cache information, thereby effectively compressing the cache information, effectively reducing its video memory occupancy, and reducing the total IO required from GPU global memory to shared memory / registers in decoding calculations, while also ensuring the accuracy of the model.

[0070] In the embodiment of the present disclosure, by using a pre-trained target machine learning model, parameter information corresponding to the query request is obtained during the forward reasoning process of the target machine learning model, so as to execute the information processing method provided by the embodiment of the present disclosure based on the query request and its corresponding parameter information. As a result, the parameters of the pre-trained target machine learning model can be tuned based on the information processing method provided by the embodiment of the present disclosure, so that the optimized target machine learning model can achieve the technical effect of effectively compressing the cache while meeting the model accuracy, reducing the video memory occupancy during the reasoning process, and reducing the total input / output of the graphics processor global memory to the shared memory / register in the decoding calculation.

[0071] The above-mentioned information processing method provided by the embodiments of the present disclosure can be applied to, but is not limited to, application scenarios involving information processing in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services and navigation services, such as: query requests for e-commerce services, query requests for academic interpretations, query requests for medical means, etc., which are not limited here.

[0072] By adopting the embodiment of the present disclosure, in the forward reasoning process of the target machine learning model, the query request is first encoded to obtain the parameter information corresponding to the query request, and then the parameter information is quantized and compressed according to a preset quantization method to obtain quantized compressed information, and then the quantized compressed information is used to perform quantized compression processing on the parameter information to obtain cache information, and finally, the query result corresponding to the query request is obtained based on the reasoning of the cache information. In this way, the purpose of dynamically quantizing the parameter information (i.e., key-value information) and compressing the cache information of the parameter information (i.e., key-value cache) is achieved without the need for quantization calibration, thereby achieving the technical effect of effectively compressing the key-value cache while meeting the model accuracy, reducing the video memory occupancy during the inference process, and reducing the total input / output of the graphics processor global memory to the shared memory / register in the decoding calculation, thereby solving the technical problems in the related art of offline compression of the key-value cache, complex quantization calibration steps, and the model accuracy being affected by the calibration data, resulting in low model accuracy.

[0073] In an optional embodiment, the parameter information includes key value information. In step S21, obtaining the parameter information corresponding to the query request includes the following method steps:

[0074] Step S211: Read key-value information corresponding to the query request from the first cache.

[0075] Key-value information can be understood as a key vector (key, K) and a value vector (value, V) obtained through a linear mapping, where K is used to retrieve or find the index of specific information, and V is the information or data associated with the key. Usually, keys and values ​​are operated in the form of key-value pairs (K / V).

[0076] Exemplarily, the precision of the key-value information includes but is not limited to FP16, BF16, FP32 and the like.

[0077] The first cache can be understood as a cache area used to store key-value information during the forward reasoning process. For example, the key-value information can be stored in the weight vector at each position in the attention weights matrix of the cache area, which is not limited here.

[0078] In the embodiment of the present disclosure, obtaining parameter information corresponding to the query request can be understood as reading key-value information corresponding to the query request from the first cache, so that the query result corresponding to the query request can be determined based on the key-value information.

[0079] In an optional embodiment, the information processing method further includes the following method steps:

[0080] Step S201: Encode the query request to obtain an encoding result;

[0081] Step S202, performing linear mapping on the encoding result to obtain key value information;

[0082] Step S203: store the key-value information in the first cache.

[0083] During the forward reasoning process of the target machine learning model, the input query request is encoded to obtain an encoding result, and then the encoding result is linearly mapped (Linear Mapping) to obtain the key-value information corresponding to the query request, and then the key-value information can be written into the first cache for storage.

[0084] For example, an encoder in the target machine learning model can convert the input query request text sequence into a fixed-length vector representation. The encoder can use a recurrent neural network (RNN) or transformer structure to gradually process the words or characters of the input text, extract the semantic and grammatical features of the text layer by layer, and encode the text into a fixed-length vector representation. The encoded vector can contain the overall information of the input query request text, which can be used in the subsequent decoding process.

[0085] After obtaining the encoding result, a linear mapping is performed on the encoding result, mapping the encoding result from a low dimension to a high dimension, thereby obtaining key-value information. This helps capture more fine-grained semantic information in the input text and extract key features. For example, the encoding result can be mapped to a new vector space through matrix multiplication and bias addition. This new vector space can have a higher dimension and thus have richer feature representation capabilities.

[0086] After the key-value information is obtained, the key-value information is written into the first cache for storage, thereby facilitating subsequent reading of the key-value information corresponding to the query request.

[0087] In an optional embodiment, the preset quantization mode includes a symmetric quantization mode, and the quantization compression information includes scale information. In step S22, the parameter information is quantized and compressed according to the preset quantization mode to obtain the quantization compression information, including the following method steps:

[0088] Step S221 : In a symmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain scale information.

[0089] When the parameter information is quantized and compressed based on a preset quantization method, the preset quantization method may be a symmetric quantization method.

[0090] The number of quantization bits is used to represent the number of binary bits of the quantization level. In the embodiment of the present disclosure, the target number of bits to be quantized is used to represent the number of bits to which the key-value information is quantized and compressed, denoted as N. The target number of bits to be quantized can be determined based on actual conditions. For example, setting the target number of bits to be quantized to 8 bits means that the key-value information is quantized and compressed to 8 bits, which can represent 2 to the power of 8 different quantization levels. By setting the target number of bits to be quantized, the storage requirements and computational overhead of the model can be reduced to a certain extent, thereby improving the efficiency and inference speed of the model.

[0091] In the symmetric quantization method, quantization compression can be performed based on the key value information and the target number of bits to be quantized, thereby determining scale information, that is, determining the compression ratio scale of the key value information.

[0092] For example, the data type after quantization and compression of the key-value information includes, but is not limited to, INT8, INT4, UINT8, UINT4, and other data types. Obtaining the quantization and compression information includes, but is not limited to, obtaining the maximum value (Max) or percentile (Percentile). During the quantization and compression process, data rounding methods include, but are not limited to, rounding up, rounding down, and rounding off, which are not limited here.

[0093] In an optional embodiment, the key-value information includes: a key to be quantized and a value to be quantized. In step S23, the parameter information is quantized and compressed using the quantization compression information to obtain cache information, including the following method steps:

[0094] In step S231 , according to the preset quantization compression granularity, the scale information is used in a symmetric quantization method to perform quantization compression processing on the key to be quantized to obtain key cache information, and the scale information is used to perform quantization compression processing on the value to be quantized to obtain value cache information.

[0095] The cache information includes key cache information (K Cache) and value cache information (V Cache). K Cache can be understood as the cache information obtained after quantizing and compressing K. Correspondingly, V Cache can be understood as the cache information obtained after quantizing and compressing V.

[0096] The expression of symmetric quantization is X q =[X / Δ], where X represents the data before quantization, X q represents the quantized data, Δ represents the scale information, which can be obtained by Δ=max(|X|) / (2 N-1 -1) OK.

[0097] Specifically, the key to be quantized and the value to be quantized can be understood as the above X, which is used to represent K and V before quantization compression. The key cache information and the value cache information can be understood as the above X q , used to represent KCache and V Cache obtained after quantization compression.

[0098] The preset quantization compression granularity includes but is not limited to per-token, per-head, and other granularities.

[0099] In the symmetric quantization mode, the key K and the value V to be quantized may be quantized and compressed according to the predetermined scale information Δ at a preset quantization compression granularity, thereby obtaining key cache information K Cache and value cache information V Cache .

[0100] In an optional embodiment, the preset quantization method includes an asymmetric quantization method, and the quantized compressed information includes scale information and zero point position information. In step S22, the parameter information is quantized and compressed according to the preset quantization method to obtain the quantized compressed information, including the following method steps:

[0101] Step S222 : In an asymmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain scale information and zero point position information.

[0102] When the parameter information is quantized and compressed based on a preset quantization method, the preset quantization method may also be an asymmetric quantization method.

[0103] In the asymmetric quantization method, in addition to determining the scale information Δ, it is also necessary to determine the zero-point position information.

[0104] In the asymmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized, and the scale information Δ and the zero point position information can be determined.

[0105] For example, the data type after quantization and compression of the key-value information includes, but is not limited to, INT8, INT4, UINT8, UINT4, and other data types. Obtaining the quantization and compression information includes, but is not limited to, obtaining the maximum value (Max) or percentile (Percentile). During the quantization and compression process, data rounding methods include, but are not limited to, rounding up, rounding down, and rounding off, which are not limited here.

[0106] In an optional embodiment, the key-value information includes: a key to be quantized and a value to be quantized. In step S23, the parameter information is quantized and compressed using the quantization compression information to obtain cache information, including the following method steps:

[0107] In step S232, according to the preset quantization compression granularity, the scale information and the zero point position information are used in an asymmetric quantization method to perform quantization compression processing on the key to be quantized to obtain key cache information, and the scale information and the zero point position information are used to perform quantization compression processing on the value to be quantized to obtain value cache information.

[0108] The expression of asymmetric quantization is X q =[X / Δ]+Z, where X represents the data before quantization, X q represents the quantized data, Δ represents the scale information, which can be obtained by Δ={max(X)-min(X)} / (2 N -1) is determined, Z represents the zero point position information, which can be determined by Z=-2 N-1 -[min(X) / Δ] is determined.

[0109] Specifically, the key to be quantized and the value to be quantized can be understood as the above X, which is used to represent K and V before quantization compression. The key cache information and the value cache information can be understood as the above X q , used to represent KCache and V Cache obtained after quantization compression.

[0110] The preset quantization compression granularity includes but is not limited to per-token, per-head, and other granularities.

[0111] In the asymmetric quantization method, the key K to be quantized and the value V to be quantized can be quantized and compressed separately according to the preset quantization compression granularity based on the determined scale information Δ and the zero point position information Z, thereby obtaining the key cache information K Cache and the value cache information V Cache respectively.

[0112] In an optional embodiment, the information processing method further includes the following method steps:

[0113] Step S25: writing the cache information into the second cache.

[0114] The second cache can be understood as a reserved buffer. In the disclosed embodiment, the reserved buffer is used as a data buffer to temporarily store data, and the key cache information and value cache information obtained in real time are written into the reserved buffer for storage, thereby reducing the number of data transmissions and improving the efficiency of data transmission.

[0115] In an optional embodiment, in step S24, obtaining a query result corresponding to the query request based on the cache information inference includes the following method steps:

[0116] Step S241: Read cache information from the second cache, where the cache information includes: key cache information and value cache information;

[0117] Step S242: performing dequantization processing on the key cache information to obtain first dequantization information, and performing dequantization processing on the value cache information to obtain second dequantization information;

[0118] Step S243 : performing a multiplication operation based on the query request, the first dequantization information, and the second dequantization information to obtain a query result.

[0119] During the large model computation process, each step generates a corresponding token, following the computational sequence. Each token is used to generate the Q, K, and V vectors for the next step. Previously generated tokens are fed into the model as contextual information, allowing new tokens to be determined based on previously generated tokens. This generation approach maintains contextual consistency, enabling the generation of expected token sequences based on generation strategies and conditions.

[0120] In the disclosed embodiment, when reading cache information from the second cache, it is necessary to obtain the cache information corresponding to the generated token, namely, the key cache information and value cache information corresponding to the generated token. The key cache information and value cache information corresponding to the generated token can be understood as the K Cache and V Cache corresponding to all previous tokens, including the current token.

[0121] The first dequantization information can be understood as the dequantization information obtained by dequantizing the K Cache stored in the second cache, that is, the dequantization information obtained by dequantizing the K Cache corresponding to the current token and the K Cache corresponding to all previous tokens. Correspondingly, the second dequantization information can be understood as the dequantization information obtained by dequantizing the V Cache stored in the second cache, that is, the dequantization information obtained by dequantizing the V Cache corresponding to the current token and the V Cache corresponding to all previous tokens.

[0122] In the disclosed embodiment, when a query result corresponding to a query request is obtained based on inference of cache information, K Cache and V Cache can be read from the second cache. Dequantization operations are performed on K Cache and V Cache, respectively, to restore K Cache and V Cache to their original precision, thereby obtaining first dequantized information obtained by dequantizing K Cache and second dequantized information obtained by dequantizing V Cache. The query result corresponding to the query request is obtained based on the obtained query request, the first dequantized information, and the second dequantized information.

[0123] Specifically, when dequantizing the key cache information, different situations will occur depending on the target number of bits to be quantized. When the target number of bits to be quantized is greater than 1, with the Q vector being [5.8, 6.3, 7.4], the K Cache will include three values ​​after quantization: [1, -4, 4] T , taking scale as 16.3 as an example, the dequantization logic during forward reasoning is: first multiply K Cache by the multiplied scale, and then get [16.3, -65.2, 65.2] T , and then multiply the result by Q, that is, (5.8×16.3-6.3×65.2+7.4×65.2)=166.26, thereby obtaining the first dequantization information.

[0124] It should be noted that when the key-value information is quantized and compressed, if the key-value information is quantized into ternary digits, then during the forward reasoning process, there is no need to dequantize the K Cache and V Cache obtained after the quantization compression. At this time, the K Cache and V Cache obtained are the first dequantized information and the second dequantized information obtained after dequantization.

[0125] For example, when the target number of bits to be quantized is equal to 1, that is, when the key value information is quantized into a ternary value, there is no need to perform inverse quantization. If the Q vector is [5.8, 6.3, 7.4], K includes three values ​​after quantization [0, -1, 1] T , taking the scale as 65.2 as an example. Since K has only three possible values ​​after quantization: 0, -1, and 1, the forward reasoning calculation logic here is: first add Q and K, that is, (0-6.3+7.4)=1.1, and then multiply the result by scale, that is, 1.1×65.2=71.72, to obtain the key cache information, which is the first dequantization information. Accordingly, the process of obtaining the second dequantization information can be referred to the description of obtaining the first dequantization information, and will not be repeated here.

[0126] In an embodiment of the present disclosure, during the decoding process of the target machine learning model, a multiplication operation can be performed based on the query request, the first inverse quantization information, and the second inverse quantization information to obtain a query result corresponding to the current query request.

[0127] For example, the decoder in the target machine learning model takes the encoded vector representation as input and generates words or characters layer by layer, producing an output text sequence. The decoder can use a structure such as an RNN or Transformer, learning generation rules and a language model to gradually generate output text that conforms to language rules and context.

[0128] It is understood that during the target machine learning model calculation process, after obtaining the query result corresponding to the query request calculated in the previous step, the obtained query result will be used as the query request for the next step of calculation, thereby determining the query result of the next step based on the query result calculated in the previous step. For example, if the initial query request is "Who are you?", the target large model determines the first query result as "I" based on the initial query request. Then, using the first query result "I" as the query request calculated in the previous step, it determines the second query result as "yes", and so on, until the complete query result corresponding to the initial query request is obtained.

[0129] In an optional embodiment, in step S243, obtaining a query result based on the query request, the first dequantization information, and the second dequantization information includes the following method steps:

[0130] Step S2431, performing a multiplication operation on the query request and the first dequantization information to obtain a weight coefficient;

[0131] Step S2432: multiply the second inverse quantization information by the weight coefficient to obtain a query result.

[0132] When obtaining a query result based on a query request, the first inverse quantization information, and the second inverse quantization information, the query request is first multiplied by the first inverse quantization information to obtain a weight coefficient, and then the second inverse quantization information is multiplied by the obtained weight coefficient to obtain the query result.

[0133] As shown in Figure 3, Figure 3 is a flow chart of another information processing method according to an embodiment of the present disclosure. First, based on the output token of the previous step, step S301 is entered to determine the Q, K, and V after the linear mapping of the input for this calculation. Then, based on the K and V after the linear mapping and the selected quantization method, step S302 is entered to determine the quantization compression information (scale and other information). Then step S303 is entered to perform quantization compression processing on the K and V after the linear mapping based on the scale and other information, and step S304 is entered to store the processed key cache information K Cache and value cache information V Cache in the reserved cache buffer (second cache). Then step S305 is entered to perform dequantization processing on the K Cache corresponding to all tokens in the reserved buffer to obtain the first dequantization information.

[0134] Then, the linearly mapped Q is multiplied by the first inverse quantization information. Exemplarily, the linearly mapped Q and the first inverse quantization information are calculated based on a scaled dot-product attention mechanism. Specifically, step S306 is entered to perform a dot multiplication operation on the linearly mapped Q and the first inverse quantization information to obtain a first operation result. Then step S307 is entered to divide the first operation result by the scaling ratio scale to obtain a second operation result. By entering steps S308 and S309, the second operation result is masked and normalized using the softmax function to finally obtain the attention weight, that is, the weight coefficient.

[0135] Finally, proceed to step S310 and step S311, perform inverse quantization on the V Cache corresponding to all tokens in the reserved buffer to obtain second inverse quantization information, and perform a dot multiplication operation on the weight coefficient and the second inverse quantization information to obtain a weighted sum, which is the query result corresponding to Q after linear mapping.

[0136] It is understandable that the multiple Transformer layers included in the target large model can all use the information processing method provided in the embodiment of the present disclosure to compress the K / V Cache.

[0137] As shown in Figure 4, Figure 4 is a flow chart of another information processing method according to an embodiment of the present disclosure. First, proceed to step S401, obtain the target machine learning model that has been trained offline, and obtain the key-value information corresponding to the query request based on the target machine learning model that has been trained offline. After determining the key-value information, proceed to step S402, and in the target machine learning model, calculate the scaling ratio scale of the key-value information in real time during forward reasoning, and proceed to step S403, quantize and compress the key-value information according to the calculated scale, and write the compressed K / V Cache into the reserved buffer. Afterwards, proceed to step S404, and dequantize the K / V Cache corresponding to the existing token in the reserved buffer in the decoding calculation of the target machine learning model, thereby placing the information calculation required for quantization such as the scale of the K / V Cache in the forward reasoning.

[0138] Tables 1 to 5 compare the information processing method provided by the embodiments of the present disclosure with the TRT-LLM quantification method in related technologies on models of different sizes in the opt and llama series. The PPL indicators are as follows:

[0139] Table 1 Comparison of PPL indicators using the opt-6.7B model

[0140] Table 2 Comparison of PPL indicators using the opt-30B model

[0141] Table 3 Comparison of PPL indicators using the llama-7B model

[0142] Table 4 Comparison of PPL indicators using the llama-13B model

[0143] Table 5 Comparison of PPL indicators using the llama-65B model

[0144] The PPL metrics for different model series and quantization methods in Tables 1-5 above demonstrate that the information processing method provided by the disclosed embodiments achieves significantly better compression results for K / V Cache than the related art TRT-LLM offline compression method for any model series, model size, and quantization method. Furthermore, it can be seen that the information processing method provided by the disclosed embodiments achieves a balance between accuracy and compression ratio when the quantization method is K8+V4, and even maintains near-uncompressed accuracy even under the extreme compression conditions of K4+V4.

[0145] It is easy to understand that the beneficial effects of the information processing method provided by the present disclosure include the following points.

[0146] The first beneficial effect is that the key-value information K / V is compressed by dynamic quantization, and no additional data is needed to calibrate the model in advance, making the use of the model simpler and more user-friendly, and there is no need to worry about the impact of the selection of calibration data on the accuracy of the compressed model.

[0147] Beneficial Effect 2: Referring to the above test data, the solution provided by the embodiments of the present disclosure, using a K8+V4 solution to compress the K / V Cache to approximately 3 / 8 the original FP16 size, achieves superior performance-per-second (PPL) metrics compared to the TRT-LLM model using a K8+V8 solution in the related art. Furthermore, from a video memory perspective, taking the llama 65B model in the related art as an example, with a batch size of 16 and FP16 precision, 80GB of video memory is required for the K / V Cache. However, using the solution provided by the embodiments of the present disclosure, the video memory usage is reduced to only approximately 30GB. Furthermore, during decoding calculations, the total amount of I / O required from the K / V Cache to the GPU global memory and shared memory / registers required for each new token generated is reduced to approximately 3 / 8 of the original amount. This demonstrates that the solution provided by the embodiments of the present disclosure can effectively reduce video memory usage during inference, reducing the total amount of input / output from the GPU global memory to the shared memory / registers during decoding calculations, and achieving superior PPL metrics compared to the models in the related art.

[0148] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0149] In addition, it should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0150] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present disclosure.

[0151] In an operating environment as in Example 1, the present disclosure provides a cloud server as shown in Figure 5. Figure 5 is a schematic diagram of a cloud server according to an embodiment of the present disclosure. As shown in Figure 5, the cloud server includes: a processor, a first cache, and a second cache.

[0152] A first cache is used to store parameter information corresponding to the query request, wherein the parameter information is obtained by encoding the query request;

[0153] a processor configured to, during a forward inference process of a target machine learning model, read parameter information from a first cache, perform quantization compression on the parameter information according to a preset quantization method to obtain quantization-compressed information, and perform quantization-compressed processing on the parameter information using the quantization-compressed information to obtain cache information, wherein the quantization-compressed information is used to indicate a ratio at which the parameter information is quantized and compressed, and the cache information is used to infer a query result corresponding to the query request;

[0154] The second cache is used to store cache information. In the embodiment of the present disclosure, the first cache can be a cache for storing recently used data, such as parameter information. The second cache can be a buffer for temporarily storing data, such as cache information.

[0155] The target machine learning model in the embodiment of the present disclosure can be deployed and run in the cloud server 10' in Figure 1. The target machine learning model can be a large model that has been pre-trained in an offline state. Exemplarily, the pre-trained large model can use a basic neural network as the target machine learning model, including an input layer, an embedding layer, an encoding layer, an attention layer, a convolutional layer, a decoding layer, a linear layer, and a loss layer, etc. Of course, the above-mentioned target machine learning model can also use existing large models, such as GPT, GPT-3, BERT, Transformer-XL, etc., which are not limited here.

[0156] A query request can be understood as a "query" (Q) in the target machine learning model, or a query vector. Q is part of the input to the target machine learning model and describes the user's question or need. Q can be a sentence or a paragraph of text. The target machine learning model aims to generate an appropriate answer or response based on the query Q.

[0157] Parameter information can be understood as information about parameters associated with a query request, such as key-value information. Quantization and compression information is used to represent the information required to perform quantization and compression on the parameter information. Exemplarily, the quantization and compression information may include a scale, which is used to adjust or control the size, capacity, or complexity of the model. In the disclosed embodiment, scale is used to represent the ratio of quantization and compression on the parameter information.

[0158] Cache information can be understood as cache information obtained by quantizing and compressing parameter information, for example, key-value cache information obtained by quantizing and compressing key-value information, including key cache information (K Cache) and value cache information (V Cache).

[0159] In an embodiment of the present disclosure, the processor in the cloud server is used to read parameter information corresponding to the query request from the first cache in the cloud server during the forward reasoning process of the target machine learning model, and quantize and compress the parameter information according to a preset quantization method, thereby obtaining the quantized compression information in real time and achieving the effect of dynamically determining the scale.

[0160] The processor is also used to, after obtaining the quantized compression information, perform quantized compression processing on the parameter information according to the quantized compression information obtained in real time, thereby dynamically compressing the parameter information and obtaining the effect of cache information that can determine the query result corresponding to the query request, and the cache information can be stored in a second cache in the cloud server. By quantizing and compressing the parameter information, the data accuracy is reduced, thereby reducing the memory usage and computational complexity. In addition, by dynamically quantizing the parameter information, it is possible to calibrate the model without using calibration data, avoiding the problem of affecting the model accuracy due to improper selection of calibration data, and there is no need to consider the acquisition of calibration data.

[0161] The above-mentioned information processing method provided by the embodiments of the present disclosure can be applied to, but is not limited to, application scenarios involving information processing in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services and navigation services, such as: query requests for e-commerce services, query requests for academic interpretations, query requests for medical means, etc., which are not limited here.

[0162] According to the disclosed embodiment, during the forward inference process of the target machine learning model, the processor in the cloud server can read the parameter information corresponding to the query request from the first cache in the cloud server, and quantize and compress the parameter information according to a preset quantization method, thereby obtaining the quantized compressed information in real time. After obtaining the quantized compressed information, the parameter information is quantized and compressed according to the quantized compressed information obtained in real time, thereby dynamically compressing the parameter information and obtaining the cache information that can determine the query result corresponding to the query request. The cache information can be stored in the second cache in the cloud server, thereby achieving the purpose of dynamically quantizing the parameter information (i.e., key-value information) and compressing the cache information of the parameter information (i.e., key-value cache), without the need for quantization calibration, thereby achieving the technical effect of effectively compressing the key-value cache while meeting the model accuracy, reducing the video memory usage during the inference process, and reducing the total input / output of the graphics processor global memory to the shared memory / register in the decoding calculation, thereby solving the technical problems in the related art of offline compression of the key-value cache, complex quantization calibration steps, and the model accuracy being affected by the calibration data, resulting in low model accuracy.

[0163] In an optional embodiment, the parameter information includes key-value information, and the processor is further configured to read the key-value information corresponding to the query request from the first cache.

[0164] Key-value information can be understood as a key vector (key, K) and a value vector (value, V) obtained through a linear mapping, where K is used to retrieve or find the index of specific information, and V is the information or data associated with the key. Usually, keys and values ​​are operated in the form of key-value pairs (K / V).

[0165] Exemplarily, the precision of the key-value information includes but is not limited to FP16, BF16, FP32 and the like.

[0166] The first cache can be understood as a cache area used to store key-value information during the forward reasoning process. For example, the key-value information can be stored in the weight vector at each position in the attention weights matrix of the cache area, which is not limited here.

[0167] In an optional embodiment, the processor is further configured to: encode the query request to obtain an encoding result; perform linear mapping on the encoding result to obtain key-value information; and store the key-value information in the first cache.

[0168] In an optional embodiment, the preset quantization method includes: a symmetric quantization method, the quantization compression information includes: scale information, and the processor is further used to: in the symmetric quantization method, perform quantization compression based on the key value information and the target number of bits to be quantized to obtain scale information.

[0169] The number of quantization bits is used to represent the number of binary bits of the quantization level. In the embodiment of the present disclosure, the target number of bits to be quantized is used to represent the number of bits to which the key-value information is quantized and compressed, denoted as N. The target number of bits to be quantized can be determined based on actual conditions. For example, setting the target number of bits to be quantized to 8 bits means that the key-value information is quantized and compressed to 8 bits, which can represent 2 to the power of 8 different quantization levels. By setting the target number of bits to be quantized, the storage requirements and computational overhead of the model can be reduced to a certain extent, thereby improving the efficiency and inference speed of the model.

[0170] In the symmetric quantization method, quantization compression can be performed based on the key value information and the target number of bits to be quantized, thereby determining scale information, that is, determining the compression ratio scale of the key value information.

[0171] For example, the data type after quantization and compression of the key-value information includes, but is not limited to, INT8, INT4, UINT8, UINT4, and other data types. Obtaining the quantization and compression information includes, but is not limited to, obtaining the maximum value (Max) or percentile (Percentile). During the quantization and compression process, data rounding methods include, but are not limited to, rounding up, rounding down, and rounding off, which are not limited here.

[0172] In an optional embodiment, the key-value information includes: the key to be quantized and the value to be quantized, and the processor is further used to: according to the preset quantization compression granularity, use the scale information to perform quantization compression processing on the key to be quantized in a symmetric quantization method to obtain key cache information, and use the scale information to perform quantization compression processing on the value to be quantized to obtain value cache information.

[0173] The cache information includes key cache information (K Cache) and value cache information (V Cache). K Cache can be understood as the cache information obtained after quantizing and compressing K. Correspondingly, V Cache can be understood as the cache information obtained after quantizing and compressing V.

[0174] The expression of symmetric quantization is X q =[X / Δ], where X represents the data before quantization, X q represents the quantized data, Δ represents the scale information, which can be obtained by Δ=max(|X|) / (2 N-1 -1) OK.

[0175] Specifically, the key to be quantized and the value to be quantized can be understood as the above X, which is used to represent K and V before quantization compression. The key cache information and the value cache information can be understood as the above X q , used to represent KCache and V Cache obtained after quantization compression.

[0176] The preset quantization compression granularity includes but is not limited to per-token, per-head, and other granularities.

[0177] In an optional embodiment, the preset quantization method includes: an asymmetric quantization method, the quantization compression information includes: scale information and zero point position information, and the processor is also used to: in the asymmetric quantization method, perform quantization compression based on the key value information and the target number of bits to be quantized to obtain scale information and zero point position information.

[0178] In the asymmetric quantization method, in addition to determining the scale information Δ, it is also necessary to determine the zero-point position information.

[0179] In the asymmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized, and the scale information Δ and the zero point position information can be determined.

[0180] In an optional embodiment, the key-value information includes: the key to be quantized and the value to be quantized, and the processor is further used to: according to the preset quantization compression granularity, use the scale information and the zero point position information in an asymmetric quantization method to perform quantization compression processing on the key to be quantized to obtain key cache information, and use the scale information and the zero point position information to perform quantization compression processing on the value to be quantized to obtain value cache information.

[0181] The expression of asymmetric quantization is X q =[X / Δ]+Z, where X represents the data before quantization, X q represents the quantized data, Δ represents the scale information, which can be obtained by Δ={max(X)-min(X)} / (2 N -1) is determined, Z represents the zero point position information, which can be determined by Z=-2 N-1 -[min(X) / Δ] is determined.

[0182] Specifically, the key to be quantized and the value to be quantized can be understood as the above X, which is used to represent K and V before quantization compression. The key cache information and the value cache information can be understood as the above X q , used to represent KCache and V Cache obtained after quantization compression.

[0183] The preset quantization compression granularity includes but is not limited to per-token, per-head, and other granularities.

[0184] In an optional embodiment, the processor is further configured to: write the cache information into the second cache.

[0185] The second cache can be understood as a reserved buffer. In the disclosed embodiment, the reserved buffer is used as a data buffer to temporarily store data, and the key cache information and value cache information obtained in real time are written into the reserved buffer for storage, thereby reducing the number of data transmissions and improving the efficiency of data transmission.

[0186] In an optional embodiment, the processor is also used to: read cache information from the second cache, wherein the cache information includes: key cache information and value cache information; perform inverse quantization processing on the key cache information to obtain first inverse quantization information, and perform inverse quantization processing on the value cache information to obtain second inverse quantization information; obtain query results based on the query request, the first inverse quantization information and the second inverse quantization information.

[0187] In the disclosed embodiment, when reading cache information from the second cache, it is necessary to obtain the cache information corresponding to the generated token, namely, the key cache information and value cache information corresponding to the generated token. The key cache information and value cache information corresponding to the generated token can be understood as the K Cache and V Cache corresponding to all previous tokens, including the current token.

[0188] The first dequantization information can be understood as the dequantization information obtained by dequantizing the K Cache stored in the second cache, that is, the dequantization information obtained by dequantizing the K Cache corresponding to the current token and the K Cache corresponding to all previous tokens. Correspondingly, the second dequantization information can be understood as the dequantization information obtained by dequantizing the V Cache stored in the second cache, that is, the dequantization information obtained by dequantizing the V Cache corresponding to the current token and the V Cache corresponding to all previous tokens.

[0189] In an optional embodiment, the processor is further configured to: perform a multiplication operation on the query request and the first dequantization information to obtain a weight coefficient; and perform a multiplication operation on the second dequantization information and the weight coefficient to obtain a query result.

[0190] It should be noted that the preferred implementation of this embodiment can be found in the relevant description in Example 1 and will not be repeated here.

[0191] In the operating environment as in Example 1, the present disclosure provides a cloud server as shown in Figure 6. Figure 6 is a flow chart of a cloud server according to an embodiment of the present disclosure. As shown in Figure 6, the cloud server includes:

[0192] Step S61: obtaining parameter information corresponding to the e-commerce service query request during the forward reasoning process of the e-commerce service machine learning model, wherein the parameter information is obtained by encoding the e-commerce service query request;

[0193] Step S62: quantize and compress the parameter information according to a preset quantization method to obtain quantized compressed information, wherein the quantized compressed information is used to indicate a ratio of quantized compression of the parameter information;

[0194] Step S63, performing quantization compression processing on the parameter information using the quantization compression information to obtain cache information, wherein the cache information is used to represent cache information obtained after the quantization compression processing is performed on the parameter information;

[0195] Step S64: Obtain the query result corresponding to the e-commerce service query request based on the cache information inference.

[0196] The e-commerce service machine learning model can be a large model pre-trained offline. For example, the pre-trained large model can use a basic neural network as the e-commerce service machine learning model, including an input layer, an embedding layer, an encoding layer, an attention layer, a convolutional layer, a decoding layer, a linear layer, and a loss layer. Of course, the above-mentioned e-commerce service machine learning model can also use existing large models, such as GPT, GPT-3, BERT, Transformer-XL, etc., which are not limited here.

[0197] The e-commerce service query request can be understood as the "e-commerce service query" (query, Q) in the e-commerce service machine learning model, that is, the e-commerce service query vector. Q is part of the input of the e-commerce service model, which is used to describe the user's e-commerce service query question or e-commerce service query demand. Q can be a sentence or a paragraph of text. The goal of the e-commerce service machine learning model is to generate a suitable answer or response based on the e-commerce service query Q. For example, a user can make an e-commerce service query request for the e-commerce service "Which courier?" The e-commerce service machine learning model can generate a suitable answer "XX courier" based on the user's e-commerce service query request.

[0198] Parameter information can be understood as information about parameters associated with an e-commerce service query request, such as key-value information.

[0199] The quantization compression information is used to represent the information required to quantize and compress the parameter information. Exemplarily, the quantization compression information may include a scale, which is used to adjust or control the size, capacity, or complexity of the e-commerce service model. In the embodiment of the present disclosure, the scale is used to represent the ratio of quantization compression of the parameter information.

[0200] Cache information can be understood as cache information obtained by quantizing and compressing parameter information, for example, key-value cache information obtained by quantizing and compressing key-value information, including key cache information (K Cache) and value cache information (V Cache).

[0201] In the disclosed embodiments, during the forward reasoning process of the e-commerce service machine learning model, parameter information corresponding to the e-commerce service query request is obtained by encoding the obtained e-commerce service query request. After obtaining the parameter information, the parameter information is quantized and compressed according to a preset quantization method to obtain quantized compressed information. This can be understood as obtaining quantized compressed information in real time based on the parameter information corresponding to the e-commerce service query request, thereby achieving a dynamic scale determination effect.

[0202] After obtaining the quantized compressed information, the parameter information can be quantized and compressed based on the real-time quantized compressed information, achieving dynamic compression of the parameter information and achieving the effect of caching the information. By quantizing and compressing the parameter information, the data precision is reduced, thereby reducing video memory usage and computational complexity. In addition, by dynamically quantizing the parameter information, the model can be calibrated without the use of calibration data, avoiding the problem of model accuracy being affected by improper selection of calibration data, and there is no need to consider the acquisition of calibration data.

[0203] After obtaining the cache information, the query result corresponding to the e-commerce service query request can be inferred based on the cache information, thereby effectively compressing the cache information, effectively reducing its video memory occupancy, and reducing the total IO required from GPU global memory to shared memory / registers in decoding calculations, while also ensuring the accuracy of the model.

[0204] The above-mentioned information processing method provided by the embodiment of the present disclosure can also be applied to, but is not limited to, application scenarios involving information processing in the fields of educational services, legal services, medical services, conference services, social network services, financial product services, logistics services and navigation services, such as: query requests for academic interpretations, query requests for medical means, etc., which are not limited here.

[0205] By adopting the embodiment of the present disclosure, in the forward reasoning process of the e-commerce service machine learning model, the parameter information corresponding to the e-commerce service query request is first obtained by encoding the e-commerce service query request, and then the parameter information is quantized and compressed according to a preset quantization method to obtain quantized compressed information, and then the quantized compressed information is used to quantize and compress the parameter information to obtain cache information, and finally, based on the cache information, the query result corresponding to the e-commerce service query request is obtained by reasoning, thereby achieving the purpose of dynamically quantizing parameter information (i.e., key-value information) and compressing the cache information of the parameter information (i.e., key-value cache), and no quantization calibration is required, thereby achieving the technical effect of effectively compressing the key-value cache while meeting the model accuracy, reducing the video memory occupancy in the reasoning process, and reducing the total input / output of the graphics processor global memory to the shared memory / register in the decoding calculation, thereby solving the technical problems in the related technology of offline compression of key-value cache, complex quantization calibration steps, and the model accuracy being affected by calibration data, resulting in low model accuracy.

[0206] It should be noted that the preferred implementation of this embodiment can be found in the relevant description in Example 1 and will not be repeated here.

[0207] According to an embodiment of the present disclosure, an embodiment of a device for implementing the above-mentioned cloud server is also provided. FIG7 is a structural diagram of a cloud server device according to an embodiment of the present disclosure. As shown in FIG7 , the device includes:

[0208] An acquisition module 701 is configured to acquire parameter information corresponding to a query request during a forward inference process of a target machine learning model, wherein the parameter information is obtained by encoding the query request;

[0209] A first processing module 702 is configured to perform quantization compression on the parameter information according to a preset quantization method to obtain quantization compression information, wherein the quantization compression information is used to indicate a ratio of quantization compression on the parameter information;

[0210] The second processing module 703 is configured to perform quantization compression processing on the parameter information using the quantization compression information to obtain cache information, wherein the cache information is used to represent cache information obtained after the quantization compression processing is performed on the parameter information;

[0211] The reasoning module 704 is configured to obtain a query result corresponding to the query request based on the cache information.

[0212] Optionally, the parameter information includes key-value information, and the acquisition module 701 is further configured to read the key-value information corresponding to the query request from the first cache.

[0213] Optionally, it further includes: an encoding module, configured to encode the query request to obtain an encoding result; perform linear mapping on the encoding result to obtain key-value information; and store the key-value information in a first cache.

[0214] Optionally, the preset quantization method includes: a symmetric quantization method, and the quantization compression information includes: scale information. The above-mentioned first processing module 702 is also used to: in the symmetric quantization method, perform quantization compression based on the key value information and the target number of bits to be quantized to obtain scale information.

[0215] Optionally, the key-value information includes: the key to be quantized and the value to be quantized. The above-mentioned second processing module 703 is also used to: according to the preset quantization compression granularity, use the scale information to perform quantization compression processing on the key to be quantized in a symmetric quantization method to obtain key cache information, and use the scale information to perform quantization compression processing on the value to be quantized to obtain value cache information.

[0216] Optionally, the preset quantization method includes: an asymmetric quantization method, and the quantization compression information includes: scale information and zero point position information. The above-mentioned first processing module 702 is also used to: in the asymmetric quantization method, perform quantization compression based on the key value information and the target number of bits to be quantized to obtain scale information and zero point position information.

[0217] Optionally, the key-value information includes: the key to be quantized and the value to be quantized. The above-mentioned second processing module 703 is also used to: according to the preset quantization compression granularity, use the scale information and the zero point position information in an asymmetric quantization method to perform quantization compression processing on the key to be quantized to obtain key cache information, and use the scale information and the zero point position information to perform quantization compression processing on the value to be quantized to obtain value cache information.

[0218] Optionally, the device further includes: a writing module, configured to write the cache information into the second cache.

[0219] Optionally, the above-mentioned reasoning module 704 is also used to: read cache information from the second cache, wherein the cache information includes: key cache information and value cache information; perform inverse quantization processing on the key cache information to obtain first inverse quantization information, and perform inverse quantization processing on the value cache information to obtain second inverse quantization information; obtain the query result based on the query request, the first inverse quantization information and the second inverse quantization information.

[0220] Optionally, the above-mentioned reasoning module 704 is further used to: perform a multiplication operation on the query request and the first dequantization information to obtain a weight coefficient; and perform a multiplication operation on the second dequantization information and the weight coefficient to obtain a query result.

[0221] By adopting the embodiment of the present disclosure, in the forward reasoning process of the target machine learning model, the query request is first encoded to obtain the parameter information corresponding to the query request, and then the parameter information is quantized and compressed according to a preset quantization method to obtain quantized compressed information, and then the quantized compressed information is used to perform quantized compression processing on the parameter information to obtain cache information, and finally, the query result corresponding to the query request is obtained based on the reasoning of the cache information. In this way, the purpose of dynamically quantizing the parameter information (i.e., key-value information) and compressing the cache information of the parameter information (i.e., key-value cache) is achieved without the need for quantization calibration, thereby achieving the technical effect of effectively compressing the key-value cache while meeting the model accuracy, reducing the video memory occupancy during the inference process, and reducing the total input / output of the graphics processor global memory to the shared memory / register in the decoding calculation, thereby solving the technical problems in the related art of offline compression of the key-value cache, complex quantization calibration steps, and the model accuracy being affected by the calibration data, resulting in low model accuracy.

[0222] It should be noted that the acquisition module 701, the first processing module 702, the second processing module 703, and the inference module 704 correspond to steps S21 to S24 in Example 1. The four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The above modules can also run in the server 10 or server 10' provided in Example 1.

[0223] According to an embodiment of the present disclosure, another embodiment of a device for implementing the above-mentioned cloud server is also provided. FIG8 is a structural diagram of another cloud server device according to an embodiment of the present disclosure. As shown in FIG8 , the device includes:

[0224] An acquisition module 801 is configured to acquire parameter information corresponding to an e-commerce service query request during a forward inference process of an e-commerce service machine learning model, wherein the parameter information is obtained by encoding the e-commerce service query request;

[0225] A first processing module 802 is configured to perform quantization compression on the parameter information according to a preset quantization method to obtain quantization compression information, wherein the quantization compression information is used to indicate a ratio of quantization compression on the parameter information;

[0226] The second processing module 803 is configured to perform quantization compression processing on the parameter information using the quantization compression information to obtain cache information, wherein the cache information is used to represent cache information obtained after the quantization compression processing is performed on the parameter information;

[0227] The reasoning module 804 is used to obtain the query result corresponding to the e-commerce service query request based on the cache information.

[0228] By adopting the embodiment of the present disclosure, in the forward reasoning process of the e-commerce service machine learning model, the parameter information corresponding to the e-commerce service query request is first obtained by encoding the e-commerce service query request, and then the parameter information is quantized and compressed according to a preset quantization method to obtain quantized compressed information, and then the quantized compressed information is used to quantize and compress the parameter information to obtain cache information, and finally, based on the cache information, the query result corresponding to the e-commerce service query request is obtained by reasoning, thereby achieving the purpose of dynamically quantizing parameter information (i.e., key-value information) and compressing the cache information of the parameter information (i.e., key-value cache), and no quantization calibration is required, thereby achieving the technical effect of effectively compressing the key-value cache while meeting the model accuracy, reducing the video memory occupancy in the reasoning process, and reducing the total input / output of the graphics processor global memory to the shared memory / register in the decoding calculation, thereby solving the technical problems in the related technology of offline compression of key-value cache, complex quantization calibration steps, and the model accuracy being affected by calibration data, resulting in low model accuracy.

[0229] It should be noted that the acquisition module 801, the first processing module 802, the second processing module 803, and the inference module 804 correspond to steps S61 to S64 in Example 3. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 3. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The above modules can also run in the server 10 or cloud server 10' provided in Example 1.

[0230] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0231] The embodiment of the present disclosure may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0232] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0233] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the cloud server: in the forward reasoning process of the target machine learning model, obtain parameter information corresponding to the query request, wherein the parameter information is obtained by encoding the query request; quantize and compress the parameter information according to a preset quantization method to obtain quantized compressed information, wherein the quantized compressed information is used to represent the ratio of quantized compression of the parameter information; use the quantized compressed information to quantize and compress the parameter information to obtain cache information, wherein the cache information is used to represent the cached information obtained after the parameter information is quantized and compressed; and obtain the query result corresponding to the query request based on reasoning of the cache information.

[0234] Alternatively, Figure 9 is a block diagram of a computer terminal according to an embodiment of the present disclosure. As shown in Figure 9, the computer terminal 9 may include: one or more (only one is shown in the figure) processors 902, a memory 904, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0235] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the cloud server and device in the embodiment of the present disclosure. The processor executes various functional applications and data processing by running the stored software programs and modules, that is, realizing the above-mentioned cloud server. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 9 via a network. Examples of the above-mentioned network include but are not limited to the Internet, corporate intranet, local area network, mobile communication network and combinations thereof.

[0236] The present disclosure also provides a processor. Figure 10 shows a schematic diagram of the structure of a processor for implementing the present disclosure. As shown in Figure 10, the processor 1000 is configured to run a program, wherein the program, when executed by the processor, executes the method in the above embodiment.

[0237] In the embodiment of the present disclosure, the processor 1000 may execute an operating program for an information processing method.

[0238] Optionally, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: in the forward reasoning process of the target machine learning model, obtain parameter information corresponding to the query request, wherein the parameter information is obtained by encoding the query request; quantize and compress the parameter information according to a preset quantization method to obtain quantized compressed information, wherein the quantized compressed information is used to represent the ratio of quantized compression of the parameter information; use the quantized compressed information to quantize and compress the parameter information to obtain cache information, wherein the cache information is used to represent the cached information obtained after the parameter information is quantized and compressed; and obtain the query result corresponding to the query request based on the inference of the cache information.

[0239] Optionally, the parameter information includes: key value information, and the processor may further execute program code of the following steps: reading the key value information corresponding to the query request from the first cache.

[0240] Optionally, the processor may further execute program codes of the following steps: encoding the query request to obtain an encoding result; performing linear mapping on the encoding result to obtain key-value information; and storing the key-value information in the first cache.

[0241] Optionally, the preset quantization method includes: a symmetric quantization method, and the quantization compression information includes: scale information. The above-mentioned processor can also execute the program code of the following steps: in the symmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain scale information.

[0242] Optionally, the key-value information includes: the key to be quantized and the value to be quantized. The above-mentioned processor can also execute the program code of the following steps: according to the preset quantization compression granularity, the scale information is used to perform quantization compression processing on the key to be quantized in a symmetric quantization method to obtain key cache information, and the scale information is used to perform quantization compression processing on the value to be quantized to obtain value cache information.

[0243] Optionally, the preset quantization method includes: an asymmetric quantization method, and the quantization compression information includes: scale information and zero point position information. The above-mentioned processor can also execute the program code of the following steps: in the asymmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain scale information and zero point position information.

[0244] Optionally, the key-value information includes: the key to be quantized and the value to be quantized. The above-mentioned processor can also execute the program code of the following steps: according to the preset quantization compression granularity, the scale information and zero point position information are used in an asymmetric quantization method to perform quantization compression processing on the key to be quantized to obtain key cache information, and the scale information and zero point position information are used to perform quantization compression processing on the value to be quantized to obtain value cache information.

[0245] Optionally, the processor may further execute program code of the following steps: writing the cache information into the second cache.

[0246] Optionally, the processor may also execute the following program code: reading cache information from the second cache, wherein the cache information includes: key cache information and value cache information; performing inverse quantization on the key cache information to obtain first inverse quantization information, and performing inverse quantization on the value cache information to obtain second inverse quantization information; obtaining query results based on the query request, the first inverse quantization information, and the second inverse quantization information.

[0247] Optionally, the processor may further execute program code of the following steps: multiplying the query request by the first dequantization information to obtain a weight coefficient; and multiplying the second dequantization information by the weight coefficient to obtain a query result.

[0248] Optionally, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: in the forward reasoning process of the e-commerce service machine learning model, obtain parameter information corresponding to the e-commerce service query request, wherein the parameter information is obtained by encoding the e-commerce service query request; quantize and compress the parameter information according to a preset quantization method to obtain quantized compressed information, wherein the quantized compressed information is used to represent the ratio of quantized compression of the parameter information; use the quantized compressed information to quantize and compress the parameter information to obtain cache information, wherein the cache information is used to represent the cached information obtained after the parameter information is quantized and compressed; and obtain the query result corresponding to the e-commerce service query request based on the inference of the cache information.

[0249] By adopting the embodiment of the present disclosure, in the forward reasoning process of the e-commerce service machine learning model, the parameter information corresponding to the e-commerce service query request is first obtained by encoding the e-commerce service query request, and then the parameter information is quantized and compressed according to a preset quantization method to obtain quantized compressed information, and then the quantized compressed information is used to quantize and compress the parameter information to obtain cache information, and finally, based on the cache information, the query result corresponding to the e-commerce service query request is obtained by reasoning, thereby achieving the purpose of dynamically quantizing parameter information (i.e., key-value information) and compressing the cache information of the parameter information (i.e., key-value cache), and no quantization calibration is required, thereby achieving the technical effect of effectively compressing the key-value cache while meeting the model accuracy, reducing the video memory occupancy in the reasoning process, and reducing the total input / output of the graphics processor global memory to the shared memory / register in the decoding calculation, thereby solving the technical problems in the related technology of offline compression of key-value cache, complex quantization calibration steps, and the model accuracy being affected by calibration data, resulting in low model accuracy.

[0250] Those skilled in the art will appreciate that the structure shown in FIG9 is merely illustrative, and that the computer terminal 9 may also be a smartphone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile internet device (MID), a PAD, or other terminal device. FIG9 does not limit the structure of the aforementioned electronic devices. For example, the computer terminal 9 may include more or fewer components (e.g., a network interface, a display device, etc.) than those shown in FIG9 , or may have a configuration different from that shown in FIG9 .

[0251] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0252] The embodiment of the present disclosure further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the cloud server provided in the first embodiment.

[0253] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0254] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: in the forward reasoning process of the target machine learning model, obtaining parameter information corresponding to the query request, wherein the parameter information is obtained by encoding the query request; quantizing and compressing the parameter information according to a preset quantization method to obtain quantized compressed information, wherein the quantized compressed information is used to represent the ratio of quantized compression of the parameter information; using the quantized compressed information to perform quantized compression processing on the parameter information to obtain cache information, wherein the cache information is used to represent the cached information obtained after the parameter information is quantized and compressed; and obtaining the query result corresponding to the query request based on reasoning of the cache information.

[0255] Optionally, the parameter information includes: key value information. In this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: reading the key value information corresponding to the query request from the first cache.

[0256] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: encoding the query request to obtain an encoding result; linearly mapping the encoding result to obtain key value information; and storing the key value information in the first cache.

[0257] Optionally, the preset quantization method includes: a symmetric quantization method, and the quantization compression information includes: scale information. In this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: in the symmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain scale information.

[0258] Optionally, the key-value information includes: a key to be quantized and a value to be quantized. In this embodiment, a computer-readable storage medium is configured to store program code for executing the following steps: according to a preset quantization compression granularity, using scale information to perform quantization compression processing on the key to be quantized in a symmetric quantization manner to obtain key cache information, and using scale information to perform quantization compression processing on the value to be quantized to obtain value cache information.

[0259] Optionally, the preset quantization method includes: an asymmetric quantization method, and the quantization compression information includes: scale information and zero point position information. In this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: in the asymmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain scale information and zero point position information.

[0260] Optionally, the key-value information includes: a key to be quantized and a value to be quantized. In this embodiment, a computer-readable storage medium is configured to store program code for executing the following steps: according to a preset quantization compression granularity, the key to be quantized is quantized and compressed using scale information and zero-point position information in an asymmetric quantization manner to obtain key cache information, and the value to be quantized is quantized and compressed using scale information and zero-point position information to obtain value cache information.

[0261] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: writing the cache information into the second cache.

[0262] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: reading cache information from the second cache, wherein the cache information includes: key cache information and value cache information; performing inverse quantization processing on the key cache information to obtain first inverse quantization information, and performing inverse quantization processing on the value cache information to obtain second inverse quantization information; obtaining query results based on the query request, the first inverse quantization information and the second inverse quantization information.

[0263] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: multiplying the query request with the first inverse quantization information to obtain a weight coefficient; and multiplying the second inverse quantization information with the weight coefficient to obtain a query result.

[0264] Computer-readable storage media may also be referred to as computer storage media. They may include data signals transmitted in baseband or as part of a carrier wave, carrying readable program code. Such transmitted data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable storage media may transmit, propagate, or transfer programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0265] The program code contained in the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency, etc., or any suitable combination of the foregoing.

[0266] An embodiment of the present disclosure also provides a computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the above-mentioned image generation method or image generation data processing method.

[0267] The embodiments of the present disclosure further provide a computer program, which, when executed by a processor, implements the steps of the above-mentioned image generation method or image generation data processing method.

[0268] Optionally, the computer program implements the following program code when executed by the processor: in the forward reasoning process of the target machine learning model, obtaining parameter information corresponding to the query request, wherein the parameter information is obtained by encoding the query request; quantizing and compressing the parameter information according to a preset quantization method to obtain quantized compressed information, wherein the quantized compressed information is used to represent the ratio of quantized compression of the parameter information; using the quantized compressed information to quantize and compress the parameter information to obtain cache information, wherein the cache information is used to represent the cached information obtained after the parameter information is quantized and compressed; and obtaining the query result corresponding to the query request based on reasoning of the cache information.

[0269] Optionally, the parameter information includes: key value information. In this embodiment, when the above computer program is executed by the processor, the program code implements the following steps: reading the key value information corresponding to the query request from the first cache.

[0270] Optionally, when the above computer program is executed by a processor, program code implementing the following steps: encoding the query request to obtain an encoding result; performing linear mapping on the encoding result to obtain key-value information; and storing the key-value information in a first cache.

[0271] Optionally, the preset quantization method includes: a symmetric quantization method, and the quantization compression information includes: scale information. In this embodiment, when the above-mentioned computer program is executed by the processor, the program code implements the following steps: in the symmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain scale information.

[0272] Optionally, the key-value information includes: a key to be quantized and a value to be quantized. In this embodiment, when the above-mentioned computer program is executed by a processor, the program code implements the following steps: according to a preset quantization compression granularity, the scale information is used in a symmetric quantization method to perform quantization compression processing on the key to be quantized to obtain key cache information, and the scale information is used to perform quantization compression processing on the value to be quantized to obtain value cache information.

[0273] Optionally, the preset quantization method includes: an asymmetric quantization method, and the quantization compression information includes: scale information and zero point position information. In this embodiment, when the above-mentioned computer program is executed by the processor, the program code implements the following steps: in the asymmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain scale information and zero point position information.

[0274] Optionally, the key-value information includes: a key to be quantized and a value to be quantized. In this embodiment, when the above-mentioned computer program is executed by a processor, the program code implements the following steps: according to a preset quantization compression granularity, the key to be quantized is quantized and compressed using scale information and zero point position information in an asymmetric quantization method to obtain key cache information, and the value to be quantized is quantized and compressed using scale information and zero point position information to obtain value cache information.

[0275] Optionally, in this embodiment, when the computer program is executed by the processor, program code for implementing the following steps: writing the cache information into the second cache.

[0276] Optionally, in this embodiment, the above-mentioned computer program implements program code of the following steps when executed by the processor: reading cache information from the second cache, wherein the cache information includes: key cache information and value cache information; performing inverse quantization processing on the key cache information to obtain first inverse quantization information, and performing inverse quantization processing on the value cache information to obtain second inverse quantization information; obtaining query results based on the query request, the first inverse quantization information and the second inverse quantization information.

[0277] Optionally, in this embodiment, when the above-mentioned computer program is executed by the processor, the program code implements the following steps: multiplying the query request with the first dequantization information to obtain a weight coefficient; multiplying the second dequantization information with the weight coefficient to obtain a query result.

[0278] In the above embodiments of the present disclosure, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0279] In the several embodiments provided in the present disclosure, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0280] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0281] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0282] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, ROM, RAM, mobile hard drives, magnetic disks or optical disks.

[0283] The above is only a preferred embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present disclosure. These improvements and modifications should also be regarded as within the scope of protection of the present disclosure. Industrial Applicability

[0284] The solution provided by the embodiment of the present disclosure can be applied to the process of information processing. In the forward reasoning process of the target machine learning model, parameter information corresponding to the query request is obtained, wherein the parameter information is obtained by encoding the query request; the parameter information is quantized and compressed according to a preset quantization method to obtain quantized compressed information, wherein the quantized compressed information is used to represent the ratio of quantized compression of the parameter information; the parameter information is quantized and compressed using the quantized compressed information to obtain cache information, wherein the cache information is used to represent cached information obtained after quantized compression of the parameter information; the query result corresponding to the query request is obtained based on reasoning of the cache information, thereby solving the technical problems in the related art of offline compressed key-value caching, complex quantization calibration steps, and the model accuracy being affected by the calibration data, resulting in low model accuracy.

Claims

1. An information processing method, comprising: In a forward reasoning process of a target machine learning model, obtaining parameter information corresponding to a query request, wherein the parameter information is obtained by encoding the query request; Quantizing and compressing the parameter information according to a preset quantization method to obtain quantized compressed information, wherein the quantized compressed information is used to indicate a ratio of quantizing and compressing the parameter information; Using the quantized compression information to perform quantized compression processing on the parameter information to obtain cache information, wherein the cache information is used to represent cache information obtained after the quantized compression processing is performed on the parameter information; A query result corresponding to the query request is obtained by inference based on the cache information.

2. The information processing method according to claim 1, wherein: The parameter information includes: key value information, and obtaining the parameter information corresponding to the query request includes: The key value information corresponding to the query request is read from the first cache.

3. The information processing method according to claim 2, wherein: The information processing method further includes: Encoding the query request to obtain an encoding result; Performing linear mapping on the encoding result to obtain the key value information; The key-value information is stored in the first cache.

4. The information processing method according to claim 2, wherein: The preset quantization method includes: a symmetric quantization method, the quantization compression information includes: scale information, and the parameter information is quantized and compressed according to the preset quantization method, and the quantization compression information obtained includes: In the symmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain the scale information.

5. The information processing method according to claim 4, wherein: The key value information includes: a key to be quantized and a value to be quantized. The parameter information is quantized and compressed using the quantization compression information to obtain the cache information including: According to the preset quantization compression granularity, the key to be quantized is quantized and compressed using the scale information in the symmetric quantization method to obtain key cache information, and the value to be quantized is quantized and compressed using the scale information to obtain value cache information.

6. The information processing method according to claim 2, wherein: The preset quantization method includes: an asymmetric quantization method, the quantization compression information includes: scale information and zero point position information, and the parameter information is quantized and compressed according to the preset quantization method, and the quantization compression information obtained includes: In the asymmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain the scale information and the zero point position information.

7. The information processing method according to claim 6, wherein: The key value information includes: a key to be quantized and a value to be quantized. The parameter information is quantized and compressed using the quantization compression information to obtain the cache information including: According to the preset quantization compression granularity, the scale information and the zero point position information are used in the asymmetric quantization method to perform quantization compression processing on the key to be quantized to obtain key cache information, and the scale information and the zero point position information are used to perform quantization compression processing on the value to be quantized to obtain value cache information.

8. The information processing method according to claim 1, wherein: The information processing method further includes: The cache information is written into the second cache.

9. The information processing method according to claim 8, wherein: The query result corresponding to the query request obtained by inference based on the cache information includes: Reading the cache information from the second cache, wherein the cache information includes: key cache information and value cache information; Performing a dequantization process on the key cache information to obtain first dequantization information, and performing a dequantization process on the value cache information to obtain second dequantization information; The query result is obtained based on the query request, the first dequantization information, and the second dequantization information.

10. The information processing method according to claim 9, wherein: Acquiring the query result based on the query request, the first dequantization information, and the second dequantization information includes: Performing a multiplication operation on the query request and the first dequantized information to obtain a weight coefficient; The second dequantization information is multiplied by the weight coefficient to obtain the query result.

11. A cloud server, comprising: A processor, a first cache and a second cache; The first cache is used to store parameter information corresponding to the query request, wherein the parameter information is obtained by encoding the query request; The processor is used to read the parameter information from the first cache during the forward reasoning process of the target machine learning model, quantize and compress the parameter information according to a preset quantization method to obtain quantized compressed information, and use the quantized compressed information to quantize and compress the parameter information to obtain cache information, wherein the quantized compressed information is used to indicate a ratio of quantizing and compressing the parameter information, and the cache information is used to infer a query result corresponding to the query request; The second cache is used to store the cache information.

12. The cloud server according to claim 11, wherein: The processor is also used to read the cache information from the second cache, wherein the cache information includes: key cache information and value cache information; perform dequantization processing on the key cache information to obtain first dequantization information, and perform dequantization processing on the value cache information to obtain second dequantization information; obtain the query result based on the query request, the first dequantization information and the second dequantization information.

13. The cloud server according to claim 11, wherein: The parameter information includes: key value information, and the processor is further configured to read the key value information corresponding to the query request from the first cache.

14. The cloud server according to claim 13, wherein: The processor is further configured to: Encoding the query request to obtain an encoding result; Performing linear mapping on the encoding result to obtain the key value information; The key-value information is stored in the first cache.

15. The cloud server according to claim 13, wherein: The preset quantization mode includes: a symmetric quantization mode, and the quantization compression information includes: scale information; the processor is further configured to perform quantization compression on the parameter information according to the preset quantization mode to obtain the quantization compression information by executing the following steps: In the symmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain the scale information.

16. An electronic device, comprising: A memory storing an executable program; A processor, configured to run the program, wherein the program, when running, executes the information processing method according to any one of claims 1 to 10.

17. A computer program product, wherein: The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 10.

18. A computer program product, wherein: The invention comprises a non-volatile computer-readable storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 10 when executed by a processor.

19. A computer program, wherein When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the information processing method according to any one of claims 1 to 10.

Images