Information processing method, electronic equipment and computer readable storage medium

Abstract

The invention discloses an information processing method, electronic equipment and a computer readable storage medium, and relates to the technical field of large model technology and data processing. The method comprises the steps that in the forward reasoning process of a target machine learning model, parameter information corresponding to a query request is obtained, and the parameter information is obtained after the query request is encoded; performing quantization compression on the parameter information according to a preset quantization mode to obtain quantization compression information; performing quantitative compression processing on the parameter information by adopting the quantitative compression information to obtain cache information; and reasoning based on the cache information to obtain a query result corresponding to the query request. According to the method and the device, the technical problems that offline compression key value caching and quantitative calibration steps are complicated, and the model precision is influenced by calibration data, so that the model precision is relatively low in the prior art are solved.

Description

Technical Field

[0001] The present application relates to the fields of large model technology and data processing technology, and more specifically, to an information processing method, an electronic device and a computer-readable storage medium. Background Art

[0002] As the amount of computational data increases, researchers have built artificial intelligence (AI) models with large parameters, also known as large language models (LLM), such as ChatGPT, by increasing the scale and complexity of the model, thereby improving the performance of the model in language understanding, generation and related tasks. In the process of large model reasoning, video memory is one of the bottlenecks. The key-value cache (K / V Cache) generated during the reasoning process will occupy a large amount of video memory, and some large models require multiple graphics processing unit (GPU) cards during the reasoning process. In addition, in the decoding calculation of large models, the bandwidth from GPU global memory to shared memory / register is also a bottleneck. Every time a new token is generated, it needs to be read from the existing K / V Cache. Therefore, compressing K / VCache can reduce the occupation of video memory during large model reasoning, and can also reduce the total input / output (IO) from GPU global memory to shared memory / register in decoding calculation, thereby improving the efficiency of decoding.

[0003] Currently, the TRT-LLM model only supports offline quantization for K / V Cache. This quantization method requires calibration data to be used in advance. The accuracy of the model is affected by the calibration data, and there is a problem that the calibration data is not available in some scenarios. In addition, the accuracy of the model is low after the TRT-LLM model compresses the K / V Cache offline, which makes the behavior of saving video memory meaningless.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide an information processing method, an electronic device and a computer-readable storage medium to at least solve the technical problems in the related art that offline compressed key-value caching, the quantization calibration steps are complicated, and the model accuracy is affected by the calibration data, resulting in low model accuracy.

[0006] According to one aspect of an embodiment of the present application, there is provided an information processing method, including: in a forward reasoning process of a target machine learning model, obtaining parameter information corresponding to a query request, wherein the parameter information is obtained by encoding the query request; quantizing and compressing the parameter information according to a preset quantization method to obtain quantized compressed information; using the quantized compressed information to quantize and compress the parameter information to obtain cache information; and obtaining a query result corresponding to the query request by reasoning based on the cache information.

[0007] According to another aspect of an embodiment of the present application, an information processing method is also provided, including: a first cache, used to store parameter information corresponding to a query request, wherein the parameter information is obtained by encoding the query request; a processor, used to read the parameter information from the first cache during a forward reasoning process of a target machine learning model, quantize and compress the parameter information according to a preset quantization method to obtain quantized compressed information, and use the quantized compressed information to quantize and compress the parameter information to obtain cache information, wherein the cache information is used to infer a query result corresponding to the query request; and a second cache, used to store the cache information.

[0008] According to another aspect of an embodiment of the present application, an electronic device is further provided, including: a memory storing an executable program; and a processor for running the program, wherein any one of the above-mentioned information processing methods is executed when the program is running.

[0009] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned information processing methods.

[0010] In an embodiment of the present application, during the forward reasoning process of the target machine learning model, the query request is first encoded to obtain parameter information corresponding to the query request, and then the parameter information is quantized and compressed according to a preset quantization method to obtain quantized compressed information, and then the quantized compressed information is used to perform quantized compression processing on the parameter information to obtain cache information, and finally, the query result corresponding to the query request is obtained based on the cache information reasoning, thereby achieving the purpose of dynamically quantizing parameter information (i.e., key-value information) and compressing cache information of the parameter information (i.e., key-value cache), and no quantization calibration is required, thereby achieving the technical effect of effectively compressing the key-value cache while meeting the model accuracy, reducing the video memory occupancy during the reasoning process, and reducing the total input / output of the graphics processor global memory to the shared memory / register in the decoding calculation, thereby solving the technical problems in the related technology of offline compression of key-value cache, complex quantization calibration steps, and model accuracy being affected by calibration data, resulting in low model accuracy.

[0011] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0013] Figure 1 This is a schematic diagram of an application scenario of an information processing method according to Embodiment 1 of the present application;

[0014] Figure 2 is a flowchart of an information processing method according to Embodiment 1 of the present application;

[0015] Figure 3 is a flowchart of another information processing method according to Embodiment 1 of the present application;

[0016] Figure 4 is a flowchart of another information processing method according to Embodiment 1 of the present application;

[0017] Figure 5 is a schematic diagram of a cloud server according to Embodiment 2 of the present application;

[0018] Figure 6 is a flowchart of an information processing method according to Embodiment 3 of the present application;

[0019] Figure 7 is a structural schematic diagram of an information processing method and device according to Embodiment 4 of the present application;

[0020] Figure 8 is a structural schematic diagram of another information processing method and device according to Embodiment 4 of the present application;

[0021] Fig. 9 It is a structural block diagram of a computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] The technical solution provided in this application is mainly implemented by large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model). The large model is pre-trained through large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (Large Language Model, LLM), multi-modal pre-training model (multi-modal pre-training model), etc.

[0025] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields, specifically in computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, etc. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0026] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0027] Large Language Model (LLM): A machine learning model with a large number of parameters and complex structure, which is an extended form of the neural network model.

[0028] K / V Cache: A key-value cache used to store and quickly access key-value pairs. K / V Cache is an efficient caching mechanism that can be used to accelerate data access and reduce data storage pressure.

[0029] Quantization: In neural networks, quantization is the process of converting floating-point parameters or activation values ​​into integers or fixed-point representations with lower digits. The purpose of quantization is to reduce the memory usage and computational effort in the neural network model, thereby improving the efficiency and inference speed of the model.

[0030] Dequantization: In neural networks, dequantization is the process of converting quantized integer or fixed-point representations back to floating-point representations. The purpose of dequantization is to convert quantized integer or fixed-point representations back to floating-point representations so that normal neural network calculations and reasoning can be performed, preserving the precision and accuracy of the model.

[0031] Symmetric Quantization: The range of floating-point values ​​is divided into a fixed number of intervals, and each interval is represented by an integer. In symmetric quantization, the value 0 is usually represented by a special integer, and the range of negative and positive values ​​is the same, thus reducing storage requirements.

[0032] Asymmetric Quantization: Dynamically adjust the range of model weights so that the range of integer representation is more suitable for the distribution of weights. Asymmetric quantization usually introduces a scaling factor and a zero offset to map floating point numbers to integer representation and perform inverse mapping during calculation. By dynamically adjusting the scaling factor and zero offset, asymmetric quantization can better preserve the accuracy of weights and reduce storage requirements.

[0033] Per-token: A quantization technique used to reduce the storage and computational cost of natural language processing (NLP) models. In NLP tasks, models usually consist of many different parameters that need to be stored and calculated, resulting in large model size and computational complexity. The goal of per-token quantization is to reduce the number of parameters per word or token.

[0034] Per-head: A quantization technique. In NLP tasks, models usually use a multi-head self-attention mechanism to process input sequences. Each attention head has its own parameters that need to be stored and calculated. The goal of per-head quantization is to reduce the number of parameters for each attention head.

[0035] Forward Inference: refers to the process of passing input data through the model's calculation process, from the input layer to the output layer, to generate the model's prediction results. In neural networks, forward inference refers to the process of passing input samples through the network's forward propagation process, calculating and passing them layer by layer, and finally obtaining the model's output. During the forward inference process, the input data undergoes a series of linear transformations, activation functions, pooling operations, etc., gradually extracting and transforming features, and finally generating the model's prediction results. Forward inference is usually used to predict or infer new input data after model training is completed.

[0036] PPL index: Perplexity index, is an index for evaluating the performance of a large model, used to measure the degree of confusion or uncertainty of the large model when predicting the next word. The smaller the PPL index, the better.

[0037] K4 / K8: A quantization method that quantizes the K Cache in the multi-head attention block to 4bit / 8bit.

[0038] V4 / V8: A quantization method that quantizes the V Cache in the multi-head attention block to 4bit / 8bit.

[0039] Cache: Cache is designed to reduce data transmission delays between the processor and main memory and is used to store recently used data so that it can be quickly accessed when needed.

[0040] Buffer: Buffer is designed to solve the problem of mismatched data transmission speeds. It is used to temporarily store data so that data can be sent and received at a more balanced rate.

[0041] The related art has the following defects in offline quantization of K / V Cache based on the TRT-LLM model.

[0042] Defect 1: The offline quantization of K / V Cache in TRT-LLM requires the acquisition of calibration data in advance for quantization and calibration. The final accuracy of the model is affected by the selection of calibration data. In addition, there is a problem that calibration data is not available in some application scenarios. Offline calibration increases the complexity of LM reasoning deployment and is not very user-friendly.

[0043] Defect 2: When TRT-LLM offline quantization compresses K / V Cache to INT4, the accuracy of the LLM model is significantly affected. For example, for the llama-13B model, when the K / V Cache is compressed to INT4, the PPL index measured on the wiki-text dataset is higher than the PPL index based on the llama-7B model without K / V Cache compression. It can be understood that the accuracy of a larger model after K / V Cache optimization is lower than that of a smaller model of the same series, making it meaningless to save video memory and reduce the IO bandwidth requirements from global memory to shared memory / registers by compressing K / V Cache.

[0044] With respect to the above-mentioned defects, no effective solution has been proposed before the present application.

[0045] Example 1

[0046] According to an embodiment of the present application, an information processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] Considering the huge amount of model parameters of the large model and the limited computing resources of the mobile terminal, the above information processing method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1 In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiment of the present application.

[0048] Alternatively, if Figure 1As shown, the large model can also be deployed in the cloud server 10', and the cloud server 10' communicates with the client device 20 through a communication protocol, for example, it can communicate using Hypertext Transfer Protocol / Hypertext Transfer Protocol Secure (HTTP / HTTPS), communicate using Transmission Control Protocol / Internet Protocol (TCP / IP), communicate using Remote Procedure Call (RPC), etc., which are not limited here.

[0049] In an embodiment of the present application, a system consisting of a client device and a server can perform the following steps: the client device executes steps such as obtaining a query request input by a user in a graphical user interface and sending the query request to a server; the server executes steps such as outputting a query result based on the obtained query request and returning the query result to the client device. It should be noted that when the operating resources of the client device can meet the deployment and operating conditions of a large model, an embodiment of the present application can be performed in the client device.

[0050] Optionally, the server may also independently perform steps such as determining corresponding query results based on the obtained query request, thereby training a large model, etc., which is not limited here.

[0051] Under the above operating environment, this application provides Figure 2 The information processing method shown. Figure 2 is a flow chart of an information processing method according to Embodiment 1 of the present application. Figure 2 As shown, the method may include the following steps:

[0052] Step S21, in the forward reasoning process of the target machine learning model, obtaining parameter information corresponding to the query request, wherein the parameter information is obtained by encoding the query request;

[0053] Step S22, quantizing and compressing the parameter information according to a preset quantization method to obtain quantized compressed information;

[0054] Step S23, using the quantization compression information to perform quantization compression processing on the parameter information to obtain cache information;

[0055] Step S24, obtaining a query result corresponding to the query request based on inference of the cache information.

[0056] The target machine learning model can be a large model pre-trained offline. Exemplarily, the pre-trained large model can use a basic neural network as the target machine learning model, including an input layer, an embedding layer, an encoding layer, an attention layer, a convolutional layer, a decoding layer, a linear layer, and a loss layer. Of course, the above-mentioned target machine learning model can also use an existing large model, such as GPT, GPT-3, BERT, Transformer-XL, etc., which is not limited here.

[0057] The query request can be understood as the "query" (Q) in the target machine learning model, that is, the query vector. Q is part of the input of the target machine learning model and is used to describe the user's question or demand. Q can be a sentence or a paragraph of text. The goal of the target machine learning model is to generate a suitable answer or response (Answer) based on the query Q.

[0058] The parameter information may be understood as information of parameters associated with the query request, such as key-value information.

[0059] The quantization compression information is used to represent the information required for quantization compression of the parameter information. Exemplarily, the quantization compression information may include a scaling ratio (scale), and the scale is used to adjust or control the size, capacity or complexity of the model. In the embodiment of the present application, the scale is used to represent the ratio of quantization compression of the parameter information.

[0060] The cache information can be understood as cache information obtained after quantizing and compressing parameter information, for example, key-value cache information obtained after quantizing and compressing key-value information, including key cache information (K Cache) and value cache information (V Cache).

[0061] In an embodiment of the present application, during the forward reasoning process of the target machine learning model, the parameter information corresponding to the query request can be obtained by encoding the obtained query request. After the parameter information is obtained, the parameter information is quantized and compressed according to a preset quantization method to obtain quantized compressed information, which can be understood as obtaining quantized compressed information in real time based on the parameter information corresponding to the query request, thereby achieving the effect of dynamically determining the scale.

[0062] After obtaining the quantized compressed information, the parameter information can be quantized and compressed according to the quantized compressed information obtained in real time, so as to achieve dynamic compression of the parameter information and obtain the effect of caching the information. By quantizing and compressing the parameter information, the data accuracy is reduced, thereby reducing the memory usage and computational complexity. In addition, by dynamically quantizing the parameter information, the model can be calibrated without using calibration data, avoiding the problem of affecting the model accuracy due to improper selection of calibration data, and there is no need to consider the acquisition of calibration data.

[0063] After obtaining the cache information, the query result corresponding to the query request can be inferred based on the cache information, thereby achieving effective compression of the cache information, effectively reducing its video memory occupancy, and reducing the total IO required from GPU global memory to shared memory / registers in decoding calculations, while also ensuring the accuracy of the model.

[0064] In an embodiment of the present application, by using a pre-trained target machine learning model, parameter information corresponding to the query request is obtained during the forward reasoning process of the target machine learning model, so as to execute the information processing method provided in the embodiment of the present application based on the query request and its corresponding parameter information. As a result, the pre-trained target machine learning model can be parameter tuned based on the information processing method provided in the embodiment of the present application, so that the optimized target machine learning model can achieve the technical effect of effectively compressing the cache while meeting the model accuracy, reducing the video memory occupancy during the reasoning process, and reducing the total input / output of the graphics processor global memory to the shared memory / register in the decoding calculation.

[0065] The above-mentioned information processing method provided in the embodiment of the present application can be applied to, but is not limited to, application scenarios involving information processing in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services and navigation services, for example: query requests for e-commerce services, query requests for academic interpretations, query requests for medical means, etc., which are not limited here.

[0066] By using the embodiment of the present application, in the forward reasoning process of the target machine learning model, the query request is first encoded to obtain the parameter information corresponding to the query request, and then the parameter information is quantized and compressed according to a preset quantization method to obtain quantized compressed information, and then the quantized compressed information is used to perform quantized compression processing on the parameter information to obtain cache information, and finally the query result corresponding to the query request is obtained based on the cache information reasoning, thereby achieving the purpose of dynamically quantizing the parameter information (i.e., key-value information) and compressing the cache information of the parameter information (i.e., key-value cache), and no quantization calibration is required, thereby achieving the technical effect of effectively compressing the key-value cache under the premise of meeting the model accuracy, reducing the video memory occupancy during the reasoning process, and reducing the total input / output of the graphics processor global memory to the shared memory / register in the decoding calculation, thereby solving the technical problems in the related technology of offline compression of the key-value cache, complex quantization calibration steps, and the model accuracy being affected by the calibration data, resulting in low model accuracy.

[0067] In an optional embodiment, the parameter information includes: key value information. In step S21, obtaining the parameter information corresponding to the query request includes the following method steps:

[0068] Step S211: read key value information corresponding to the query request from the first cache.

[0069] Key-value information can be understood as a key vector (key, K) and a value vector (value, V) obtained through linear mapping, where K is used to retrieve or find the index of specific information, and V is the information or data associated with the key. Usually, keys and values ​​are operated in the form of key-value pairs (K / V).

[0070] Exemplarily, the precision of the key-value information includes, but is not limited to, FP16, BF16, FP32, and the like.

[0071] The first cache can be understood as a cache area for storing key value information during the forward reasoning process. For example, the key value information can be stored in the weight vector at each position in the attention weights matrix of the cache area, which is not limited here.

[0072] In the embodiment of the present application, obtaining parameter information corresponding to the query request can be understood as reading key value information corresponding to the query request from the first cache, so that the query result corresponding to the query request can be determined based on the key value information.

[0073] In an optional embodiment, the information processing method further includes the following method steps:

[0074] Step S201, encode the query request to obtain an encoding result;

[0075] Step S202, linear mapping is performed on the encoding result to obtain key value information;

[0076] Step S203: store the key value information in the first cache.

[0077] During the forward reasoning process of the target machine learning model, the input query request is encoded to obtain an encoding result, which is then linearly mapped to obtain the key-value information corresponding to the query request, and the key-value information can be written into the first cache for storage.

[0078] Exemplarily, the input query request text sequence can be converted into a fixed-length vector representation based on the encoder in the target machine learning model. The encoder can use a recurrent neural network (RNN) or Transformer structure to gradually process the words or characters of the input text, extract the semantic and grammatical features of the text layer by layer, and encode the text into a fixed-length vector representation. The encoded vector can contain the overall information of the input query request text, so as to be used in the subsequent decoding process.

[0079] After obtaining the encoding result, the encoding result is linearly mapped from a low dimension to a high dimension, thereby obtaining key value information, which is conducive to capturing more fine-grained semantic information in the input text and extracting key features. For example, the encoding result can be mapped to a new vector space by matrix multiplication and bias addition, and the new vector space can have a higher dimension and thus have richer feature representation capabilities.

[0080] After the key value information is obtained, the key value information is written into the first cache for storage, thereby facilitating subsequent reading of the key value information corresponding to the query request.

[0081] In an optional embodiment, the preset quantization method includes: a symmetric quantization method, and the quantization compression information includes: scale information. In step S22, the parameter information is quantized and compressed according to the preset quantization method to obtain the quantization compression information, including the following method steps:

[0082] Step S221 , in a symmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain scale information.

[0083] When the parameter information is quantized and compressed based on a preset quantization method, the preset quantization method may be a symmetric quantization method.

[0084] The number of quantization bits is used to represent the number of binary bits of the quantization level. In the embodiment of the present application, the target number of bits to be quantized is used to represent how many bits the key value information is quantized and compressed to, denoted as N. The target number of bits to be quantized can be determined according to the actual situation. For example, setting the target number of bits to be quantized to 8 bits means compressing the key value information to 8 bits, which can represent 2 to the power of 8 different quantization levels. By setting the target number of bits to be quantized, the storage requirements and computational overhead of the model can be reduced to a certain extent, thereby improving the efficiency and reasoning speed of the model.

[0085] In the symmetric quantization method, quantization compression can be performed based on the key value information and the target number of bits to be quantized, thereby determining the scale information, that is, determining the compression ratio scale of the key value information.

[0086] Exemplarily, the data type after quantization and compression of the key value information includes but is not limited to INT8, INT4, UINT8, UINT4 and other data types. Acquiring quantization and compression information includes but is not limited to obtaining the maximum value (Max) or percentile (Percentile). In the process of quantization and compression, the method of rounding the data includes but is not limited to rounding up, rounding down, rounding to the nearest integer, etc., which are not limited here.

[0087] In an optional embodiment, the key-value information includes: a key to be quantized and a value to be quantized. In step S23, the parameter information is quantized and compressed using the quantization compression information to obtain cache information, including the following method steps:

[0088] Step S231, according to the preset quantization compression granularity, the scale information is used in a symmetric quantization method to perform quantization compression processing on the key to be quantized to obtain key cache information, and the scale information is used to perform quantization compression processing on the value to be quantized to obtain value cache information.

[0089] The cache information includes key cache information (K Cache) and value cache information (V Cache), wherein K Cache can be understood as cache information obtained after quantizing and compressing K, and correspondingly, V Cache can be understood as cache information obtained after quantizing and compressing V.

[0090] The expression of symmetric quantization is X q =[X / Δ], where X represents the data before quantization, X q represents the quantized data, Δ represents the scale information, which can be expressed by Δ=max(|X|) / (2 N-1 -1) OK.

[0091] Specifically, the key to be quantized and the value to be quantized can be understood as the above X, which is used to represent K and V before quantization compression. The key cache information and the value cache information can be understood as the above X q , used to represent KCache and VCache obtained after quantization compression.

[0092] The preset quantization compression granularity includes but is not limited to per-token, per-head and other granularities.

[0093] In the symmetric quantization method, the key K to be quantized and the value V to be quantized may be quantized and compressed according to the preset quantization compression granularity and the determined scale information Δ, thereby obtaining the key cache information K Cache and the value cache information V Cache respectively.

[0094] In an optional embodiment, the preset quantization method includes: an asymmetric quantization method, and the quantization compression information includes: scale information and zero point position information. In step S22, the parameter information is quantized and compressed according to the preset quantization method to obtain the quantization compression information, including the following method steps:

[0095] Step S222, in an asymmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain scale information and zero point position information.

[0096] When the parameter information is quantized and compressed based on a preset quantization method, the preset quantization method may also be an asymmetric quantization method.

[0097] In the asymmetric quantization method, in addition to determining the scale information Δ, it is also necessary to determine the zero-point position information.

[0098] In the asymmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized, and the scale information Δ and the zero point position information can be determined.

[0099] Exemplarily, the data type after quantization and compression of the key value information includes but is not limited to INT8, INT4, UINT8, UINT4 and other data types. Acquiring quantization and compression information includes but is not limited to obtaining the maximum value (Max) or percentile (Percentile). In the process of quantization and compression, the method of rounding the data includes but is not limited to rounding up, rounding down, rounding to the nearest integer, etc., which are not limited here.

[0100] In an optional embodiment, the key-value information includes: a key to be quantized and a value to be quantized. In step S23, the parameter information is quantized and compressed using the quantization compression information to obtain cache information, including the following method steps:

[0101] Step S232, according to the preset quantization compression granularity, the scale information and the zero point position information are used in an asymmetric quantization method to perform quantization compression processing on the key to be quantized to obtain key cache information, and the scale information and the zero point position information are used to perform quantization compression processing on the value to be quantized to obtain value cache information.

[0102] The expression of asymmetric quantization is X q =[X / Δ]+Z, where X represents the data before quantization, X q represents the quantized data, Δ represents the scale information, which can be expressed by Δ = {max(X)-min(X)} / (2 N -1) is determined, Z represents the zero point position information, which can be determined by Z = -2 N-1 -[min(X) / Δ] is determined.

[0103] Specifically, the key to be quantized and the value to be quantized can be understood as the above X, which is used to represent K and V before quantization compression. The key cache information and the value cache information can be understood as the above X q , used to represent KCache and VCache obtained after quantization compression.

[0104] The preset quantization compression granularity includes but is not limited to per-token, per-head and other granularities.

[0105] In an asymmetric quantization method, the key K to be quantized and the value V to be quantized can be quantized and compressed according to the preset quantization compression granularity according to the determined scale information Δ and the zero point position information Z, thereby obtaining the key cache information K Cache and the value cache information V Cache respectively.

[0106] In an optional embodiment, the information processing method further includes the following method steps:

[0107] Step S25, writing the cache information into the second cache.

[0108] The second cache can be understood as a reserved buffer. In the embodiment of the present application, the reserved buffer is used as a data buffer to temporarily store data, and the key cache information and value cache information obtained in real time are written into the reserved buffer for storage, thereby reducing the number of data transmissions and improving the efficiency of data transmission.

[0109] In an optional embodiment, in step S24, obtaining a query result corresponding to the query request based on the cache information inference includes the following method steps:

[0110] Step S241, reading cache information from the second cache, wherein the cache information includes: key cache information and value cache information;

[0111] Step S242, performing dequantization processing on the key cache information to obtain first dequantization information, and performing dequantization processing on the value cache information to obtain second dequantization information;

[0112] Step S243: Perform a multiplication operation based on the query request, the first dequantization information, and the second dequantization information to obtain a query result.

[0113] In the large model calculation process, each step of the calculation will generate a corresponding token according to the calculation order, and each token is used to generate the three vectors Q, K, and V for the next step of calculation. The generated tokens will be input into the model as context information, so that new tokens can be determined based on the previously generated tokens. This generation method can maintain the consistency of the context, so that the expected token sequence can be generated according to the generation strategy and conditions.

[0114] In the embodiment of the present application, when reading cache information from the second cache, it is necessary to obtain the cache information corresponding to the generated token, that is, the key cache information and value cache information corresponding to the generated token. The key cache information and value cache information corresponding to the generated token can be understood as the K Cache and V Cache corresponding to all previous tokens including the current token.

[0115] The first dequantization information can be understood as the dequantization information obtained after dequantization processing is performed on the K Cache stored in the second cache, that is, the dequantization information obtained after dequantization processing is performed on the K Cache corresponding to the current token and the K Cache corresponding to all the previous tokens. Correspondingly, the second dequantization information can be understood as the dequantization information obtained after dequantization processing is performed on the V Cache stored in the second cache, that is, the dequantization information obtained after dequantization processing is performed on the V Cache corresponding to the current token and the V Cache corresponding to all the previous tokens.

[0116] In an embodiment of the present application, when the query result corresponding to the query request is obtained based on the cache information inference, K Cache and V Cache can be read from the second cache, and by performing dequantization operations on K Cache and V Cache respectively, K Cache and V Cache are restored to the original precision to obtain the first dequantization information obtained after dequantizing K Cache and the second dequantization information obtained after dequantizing VCache. The query result corresponding to the query request is obtained according to the obtained query request, the first dequantization information and the second dequantization information.

[0117] Specifically, when dequantizing the key cache information, different situations will occur depending on the selection of the target number of bits to be quantized. When the target number of bits to be quantized is greater than 1, the Q vector is [5.8, 6.3, 7.4], and the K Cache quantization includes three values ​​[1, -4, 4] T , taking scale as 16.3 as an example, the dequantization logic during forward reasoning is: first multiply K Cache by the multiplied scale, and then get [16.3, -65.2, 65.2] T , and then multiply the result by Q, that is, (5.8×16.3-6.3×65.2+7.4×65.2)=166.26, thereby obtaining the first dequantization information.

[0118] It should be noted that when the key-value information is quantized and compressed, if the key-value information is quantized into ternary digits, there is no need to dequantize the K Cache and V Cache obtained after the quantization compression during the forward reasoning process. The K Cache and V Cache obtained at this time are the first dequantized information and the second dequantized information obtained after dequantization.

[0119] For example, when the target number of bits to be quantized is equal to 1, that is, the key value information is quantized into a ternary value, there is no need to perform inverse quantization. If the Q vector is [5.8, 6.3, 7.4], K includes three values ​​after quantization [0, -1, 1] T, take scale as 65.2 as an example. Since K has only three possible values ​​of 0, -1 and 1 after quantization, the calculation logic of forward reasoning here is: first add Q and K, that is, (0-6.3+7.4)=1.1, and then multiply the result by scale, that is, 1.1×65.2=71.72, so as to obtain the key cache information, which is the first dequantization information. Correspondingly, the process of obtaining the second dequantization information can refer to the description of obtaining the first dequantization information, which will not be repeated here.

[0120] In an embodiment of the present application, during the decoding process of the target machine learning model, a multiplication operation can be performed based on the query request, the first dequantization information and the second dequantization information to obtain a query result corresponding to the current query request.

[0121] For example, the decoder in the target machine learning model takes the encoded vector representation as input, and generates output text sequences layer by layer by gradually generating words or characters. The decoder can use structures such as RNN or Transformer to gradually generate output text that conforms to language rules and context by learning generation rules and language models.

[0122] It is understandable that, in the target machine learning model calculation process, after obtaining the query result corresponding to the query request calculated in the previous step, the query result obtained will be used as the query request for the next step calculation, thereby determining the query result for the next step calculation based on the query result calculated in the previous step. For example, the initial query request is "Who are you?", and the target large model determines that the first query result is "I" based on the initial query request, and then uses the first query result "I" as the query request calculated in the previous step, thereby determining that the second query result is "yes", and so on, until the complete query result corresponding to the initial query request is obtained.

[0123] In an optional embodiment, in step S243, obtaining a query result based on the query request, the first inverse quantization information, and the second inverse quantization information includes the following method steps:

[0124] Step S2431, performing a multiplication operation on the query request and the first dequantization information to obtain a weight coefficient;

[0125] Step S2432, performing a multiplication operation on the second inverse quantization information and the weight coefficient to obtain a query result.

[0126] When obtaining a query result based on the query request, the first inverse quantization information and the second inverse quantization information, the query request is first multiplied by the first inverse quantization information to obtain a weight coefficient, and then the second inverse quantization information is multiplied by the obtained weight coefficient to obtain the query result.

[0127] like Figure 3 As shown, Figure 3 It is a flowchart of another information processing method according to Example 1 of the present application. First, based on the output token of the previous step, determine the Q, K, and V of the linear mapping input of this calculation. Then determine the quantization compression information (scale and other information) based on the linear mapping K and V and the selected quantization method. Then, based on the scale and other information, perform quantization compression processing on the linearly mapped K and V, and store the processed key cache information K Cache and value cache information V Cache in the reserved cache buffer (second cache). Then, perform dequantization processing on the K Cache corresponding to all tokens in the reserved buffer to obtain the first dequantization information.

[0128] Then, the linearly mapped Q is multiplied by the first inverse quantization information. Exemplarily, the linearly mapped Q is calculated based on the scaled dot-product attention mechanism with the first inverse quantization information. Specifically, the linearly mapped Q is first multiplied by the first inverse quantization information to obtain a first operation result. The first operation result is then divided by the scaling ratio scale to obtain a second operation result. The second operation result is masked (Mask) and normalized using the softmax function to finally obtain the attention weight, that is, the weight coefficient.

[0129] Finally, the V Cache corresponding to all tokens in the reserved buffer is dequantized to obtain the second dequantized information. The weight coefficient is multiplied by the second dequantized information to obtain the weighted sum, which is the query result corresponding to Q after linear mapping.

[0130] It can be understood that the multiple Transformer layers included in the target large model can all use the information processing method provided in the embodiment of the present application to compress the K / V Cache.

[0131] like Figure 4 As shown, Figure 4This is a flowchart of another information processing method according to Example 1 of the present application. First, the target machine learning model that has been trained offline is obtained, and the key-value information corresponding to the query request is obtained based on the target machine learning model that has been trained offline. After determining the key-value information, the scaling ratio scale of the key-value information is calculated in real time during the forward reasoning of the target machine learning model, and the key-value information is quantized and compressed according to the calculated scale, and the compressed K / VCache is written into the reserved buffer. Afterwards, the K / V Cache corresponding to the existing token in the reserved buffer is dequantized in the decoding calculation of the target machine learning model, so that the information calculation required for quantization such as the scale of the K / V Cache is placed in the forward reasoning.

[0132] Tables 1 to 5 compare the information processing method provided in the embodiment of the present application and the TRT-LLM quantification method in the related art on models of different sizes in the opt and llama series. The PPL index is as follows:

[0133] Table 1 Comparison of PPL indicators using the opt-6.7B model

[0134] Quantitative methods K4 V4 K8 V8 K4+V4 K4+V8 K8+V4 K8+V8 TRT-LLM 11.8047 11.8516 10.8594 10.8750 13.3047 11.8516 11.8516 10.8750 This program 10.8984 10.8594 10.8594 10.8594 10.9219 10.8984 10.8594 10.8594

[0135] Table 2 Comparison of PPL indicators using the opt-30B model

[0136] Quantitative methods K4 V4 K8 V8 K4+V4 K4+V8 K8+V4 K8+V8 TRT-LLM 10.3828 11.6753 9.5625 9.5625 13.4844 10.3828 11.7188 9.5625 This program 9.6016 9.5625 9.5625 9.5625 9.6016 9.6016 9.5625 9.5625

[0137] Table 3 Comparison of PPL indicators using the llama-7B model

[0138] Quantitative methods K4 V4 K8 V8 K4+V4 K4+V8 K8+V4 K8+V8 TRT-LLM 10.3984 10.1953 5.6758 5.7031 74.9375 10.6250 10.2188 5.7109 This program 5.8125 5.6992 5.6719 5.6719 5.8438 5.8125 5.6992 5.6719

[0139] Table 4 Comparison of PPL indicators using the llama-13B model

[0140] Quantitative methods K4 V4 K8 V8 K4+V4 K4+V8 K8+V4 K8+V8 TRT-LLM 6.1562 9.1953 5.0898 5.0938 44.7500 6.2266 9.2109 5.0938 This program 5.1875 5.0977 5.0898 5.0898 5.1992 5.1875 5.0977 5.0898

[0141] Table 5 PPL index comparison of llama-65B model

[0142] Quantitative methods K4 V4 K8 V8 K4+V4 K4+V8 K8+V4 K8+V8 TRT-LLM 4.7031 8.0859 3.5312 3.5762 18.3906 4.8086 8.0859 3.5762 This program 3.5977 3.5352 3.5273 3.5273 3.6055 3.5977 3.5352 3.5312

[0143] It can be seen from the PPL indicators of different series models and different quantization methods in Tables 1 to 5 above that the information processing method provided in the embodiment of the present application has a significantly better compression effect on K / VCache under any series model, any size model and any quantization method than the offline compression method of TRT-LLM in the related art. It can also be seen that the information processing method provided in the embodiment of the present application can balance the accuracy and compression ratio when the quantization method is K8+V4, and can also ensure the accuracy close to that of no compression under the extreme compression of the quantization method K4+V4.

[0144] It is easy to understand that the beneficial effects of the information processing method provided by the present application include the following points.

[0145] Beneficial effect (1): Dynamic quantization is used to compress key value information K / V, and no additional data is needed to calibrate the model in advance, making the model easier to use and more user-friendly. There is no need to worry about the impact of the selection of calibration data on the accuracy of the compressed model.

[0146] Beneficial effect (2), referring to the above test data, the solution provided in the embodiment of the present application adopts the K8+V4 solution to compress the K / V Cache to about 3 / 8 of the original FP16 size, and the PPL index is better than the TRT-LLM model in the related art using the K8+V8 solution. And from the perspective of video memory, taking the llama 65B model in the related art as an example, when the batch size is 16 and the precision is FP16, 80G of video memory needs to be reserved for the K / V Cache. However, using the solution provided in the embodiment of the present application, the video memory occupancy is only about 30G. And in the decoding calculation of the solution provided in the embodiment of the present application, for each new token generated, the total IO amount of the GPU global memory to the shared memory / register read by the K / VCache is about 3 / 8 of the original. It can be seen that the solution provided in the embodiment of the present application can effectively reduce the video memory occupancy during the inference process, reduce the total input / output amount of the graphics processor global memory to the shared memory / register in the decoding calculation, and the PPL index of the model is better than the model in the related art.

[0147] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0148] In addition, it should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0149] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including several instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0150] Example 2

[0151] In the operating environment as in Example 1, the present application provides Figure 5 A cloud server is shown. Figure 5 is a schematic diagram of a cloud server according to Example 2 of the present application, such as Figure 5 As shown, the information processing method includes: a processor, a first cache and a second cache.

[0152] A first cache is used to store parameter information corresponding to the query request, wherein the parameter information is obtained by encoding the query request;

[0153] A processor is used to read parameter information from a first cache during a forward reasoning process of a target machine learning model, quantize and compress the parameter information according to a preset quantization method to obtain quantized compressed information, and use the quantized compressed information to quantize and compress the parameter information to obtain cache information, wherein the cache information is used to infer a query result corresponding to the query request;

[0154] The second cache is used to store cache information. In the embodiment of the present application, the first cache may be a cache Cache, which is used to store recently used data, such as parameter information. The second cache may be a buffer Buffer, which is used to temporarily store data, such as cache information.

[0155] The target machine learning model in the embodiment of the present application can be deployed in Figure 1The target machine learning model can be a large model pre-trained offline. Exemplarily, the pre-trained large model can use a basic neural network as the target machine learning model, including an input layer, an embedding layer, an encoding layer, an attention layer, a convolutional layer, a decoding layer, a linear layer, and a loss layer. Of course, the above-mentioned target machine learning model can also use an existing large model, such as GPT, GPT-3, BERT, Transformer-XL, etc., which is not limited here.

[0156] The query request can be understood as the "query" (Q) in the target machine learning model, that is, the query vector. Q is part of the input of the target machine learning model and is used to describe the user's question or demand. Q can be a sentence or a paragraph of text. The goal of the target machine learning model is to generate a suitable answer or response based on the query Q.

[0157] Parameter information can be understood as information about parameters associated with a query request, such as key-value information. Quantitative compression information is used to represent information required for quantitative compression of parameter information. Exemplarily, the quantitative compression information may include a scaling ratio (scale), which is used to adjust or control the size, capacity, or complexity of the model. In an embodiment of the present application, scale is used to represent the ratio of quantitative compression of parameter information.

[0158] The cache information can be understood as cache information obtained after quantizing and compressing parameter information, for example, key-value cache information obtained after quantizing and compressing key-value information, including key cache information (K Cache) and value cache information (V Cache).

[0159] In an embodiment of the present application, the processor in the cloud server is used to read parameter information corresponding to the query request from the first cache in the cloud server during the forward reasoning process of the target machine learning model, and quantize and compress the parameter information according to a preset quantization method, thereby obtaining the quantized compression information in real time and achieving the effect of dynamically determining the scale.

[0160] The processor is also used to perform quantitative compression processing on the parameter information according to the quantitative compression information obtained in real time after obtaining the quantitative compression information, so as to achieve dynamic compression of the parameter information, and obtain the effect of cache information that can determine the query result corresponding to the query request, and the cache information can be stored in the second cache in the cloud server. By quantizing and compressing the parameter information, the data accuracy is reduced, thereby reducing the video memory usage and the computational complexity. In addition, by dynamically quantizing the parameter information, it is possible to calibrate the model without using calibration data, avoiding the problem of affecting the model accuracy due to improper selection of calibration data, and there is no need to consider the acquisition of calibration data.

[0161] The above-mentioned information processing method provided in the embodiment of the present application can be applied to, but is not limited to, application scenarios involving information processing in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services and navigation services, for example: query requests for e-commerce services, query requests for academic interpretations, query requests for medical means, etc., which are not limited here.

[0162] By adopting the embodiment of the present application, through the processor in the cloud server, the parameter information corresponding to the query request can be read from the first cache in the cloud server during the forward reasoning process of the target machine learning model, and the parameter information can be quantized and compressed according to the preset quantization method, so as to obtain the quantized compressed information in real time. After obtaining the quantized compressed information, the parameter information is quantized and compressed according to the quantized compressed information obtained in real time, so as to achieve the effect of dynamically compressing the parameter information, and obtaining the cache information of the query result corresponding to the query request, and the cache information can be stored in the second cache in the cloud server, thereby achieving the purpose of dynamically quantizing the parameter information (i.e., key value information) and compressing the cache information of the parameter information (i.e., key value cache), and no quantization calibration is required, thereby achieving the technical effect of effectively compressing the key value cache under the premise of meeting the model accuracy, reducing the video memory occupancy in the reasoning process, and reducing the total input / output of the global memory of the graphics processor to the shared memory / register in the decoding calculation, thereby solving the technical problems of offline compression of the key value cache, complex quantization calibration steps, and the model accuracy being affected by the calibration data, resulting in low model accuracy.

[0163] In an optional embodiment, the parameter information includes: key value information, and the processor is further configured to: read the key value information corresponding to the query request from the first cache.

[0164] Key-value information can be understood as a key vector (key, K) and a value vector (value, V) obtained through linear mapping, where K is used to retrieve or find the index of specific information, and V is the information or data associated with the key. Usually, keys and values ​​are operated in the form of key-value pairs (K / V).

[0165] Exemplarily, the precision of the key-value information includes, but is not limited to, FP16, BF16, FP32, and the like.

[0166] The first cache can be understood as a cache area for storing key value information during the forward reasoning process. For example, the key value information can be stored in the weight vector at each position in the attention weights matrix of the cache area, which is not limited here.

[0167] In an optional embodiment, the processor is further used to: encode the query request to obtain an encoding result; perform linear mapping on the encoding result to obtain key value information; and store the key value information in the first cache.

[0168] In an optional embodiment, the preset quantization method includes: a symmetric quantization method, the quantization compression information includes: scale information, and the processor is also used to: in the symmetric quantization method, perform quantization compression based on the key value information and the target number of bits to be quantized to obtain scale information.

[0169] The number of quantization bits is used to represent the number of binary bits of the quantization level. In the embodiment of the present application, the target number of bits to be quantized is used to represent how many bits the key value information is quantized and compressed to, denoted as N. The target number of bits to be quantized can be determined according to the actual situation. For example, setting the target number of bits to be quantized to 8 bits means compressing the key value information to 8 bits, which can represent 2 to the power of 8 different quantization levels. By setting the target number of bits to be quantized, the storage requirements and computational overhead of the model can be reduced to a certain extent, thereby improving the efficiency and reasoning speed of the model.

[0170] In the symmetric quantization method, quantization compression can be performed based on the key value information and the target number of bits to be quantized, thereby determining the scale information, that is, determining the compression ratio scale of the key value information.

[0171] Exemplarily, the data type after quantization and compression of the key value information includes but is not limited to INT8, INT4, UINT8, UINT4 and other data types. Acquiring quantization and compression information includes but is not limited to obtaining the maximum value (Max) or percentile (Percentile). In the process of quantization and compression, the method of rounding the data includes but is not limited to rounding up, rounding down, rounding to the nearest integer, etc., which are not limited here.

[0172] In an optional embodiment, the key-value information includes: a key to be quantized and a value to be quantized, and the processor is further used to: according to a preset quantization compression granularity, use scale information to perform quantization compression processing on the key to be quantized in a symmetric quantization method to obtain key cache information, and use scale information to perform quantization compression processing on the value to be quantized to obtain value cache information.

[0173] The cache information includes key cache information (K Cache) and value cache information (V Cache), wherein K Cache can be understood as cache information obtained after quantizing and compressing K, and correspondingly, V Cache can be understood as cache information obtained after quantizing and compressing V.

[0174] The expression of symmetric quantization is X q =[X / Δ], where X represents the data before quantization, X qrepresents the quantized data, Δ represents the scale information, which can be expressed by Δ=max(|X|) / (2 N-1 -1) OK.

[0175] Specifically, the key to be quantized and the value to be quantized can be understood as the above X, which is used to represent K and V before quantization compression. The key cache information and the value cache information can be understood as the above X q , used to represent KCache and VCache obtained after quantization compression.

[0176] The preset quantization compression granularity includes but is not limited to per-token, per-head and other granularities.

[0177] In an optional embodiment, the preset quantization method includes: an asymmetric quantization method, the quantization compression information includes: scale information and zero point position information, and the processor is also used to: in the asymmetric quantization method, perform quantization compression based on the key value information and the target number of bits to be quantized to obtain the scale information and zero point position information.

[0178] In the asymmetric quantization method, in addition to determining the scale information Δ, it is also necessary to determine the zero-point position information.

[0179] In the asymmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized, and the scale information Δ and the zero point position information can be determined.

[0180] In an optional embodiment, the key-value information includes: a key to be quantized and a value to be quantized, and the processor is further used to: perform quantization compression processing on the key to be quantized using scale information and zero point position information in an asymmetric quantization manner according to a preset quantization compression granularity to obtain key cache information, and perform quantization compression processing on the value to be quantized using scale information and zero point position information to obtain value cache information.

[0181] The expression of asymmetric quantization is X q =[X / Δ]+Z, where X represents the data before quantization, X q represents the quantized data, Δ represents the scale information, which can be expressed by Δ = {max(X)-min(X)} / (2 N -1) is determined, Z represents the zero point position information, which can be determined by Z = -2 N-1 -[min(X) / Δ] is determined.

[0182] Specifically, the key to be quantized and the value to be quantized can be understood as the above X, which is used to represent K and V before quantization compression. The key cache information and the value cache information can be understood as the above X q , used to represent KCache and VCache obtained after quantization compression.

[0183] The preset quantization compression granularity includes but is not limited to per-token, per-head and other granularities.

[0184] In an optional embodiment, the processor is further configured to: write the cache information into the second cache.

[0185] The second cache can be understood as a reserved buffer. In the embodiment of the present application, the reserved buffer is used as a data buffer to temporarily store data, and the key cache information and value cache information obtained in real time are written into the reserved buffer for storage, thereby reducing the number of data transmissions and improving the efficiency of data transmission.

[0186] In an optional embodiment, the processor is also used to: read cache information from the second cache, wherein the cache information includes: key cache information and value cache information; perform inverse quantization processing on the key cache information to obtain first inverse quantization information, and perform inverse quantization processing on the value cache information to obtain second inverse quantization information; obtain query results based on the query request, the first inverse quantization information and the second inverse quantization information.

[0187] In the embodiment of the present application, when reading cache information from the second cache, it is necessary to obtain the cache information corresponding to the generated token, that is, the key cache information and value cache information corresponding to the generated token. The key cache information and value cache information corresponding to the generated token can be understood as the K Cache and V Cache corresponding to all previous tokens including the current token.

[0188] The first dequantization information can be understood as the dequantization information obtained after dequantization processing is performed on the K Cache stored in the second cache, that is, the dequantization information obtained after dequantization processing is performed on the K Cache corresponding to the current token and the K Cache corresponding to all the previous tokens. Correspondingly, the second dequantization information can be understood as the dequantization information obtained after dequantization processing is performed on the V Cache stored in the second cache, that is, the dequantization information obtained after dequantization processing is performed on the V Cache corresponding to the current token and the V Cache corresponding to all the previous tokens.

[0189] In an optional embodiment, the processor is further configured to: perform a multiplication operation on the query request and the first dequantization information to obtain a weight coefficient; and perform a multiplication operation on the second dequantization information and the weight coefficient to obtain a query result.

[0190] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.

[0191] Example 3

[0192] In the operating environment as in Example 1, the present application provides Figure 6 An information processing method is shown. Figure 6 is a flowchart of an information processing method according to Embodiment 3 of the present application, such as Figure 6 As shown, the information processing method includes:

[0193] Step S61, in the forward reasoning process of the e-commerce service machine learning model, obtaining parameter information corresponding to the e-commerce service query request, wherein the parameter information is obtained by encoding the e-commerce service query request;

[0194] Step S62, quantizing and compressing the parameter information according to a preset quantization method to obtain quantized compressed information;

[0195] Step S63, using the quantization compression information to perform quantization compression processing on the parameter information to obtain cache information;

[0196] Step S64, obtaining a query result corresponding to the e-commerce service query request based on inference of the cache information.

[0197] The e-commerce service machine learning model can be a large model pre-trained offline. Exemplarily, the pre-trained large model can use a basic neural network as the e-commerce service machine learning model, including an input layer, an embedding layer, an encoding layer, an attention layer, a convolutional layer, a decoding layer, a linear layer, and a loss layer. Of course, the above-mentioned e-commerce service machine learning model can also use existing large models, such as GPT, GPT-3, BERT, Transformer-XL, etc., which are not limited here.

[0198] The e-commerce service query request can be understood as the "e-commerce service query" (query, Q) in the e-commerce service machine learning model, that is, the e-commerce service query vector. Q is part of the input of the e-commerce service model, which is used to describe the user's e-commerce service query question or e-commerce service query demand. Q can be a sentence or a paragraph of text. The goal of the e-commerce service machine learning model is to generate a suitable answer or response based on the e-commerce service query Q. For example, a user can make an e-commerce service query request for the e-commerce service "Which courier should I send?" The e-commerce service machine learning model can generate a suitable answer "XX courier" based on the user's e-commerce service query request.

[0199] The parameter information may be understood as information of parameters associated with the e-commerce service query request, such as key-value information.

[0200] The quantization compression information is used to represent the information required for quantization compression of the parameter information. Exemplarily, the quantization compression information may include a scaling ratio (scale), which is used to adjust or control the size, capacity or complexity of the e-commerce service model. In an embodiment of the present application, scale is used to represent the ratio of quantization compression of the parameter information.

[0201] The cache information can be understood as cache information obtained after quantizing and compressing parameter information, for example, key-value cache information obtained after quantizing and compressing key-value information, including key cache information (K Cache) and value cache information (V Cache).

[0202] In an embodiment of the present application, during the forward reasoning process of the e-commerce service machine learning model, the parameter information corresponding to the e-commerce service query request can be obtained by encoding the obtained e-commerce service query request. After obtaining the parameter information, the parameter information is quantized and compressed according to a preset quantization method to obtain quantized compressed information, which can be understood as obtaining quantized compressed information in real time based on the parameter information corresponding to the e-commerce service query request, thereby achieving the effect of dynamically determining the scale.

[0203] After obtaining the quantized compressed information, the parameter information can be quantized and compressed according to the quantized compressed information obtained in real time, so as to achieve dynamic compression of the parameter information and obtain the effect of caching the information. By quantizing and compressing the parameter information, the data accuracy is reduced, thereby reducing the memory usage and computational complexity. In addition, by dynamically quantizing the parameter information, the model can be calibrated without using calibration data, avoiding the problem of affecting the model accuracy due to improper selection of calibration data, and there is no need to consider the acquisition of calibration data.

[0204] After obtaining the cache information, the query result corresponding to the e-commerce service query request can be inferred based on the cache information, thereby achieving effective compression of the cache information, effectively reducing its video memory occupancy, and reducing the total IO required from GPU global memory to shared memory / registers in decoding calculations, while also ensuring the accuracy of the model.

[0205] The above-mentioned information processing method provided in the embodiment of the present application can also be applied to application scenarios involving information processing in the fields of educational services, legal services, medical services, conference services, social network services, financial product services, logistics services and navigation services, such as: query requests for academic interpretations, query requests for medical means, etc., which are not limited here.

[0206] By using the embodiment of the present application, in the forward reasoning process of the e-commerce service machine learning model, the parameter information corresponding to the e-commerce service query request is first obtained by encoding the e-commerce service query request, and then the parameter information is quantized and compressed according to a preset quantization method to obtain quantized compressed information, and then the quantized compressed information is used to perform quantized compression processing on the parameter information to obtain cache information, and finally, based on the cache information, the query result corresponding to the e-commerce service query request is obtained by reasoning, thereby achieving the purpose of dynamically quantizing parameter information (i.e., key-value information) and compressing the cache information of the parameter information (i.e., key-value cache), and no quantization calibration is required, thereby achieving the technical effect of effectively compressing the key-value cache, reducing the video memory occupancy during the reasoning process, and reducing the total input / output of the graphics processor global memory to the shared memory / register in the decoding calculation while meeting the model accuracy, thereby solving the technical problems in the related technology of offline compression of key-value cache, complex quantization calibration steps, and model accuracy being affected by calibration data, resulting in low model accuracy.

[0207] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.

[0208] Example 4

[0209] According to an embodiment of the present application, a device embodiment for implementing the above information processing method is also provided. Figure 7 is a structural diagram of an information processing method and device according to Embodiment 4 of the present application, such as Figure 7 As shown, the device comprises:

[0210] An acquisition module 701 is used to acquire parameter information corresponding to a query request during a forward reasoning process of a target machine learning model, wherein the parameter information is obtained by encoding the query request;

[0211] The first processing module 702 is used to quantize and compress the parameter information according to a preset quantization method to obtain quantized compressed information;

[0212] The second processing module 703 is used to perform quantization compression processing on the parameter information using the quantization compression information to obtain cache information;

[0213] The reasoning module 704 is used to obtain the query result corresponding to the query request based on the cache information.

[0214] Optionally, the parameter information includes: key value information, and the acquisition module 701 is further used to: read the key value information corresponding to the query request from the first cache.

[0215] Optionally, it also includes: an encoding module, used to encode the query request to obtain an encoding result; perform linear mapping on the encoding result to obtain key value information; and store the key value information in the first cache.

[0216] Optionally, the preset quantization method includes: a symmetric quantization method, and the quantization compression information includes: scale information. The above-mentioned first processing module 702 is also used to: in the symmetric quantization method, perform quantization compression based on the key value information and the target number of bits to be quantized to obtain scale information.

[0217] Optionally, the key-value information includes: the key to be quantized and the value to be quantized. The above-mentioned second processing module 703 is also used to: according to the preset quantization compression granularity, use the scale information to perform quantization compression processing on the key to be quantized in a symmetric quantization method to obtain key cache information, and use the scale information to perform quantization compression processing on the value to be quantized to obtain value cache information.

[0218] Optionally, the preset quantization method includes: an asymmetric quantization method, and the quantization compression information includes: scale information and zero point position information. The above-mentioned first processing module 702 is also used for: in the asymmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain scale information and zero point position information.

[0219] Optionally, the key-value information includes: the key to be quantized and the value to be quantized. The above-mentioned second processing module 703 is also used to: according to the preset quantization compression granularity, use the scale information and the zero point position information in an asymmetric quantization method to perform quantization compression processing on the key to be quantized to obtain key cache information, and use the scale information and the zero point position information to perform quantization compression processing on the value to be quantized to obtain value cache information.

[0220] Optionally, the device further includes: a writing module, configured to write the cache information into the second cache.

[0221] Optionally, the above-mentioned reasoning module 704 is also used to: read cache information from the second cache, wherein the cache information includes: key cache information and value cache information; perform inverse quantization processing on the key cache information to obtain first inverse quantization information, and perform inverse quantization processing on the value cache information to obtain second inverse quantization information; obtain query results based on the query request, the first inverse quantization information and the second inverse quantization information.

[0222] Optionally, the above-mentioned reasoning module 704 is further used to: perform a multiplication operation on the query request and the first dequantization information to obtain a weight coefficient; and perform a multiplication operation on the second dequantization information and the weight coefficient to obtain a query result.

[0223] By using the embodiment of the present application, in the forward reasoning process of the target machine learning model, the query request is first encoded to obtain the parameter information corresponding to the query request, and then the parameter information is quantized and compressed according to a preset quantization method to obtain quantized compressed information, and then the quantized compressed information is used to perform quantized compression processing on the parameter information to obtain cache information, and finally the query result corresponding to the query request is obtained based on the cache information reasoning, thereby achieving the purpose of dynamically quantizing the parameter information (i.e., key-value information) and compressing the cache information of the parameter information (i.e., key-value cache), and no quantization calibration is required, thereby achieving the technical effect of effectively compressing the key-value cache under the premise of meeting the model accuracy, reducing the video memory occupancy during the reasoning process, and reducing the total input / output of the graphics processor global memory to the shared memory / register in the decoding calculation, thereby solving the technical problems in the related technology of offline compression of the key-value cache, complex quantization calibration steps, and the model accuracy being affected by the calibration data, resulting in low model accuracy.

[0224] It should be noted that the acquisition module 701, the first processing module 702, the second processing module 703 and the reasoning module 704 correspond to steps S21 to S24 in Example 1, and the four modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be run in the server 10 or server 10' provided in Example 1.

[0225] According to an embodiment of the present application, another device embodiment for implementing the above-mentioned information processing method is also provided. Figure 8 is a structural diagram of another information processing method and device according to Embodiment 4 of the present application, such as Figure 8 As shown, the device comprises:

[0226] An acquisition module 801 is used to acquire parameter information corresponding to an e-commerce service query request during the forward reasoning process of the e-commerce service machine learning model, wherein the parameter information is obtained by encoding the e-commerce service query request;

[0227] The first processing module 802 is used to quantize and compress the parameter information according to a preset quantization method to obtain quantized compressed information;

[0228] The second processing module 803 is used to perform quantization compression processing on the parameter information using the quantization compression information to obtain cache information;

[0229] The reasoning module 804 is used to obtain the query result corresponding to the e-commerce service query request based on the cache information.

[0230] By using the embodiment of the present application, in the forward reasoning process of the e-commerce service machine learning model, the parameter information corresponding to the e-commerce service query request is first obtained by encoding the e-commerce service query request, and then the parameter information is quantized and compressed according to a preset quantization method to obtain quantized compressed information, and then the quantized compressed information is used to perform quantized compression processing on the parameter information to obtain cache information, and finally, based on the cache information, the query result corresponding to the e-commerce service query request is obtained by reasoning, thereby achieving the purpose of dynamically quantizing parameter information (i.e., key-value information) and compressing the cache information of the parameter information (i.e., key-value cache), and no quantization calibration is required, thereby achieving the technical effect of effectively compressing the key-value cache, reducing the video memory occupancy during the reasoning process, and reducing the total input / output of the graphics processor global memory to the shared memory / register in the decoding calculation while meeting the model accuracy, thereby solving the technical problems in the related technology of offline compression of key-value cache, complex quantization calibration steps, and model accuracy being affected by calibration data, resulting in low model accuracy.

[0231] It should be noted that the acquisition module 801, the first processing module 802, the second processing module 803, and the reasoning module 804 correspond to steps S61 to S64 in Example 3, and the four modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 3. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be run in the server 10 or cloud server 10' provided in Example 1.

[0232] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0233] Example 5

[0234] The embodiment of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0235] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0236] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the information processing method: in the forward reasoning process of the target machine learning model, obtain parameter information corresponding to the query request, wherein the parameter information is obtained by encoding the query request; quantize and compress the parameter information according to a preset quantization method to obtain quantized compressed information; use the quantized compressed information to quantize and compress the parameter information to obtain cache information; and obtain the query result corresponding to the query request based on the cache information by reasoning.

[0237] Optionally, Fig. 9 is a structural block diagram of a computer terminal according to an embodiment of the present application. Fig. 9 As shown, the computer terminal 9 may include: one or more (only one is shown in the figure) processors 902, a memory 904, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0238] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the information processing method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the stored software programs and modules, that is, realizing the above-mentioned information processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 9 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0239] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: in the forward reasoning process of the target machine learning model, obtain parameter information corresponding to the query request, wherein the parameter information is obtained by encoding the query request; quantize and compress the parameter information according to a preset quantization method to obtain quantized compressed information; use the quantized compressed information to quantize and compress the parameter information to obtain cache information; and obtain the query result corresponding to the query request based on the cache information.

[0240] Optionally, the parameter information includes: key value information, and the processor may further execute program code of the following steps: reading the key value information corresponding to the query request from the first cache.

[0241] Optionally, the processor may also execute program codes of the following steps: encoding the query request to obtain an encoding result; linearly mapping the encoding result to obtain key value information; and storing the key value information in the first cache.

[0242] Optionally, the preset quantization method includes: a symmetric quantization method, and the quantization compression information includes: scale information. The above-mentioned processor can also execute the program code of the following steps: in the symmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain scale information.

[0243] Optionally, the key-value information includes: the key to be quantized and the value to be quantized. The above-mentioned processor can also execute the program code of the following steps: according to the preset quantization compression granularity, the scale information is used to perform quantization compression processing on the key to be quantized in a symmetric quantization method to obtain key cache information, and the scale information is used to perform quantization compression processing on the value to be quantized to obtain value cache information.

[0244] Optionally, the preset quantization method includes: an asymmetric quantization method, and the quantization compression information includes: scale information and zero point position information. The above-mentioned processor can also execute the program code of the following steps: in the asymmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain scale information and zero point position information.

[0245] Optionally, the key-value information includes: the key to be quantized and the value to be quantized. The above-mentioned processor can also execute the program code of the following steps: according to the preset quantization compression granularity, the key to be quantized is quantized and compressed using the scale information and the zero point position information in an asymmetric quantization method to obtain key cache information, and the value to be quantized is quantized and compressed using the scale information and the zero point position information to obtain value cache information.

[0246] Optionally, the processor may further execute program code of the following steps: writing cache information into the second cache.

[0247] Optionally, the processor may also execute the following program code: reading cache information from the second cache, wherein the cache information includes: key cache information and value cache information; performing inverse quantization processing on the key cache information to obtain first inverse quantization information, and performing inverse quantization processing on the value cache information to obtain second inverse quantization information; and obtaining query results based on the query request, the first inverse quantization information and the second inverse quantization information.

[0248] Optionally, the processor may also execute program codes of the following steps: multiplying the query request by the first inverse quantization information to obtain a weight coefficient; and multiplying the second inverse quantization information by the weight coefficient to obtain a query result.

[0249] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: in the forward reasoning process of the e-commerce service machine learning model, obtain parameter information corresponding to the e-commerce service query request, wherein the parameter information is obtained by encoding the e-commerce service query request; quantize and compress the parameter information according to a preset quantization method to obtain quantized compressed information; use the quantized compressed information to quantize and compress the parameter information to obtain cache information; and obtain the query result corresponding to the e-commerce service query request based on the cache information.

[0250] By using the embodiment of the present application, in the forward reasoning process of the e-commerce service machine learning model, the parameter information corresponding to the e-commerce service query request is first obtained by encoding the e-commerce service query request, and then the parameter information is quantized and compressed according to a preset quantization method to obtain quantized compressed information, and then the quantized compressed information is used to perform quantized compression processing on the parameter information to obtain cache information, and finally, based on the cache information, the query result corresponding to the e-commerce service query request is obtained by reasoning, thereby achieving the purpose of dynamically quantizing parameter information (i.e., key-value information) and compressing the cache information of the parameter information (i.e., key-value cache), and no quantization calibration is required, thereby achieving the technical effect of effectively compressing the key-value cache, reducing the video memory occupancy during the reasoning process, and reducing the total input / output of the graphics processor global memory to the shared memory / register in the decoding calculation while meeting the model accuracy, thereby solving the technical problems in the related technology of offline compression of key-value cache, complex quantization calibration steps, and model accuracy being affected by calibration data, resulting in low model accuracy.

[0251] It can be understood by those skilled in the art that Fig. 9 The structure shown is for illustration only, and the computer terminal 9 may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig. 9 The structure of the electronic device is not limited. For example, the computer terminal 9 may also include Fig. 9 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig. 9 Different configurations are shown.

[0252] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0253] Example 6

[0254] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the information processing method provided in the first embodiment.

[0255] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

[0256] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: in the forward reasoning process of the target machine learning model, obtaining parameter information corresponding to the query request, wherein the parameter information is obtained by encoding the query request; quantizing and compressing the parameter information according to a preset quantization method to obtain quantized compressed information; using the quantized compressed information to quantize and compress the parameter information to obtain cache information; and obtaining a query result corresponding to the query request by reasoning based on the cache information.

[0257] Optionally, the parameter information includes: key value information. In this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: reading the key value information corresponding to the query request from the first cache.

[0258] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: encoding the query request to obtain an encoding result; linearly mapping the encoding result to obtain key value information; and storing the key value information in a first cache.

[0259] Optionally, the preset quantization method includes: a symmetric quantization method, and the quantization compression information includes: scale information. In this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: in the symmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain scale information.

[0260] Optionally, the key-value information includes: a key to be quantized and a value to be quantized. In this embodiment, a computer-readable storage medium is configured to store program codes for executing the following steps: according to a preset quantization compression granularity, the key to be quantized is quantized and compressed using scale information in a symmetric quantization manner to obtain key cache information, and the value to be quantized is quantized and compressed using scale information to obtain value cache information.

[0261] Optionally, the preset quantization method includes: an asymmetric quantization method, and the quantization compression information includes: scale information and zero point position information. In this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: in an asymmetric quantization method, quantization compression is performed based on key value information and the target number of bits to be quantized to obtain scale information and zero point position information.

[0262] Optionally, the key-value information includes: a key to be quantized and a value to be quantized. In this embodiment, a computer-readable storage medium is configured to store program codes for executing the following steps: according to a preset quantization compression granularity, the key to be quantized is quantized and compressed using scale information and zero-point position information in an asymmetric quantization manner to obtain key cache information, and the value to be quantized is quantized and compressed using scale information and zero-point position information to obtain value cache information.

[0263] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: writing the cache information into the second cache.

[0264] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for executing the following steps: reading cache information from a second cache, wherein the cache information includes: key cache information and value cache information; performing inverse quantization processing on the key cache information to obtain first inverse quantization information, and performing inverse quantization processing on the value cache information to obtain second inverse quantization information; and obtaining a query result based on the query request, the first inverse quantization information, and the second inverse quantization information.

[0265] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: multiplying the query request with the first dequantization information to obtain a weight coefficient; multiplying the second dequantization information with the weight coefficient to obtain a query result.

[0266] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0267] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0268] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0269] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0270] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0271] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0272] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An information processing method, characterized in that: include: In the forward reasoning process of the target machine learning model, obtaining parameter information corresponding to the query request, wherein the parameter information is obtained by encoding the query request; Quantize and compress the parameter information according to a preset quantization method to obtain quantized compressed information; Using the quantized compression information to perform quantized compression processing on the parameter information to obtain cache information; A query result corresponding to the query request is obtained by inference based on the cache information.

2. The information processing method according to claim 1, characterized in that: The parameter information includes: key value information, and obtaining the parameter information corresponding to the query request includes: The key value information corresponding to the query request is read from the first cache.

3. The information processing method according to claim 2, characterized in that: The information processing method further includes: Encoding the query request to obtain an encoding result; Performing linear mapping on the encoding result to obtain the key value information; The key value information is stored in the first cache.

4. The information processing method according to claim 2, characterized in that: The preset quantization method includes: a symmetric quantization method, the quantization compression information includes: scale information, and the parameter information is quantized and compressed according to the preset quantization method, and the quantization compression information obtained includes: In the symmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain the scale information.

5. The information processing method according to claim 4, characterized in that: The key value information includes: a key to be quantized and a value to be quantized. The parameter information is quantized and compressed using the quantization compression information to obtain the cache information including: According to the preset quantization compression granularity, the key to be quantized is quantized and compressed using the scale information in the symmetric quantization method to obtain key cache information, and the value to be quantized is quantized and compressed using the scale information to obtain value cache information.

6. The information processing method according to claim 2, characterized in that: The preset quantization method includes: an asymmetric quantization method, the quantization compression information includes: scale information and zero point position information, and the parameter information is quantized and compressed according to the preset quantization method, and the quantization compression information obtained includes: In the asymmetric quantization method, quantization compression is performed based on the key value information and the target number of bits to be quantized to obtain the scale information and the zero point position information.

7. The information processing method according to claim 6, characterized in that: The key value information includes: a key to be quantized and a value to be quantized. The parameter information is quantized and compressed using the quantization compression information to obtain the cache information including: According to the preset quantization compression granularity, the scale information and the zero point position information are used in the asymmetric quantization method to perform quantization compression processing on the key to be quantized to obtain key cache information, and the scale information and the zero point position information are used to perform quantization compression processing on the value to be quantized to obtain value cache information.

8. The information processing method according to claim 1, characterized in that: The information processing method further includes: The cache information is written into the second cache.

9. The information processing method according to claim 8, characterized in that: The query result corresponding to the query request obtained by inference based on the cache information includes: Reading the cache information from the second cache, wherein the cache information includes: key cache information and value cache information; Performing a dequantization process on the key cache information to obtain first dequantization information, and performing a dequantization process on the value cache information to obtain second dequantization information; The query result is obtained based on the query request, the first dequantization information, and the second dequantization information.

10. The information processing method according to claim 9, characterized in that: Acquiring the query result based on the query request, the first dequantization information, and the second dequantization information includes: Performing a multiplication operation on the query request and the first dequantized information to obtain a weight coefficient; The second dequantization information is multiplied by the weight coefficient to obtain the query result.

11. A cloud server, characterized in that: include: A processor, a first cache and a second cache; The first cache is used to store parameter information corresponding to the query request, wherein the parameter information is obtained by encoding the query request; The processor is used to read the parameter information from the first cache during the forward reasoning process of the target machine learning model, quantize and compress the parameter information according to a preset quantization method to obtain quantized compressed information, and use the quantized compressed information to quantize and compress the parameter information to obtain cache information, wherein the cache information is used to infer the query result corresponding to the query request; The second cache is used to store the cache information.

12. The cloud server according to claim 11, characterized in that: The processor is also used to read the cache information from the second cache, wherein the cache information includes: key cache information and value cache information; perform inverse quantization processing on the key cache information to obtain first inverse quantization information, and perform inverse quantization processing on the value cache information to obtain second inverse quantization information; obtain the query result based on the query request, the first inverse quantization information and the second inverse quantization information.

13. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program, when running, executes the information processing method according to any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the information processing method according to any one of claims 1 to 10.

Images