Data processing method and device

Abstract

The data processing method comprises the steps that first KV data and second KV data are obtained, the first KV data and the second KV data are generated by conducting a full-amount reasoning process on input data through a machine learning model, and the first KV data and the second KV data are obtained based on different tokens, the first KV data and the second KV data are obtained by dividing KV data continuously generated by the machine learning model, and the division is performed through semantic similarity between tokens based on which the KV data are generated; constructing a first index according to the first KV data; constructing a second index according to the second KV data; and storing a first mapping relationship between the first index and the first KV data and a second mapping relationship between the second index and the second KV data. According to the method, the semantic consistency of the cache in the block can be improved, so that the retrieval precision of a subsequent incremental reasoning stage can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a data processing method and apparatus thereof. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] Today, large language models have achieved great success in general and various vertical fields, and are widely used in applications such as dialogue systems, document summarization and translation, code generation, and question-answering assistants. The ever-increasing expectations for model capabilities have led to greater attention being paid to the long-sequence capabilities of models. As a result, long-sequence large language models, implemented through techniques such as long text fine-tuning, positional encoding expansion, and improvements, have emerged.

[0004] However, the architectural characteristics of large language models dictate that their computational and memory consumption are quadratically related to the sequence length. With increasingly larger deployments and access volumes in real-world scenarios, this issue poses a severe challenge to the inference efficiency and computational timeliness of long-sequence large language models. To alleviate this problem, a common practice is to employ caching, a traditional optimization method called key-value caching, in the attention layer of large language models, thereby reducing the computational burden in autoregressive inference. However, the cache size occupied by the KV cache is still proportional to the sequence length and does not reduce the memory consumption of large language models. Especially in long-sequence inference, the memory occupied by the KV cache can easily exceed the model's parameter size, leading not only to significant memory overhead but also greatly increasing the system's input / output (I / O) time.

[0005] Against this backdrop, memory module methods based on key-value cache eviction retrieval have been proposed. These memory module methods can discard non-critical parts of the current cache while minimizing the impact of cache discarding. These methods do not require modification to the large language model itself, enabling plug-and-play application and thus applicability to almost all large language models. Most of these methods retrieve the top K cache blocks from the past key-value cache as key cache pairs by using the query value of the next token and the attention score obtained through a dot product with the key value.

[0006] However, these methods have shortcomings in retrieval accuracy. Summary of the Invention

[0007] In a first aspect, this application provides a data processing method, the method comprising: acquiring first key-value (KV) data and second key-value (KV) data, wherein the first KV data and the second KV data are generated by performing a full inference process on input data through a machine learning model, the first KV data and the second KV data are obtained based on different tokens, and the first KV data and the second KV data are obtained by dividing the KV data continuously generated by the machine learning model, the division being performed based on the semantic similarity between the tokens on which the KV data are generated; constructing a first index based on the first KV data; constructing a second index based on the second KV data; and storing a first mapping relationship between the first index and the first KV data, and a second mapping relationship between the second index and the second KV data.

[0008] In this embodiment of the application, when determining the cache block, the KV data is divided based on semantic similarity, which can improve the semantic consistency of the cache within the block, thereby improving the retrieval accuracy in the subsequent incremental inference stage.

[0009] In one possible implementation, the semantic similarity between tokens of different KV data in the first KV data satisfies a first preset condition, the semantic similarity between tokens of different KV data in the second KV data satisfies the first preset condition, and the semantic similarity between tokens of the first KV data and tokens of the second KV data does not satisfy the first preset condition.

[0010] In one possible implementation, the first target KV data is the KV data initially generated by the machine learning model in the second KV data. Based on the token corresponding to the first KV data, the token corresponding to the first target KV data can be predicted, obtaining the perplexity of the prediction result. When the perplexity abruptly changes compared to the perplexity when predicting the token corresponding to the first KV data, it is determined that the semantic similarity between the token of the first KV data and the token corresponding to the first target KV data does not satisfy the preset condition. That is, the first target KV data can be used as a block location (or adjusted subsequently based on this location) for block division, resulting in two blocks (i.e., cache blocks).

[0011] Based on the perplexity of the current sequence, it is segmented when the perplexity jumps significantly. It should be understood that after segmentation, the segmentation boundary can be fine-tuned based on the modulus of the similarity graph between tokens within the block.

[0012] Furthermore, regardless of the sequence partitioning method used, a correlation graph can be constructed for the intra-block cache after segmentation. Within this correlation graph, the modulus is maximized by adjusting the segment boundaries, thereby maximizing the semantic similarity of tokens within the block.

[0013] The full inference process can be a sliding window process. In one possible implementation, the token corresponding to the first KV data includes a first token. Based on the first token, multiple KV data that meet the second preset condition in semantic similarity with the first token can be selected from the generated KV data (that is, the data included in the time window for KV data inference of the first token). Based on the multiple KV data, the KV data of the first token is generated.

[0014] In one possible implementation, based on the Q vector of the first token, multiple KV data that satisfy the second preset condition in semantic similarity with the first token can be selected from the generated KV data by comparing their similarity with the K data in the generated KV data.

[0015] In one possible implementation, the method further includes:

[0016] During the incremental inference process of the first input data and the second input data on the incremental server, attention calculation is performed on the KV data required for incremental inference of the first input data and the KV data required for incremental inference of the second input data in the database. The retrieval and the attention calculation are performed in parallel. The first input data and the second input data are data from different requests.

[0017] In one possible implementation, the method further includes:

[0018] During the incremental inference process of the first input data and the second input data on the incremental server, the key-value data required for the incremental inference of the first input data is retrieved from the database, and the key-value data required for the incremental inference of the second input data is obtained from the database. The retrieval and the acquisition are performed in parallel. The first input data and the second input data are data from different requests.

[0019] In one possible implementation, the method further includes:

[0020] During the incremental inference process of the first input data and the second input data in the incremental server, attention calculation is performed on the key-value data required for incremental inference of the first input data and the key-value data required for incremental inference of the second input data obtained from the database. The acquisition and attention calculation are performed in parallel. The first input data and the second input data are data from different requests.

[0021] When multiple sequence tasks exist, insertion and retrieval can be implemented in a ping-pong pipeline for each split vector database instance, thereby improving the throughput of large language model inference.

[0022] In one possible implementation, the method further includes:

[0023] During the incremental inference process, attention operations are performed on the incremental inference of the second token based on the key-value data of the first token. The first token is the token predicted before the second token, and the key-value data of the first token is obtained by attention operations based on the Q data of the first token.

[0024] For the Decode phase, compared to the original model's inference process, the memory-based approach introduces additional inference latency: the retrieval and transmission of the KV cache for the current query. Therefore, this invention proposes a speculative and verification implementation scheme. It approximates the KV corresponding to the current query by using the KV corresponding to the query before n tokens, thus masking the latency bubbles caused by retrieval and transmission. Furthermore, to address the accuracy loss due to approximation, this invention uses delayed verification, i.e., verification with the actual KV passed after n tokens (which is also the approximate KV for the query after n tokens), thereby essentially maintaining the original accuracy of the model.

[0025] In one possible implementation, the method further includes:

[0026] Verify the key-value data corresponding to the first token;

[0027] If the verification fails, attention operation is performed on the incremental reasoning of the second token based on the KV data of the second token. The KV data of the second token is obtained by attention operation based on the Q data of the second token.

[0028] In existing solutions, using the current token query for retrieval creates data dependency, preventing the retrieval and computation processes from being parallelized. This application's embodiment can use historical queries (e.g., the previous token query) for retrieval, essentially pre-fetching key-value data and accelerating the inference process. However, this approach can lead to accuracy loss in situations with semantic shifts. By validating the key-value context corresponding to the actual query, a balance between speed and accuracy can be achieved.

[0029] In one possible implementation, the first key-value data includes second target key-value data and third target key-value data, wherein the second target key-value data and the third target key-value data are key-value data obtained from different network layers in the machine learning model, and storing the first mapping relationship between the first index and the first key-value data, and the second mapping relationship between the second index and the second key-value data, includes:

[0030] The first mapping relationship between the first index and the first KV data is stored on the first server, and the second mapping relationship between the second index and the second KV data is stored on the second server; wherein...

[0031] The first server and the second server are different servers responsible for incremental inference; or,

[0032] The first server is responsible for full inference, and the second server is responsible for incremental inference; or,

[0033] The second server is responsible for full inference, and the first server is responsible for incremental inference.

[0034] This application proposes an acceleration scheme that uses multiple servers as full servers, thereby increasing the system's batch size and the total throughput of large language models when prefill becomes a bottleneck.

[0035] Furthermore, cache block budgets can be dynamically allocated hierarchically. For example, more cache blocks can be allocated to shallow attention layers, forming an inverted pyramid-shaped cache block structure. Existing solutions retrieve the same number of cache blocks for each transformer layer during the decoding phase. This application's embodiments can leverage the attention characteristics of the transformer to increase the number of searches for shallower layers with more dispersed attention, and reduce searches for lower layers, thereby improving the utilization rate of the search budget.

[0036] In one possible implementation,

[0037] The step of constructing a first index based on the first KV data includes: constructing a first index based on the numerical relationships between multiple K data points included in the first KV data or the fusion of multiple K data points; or,

[0038] The step of constructing a second index based on the second KV data includes: constructing a second index based on the numerical relationship between multiple K data included in the second KV data or the fusion of multiple K data.

[0039] Existing solutions suffer from semantic loss when only a subset of queries is selected as the index, especially when cached blocks have multiple semantic meanings. In this application, an improved index construction method (incorporating information from each key-value pair during index construction) enhances recall during the retrieval phase.

[0040] Secondly, this application provides a data processing apparatus, the apparatus comprising:

[0041] The full-scale reasoning module is used to acquire first key-value (KV) data and second key-value (KV) data. The first KV data and the second KV data are generated by performing a full-scale reasoning process on the input data through a machine learning model. The first KV data and the second KV data are obtained based on different tokens. Furthermore, the first KV data and the second KV data are obtained by dividing the KV data continuously generated by the machine learning model. The division is performed based on the semantic similarity between the tokens on which the KV data is generated.

[0042] An index building module is used to build a first index based on the first KV data and to build a second index based on the second KV data.

[0043] The storage module is used to store the first mapping relationship between the first index and the first KV data, and the second mapping relationship between the second index and the second KV data.

[0044] In one possible implementation, the semantic similarity between tokens of different KV data in the first KV data satisfies a first preset condition, the semantic similarity between tokens of different KV data in the second KV data satisfies the first preset condition, and the semantic similarity between tokens of the first KV data and tokens of the second KV data does not satisfy the first preset condition.

[0045] In one possible implementation, the first target key-value data is the key-value data initially generated by the machine learning model in the second key-value data, and the full inference module is further used for:

[0046] Based on the token corresponding to the first KV data, predict the token corresponding to the first target KV data, and obtain the perplexity of the prediction result;

[0047] When the perplexity changes abruptly compared to the perplexity when predicting the token corresponding to the first KV data, it is determined that the semantic similarity between the token of the first KV data and the token corresponding to the first target KV data does not meet the preset condition.

[0048] In one possible implementation, the token corresponding to the first KV data includes a first token, and the full inference module is further configured to:

[0049] Based on the first token, select multiple key-value pairs from the generated key-value pairs that have a semantic similarity to the first token that meets the second preset condition;

[0050] Based on the multiple key-value data, generate the key-value data for the first token.

[0051] In one possible implementation, the full inference module is specifically used for:

[0052] Based on the Q vector of the first token, multiple KV data that meet the second preset condition in semantic similarity with the first token are selected from the generated KV data by comparing their similarity with the K data in the generated KV data.

[0053] In one possible implementation, the device further includes:

[0054] The incremental inference module is used to retrieve the key-value (KV) data required for incremental inference of the first input data and the KV data required for incremental inference of the second input data from the database during the incremental inference process of the first input data and the second input data in the incremental server, and to perform attention calculation on the KV data. The retrieval and the attention calculation are performed in parallel. The first input data and the second input data are data from different requests.

[0055] In one possible implementation, the incremental inference module is further configured to:

[0056] During the incremental inference process of the first input data and the second input data on the incremental server, the key-value data required for the incremental inference of the first input data is retrieved from the database, and the key-value data required for the incremental inference of the second input data is obtained from the database. The retrieval and the acquisition are performed in parallel. The first input data and the second input data are data from different requests.

[0057] In one possible implementation, the incremental inference module is further configured to:

[0058] During the incremental inference process of the first input data and the second input data in the incremental server, attention calculation is performed on the key-value data required for incremental inference of the first input data and the key-value data required for incremental inference of the second input data obtained from the database. The acquisition and attention calculation are performed in parallel. The first input data and the second input data are data from different requests.

[0059] In one possible implementation, the incremental inference module is further configured to:

[0060] During the incremental inference process, attention operations are performed on the incremental inference of the second token based on the key-value data of the first token. The first token is the token predicted before the second token, and the key-value data of the first token is obtained by attention operations based on the Q data of the first token.

[0061] In one possible implementation, the incremental inference module is further configured to:

[0062] Verify the key-value data corresponding to the first token;

[0063] If the verification fails, attention operation is performed on the incremental reasoning of the second token based on the KV data of the second token. The KV data of the second token is obtained by attention operation based on the Q data of the second token.

[0064] In one possible implementation, the first key-value data includes second target key-value data and third target key-value data, wherein the second target key-value data and the third target key-value data are key-value data obtained from different network layers in the machine learning model, and the storage module is specifically used for:

[0065] The first mapping relationship between the first index and the first KV data is stored on the first server, and the second mapping relationship between the second index and the second KV data is stored on the second server; wherein...

[0066] The first server and the second server are different servers responsible for incremental inference; or,

[0067] The first server is responsible for full inference, and the second server is responsible for incremental inference; or,

[0068] The second server is responsible for full inference, and the first server is responsible for incremental inference.

[0069] In one possible implementation,

[0070] The index building module is specifically used to: build a first index based on the numerical relationship between multiple K data included in the first KV data or the fusion of multiple K data; or to build a second index based on the numerical relationship between multiple K data included in the second KV data or the fusion of multiple K data.

[0071] Thirdly, embodiments of this application provide a data processing apparatus, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform the methods described in the first aspect above and any of its optional methods.

[0072] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect and any of its optional methods.

[0073] Fifthly, embodiments of this application provide a computer program that, when run on a computer, causes the computer to perform the first aspect and any of its optional methods described above.

[0074] Sixthly, this application provides a chip system including a processor for supporting an execution data processing device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device or training device. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0075] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0076] Figures 2 to 4 This is a schematic diagram of the application system framework of the present invention;

[0077] Figure 5 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0078] Figures 6 to 7J This application system framework is illustrated in the embodiments of this application.

[0079] Figure 8 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application;

[0080] Figure 9 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0081] Figure 10 A schematic diagram of the structure of the training device provided in the embodiments of this application;

[0082] Figure 11 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0083] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0084] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0085] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0086] The terms “substantially,” “about,” and similar terms used herein are used as approximations rather than as terms of degree, and are intended to take into account the inherent biases of measurements or calculations known to those skilled in the art. Furthermore, the use of “may” in describing embodiments of the invention refers to “one or more possible embodiments.” The terms “use,” “using,” and “used” used herein are to be considered synonymous with the terms “utilize,” “utilizing,” and “utilized,” respectively. Additionally, the term “exemplary” is intended to refer to an instance or illustration.

[0087] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0088] (1) Infrastructure

[0089] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0090] (2) Data

[0091] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, as well as IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0092] (3) Data processing

[0093] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0094] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data by symbolizing and formalizing it.

[0095] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0096] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0097] (4) General ability

[0098] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0099] (5) Smart Products and Industry Applications

[0100] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0101] The following is combined Figure 2 The system architecture provided in the embodiments of this application will be described in detail.

[0102] Figure 2 This is a schematic diagram of the system architecture provided for an embodiment of this application. Figure 2 As shown, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data acquisition system 560.

[0103] The execution device 510 includes a calculation module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The calculation module 511 may include a target model / rule 501, while the preprocessing modules 513 and 514 are optional.

[0104] The data acquisition device 560 is used to collect training samples. After collecting the training samples, the data acquisition device 560 stores these training samples in the database 530.

[0105] The training device 520 can maintain training samples in the database 530 to obtain the target model / rule 501 from the neural network to be trained (e.g., the machine learning model in the embodiments of this application).

[0106] It should be understood that the training device 520 can perform a pre-training process on the neural network to be trained based on the training samples maintained in the database 530, or fine-tune the model based on the pre-training.

[0107] It should be noted that in practical applications, the training samples maintained in database 530 may not all come from the data acquisition device 560; they may also be received from other devices. Furthermore, it should be noted that training device 520 may not necessarily train the target model / rule 501 entirely based on the training samples maintained in database 530; it may also obtain training samples from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0108] The target model / rule 501 trained using training device 520 can be applied to different systems or devices, such as... Figure 2 The execution device 510 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server, etc.

[0109] Specifically, the training device 520 can transfer the trained model to the execution device 510.

[0110] exist Figure 2 In the execution device 510, an input / output (I / O) interface 512 is configured for data interaction with external devices. Users can input data to the I / O interface 512 through the client device 540.

[0111] Preprocessing modules 513 and 514 are used to preprocess the input data received from the I / O interface 512. It should be understood that preprocessing modules 513 and 514 may be absent, or only one preprocessing module may be used. When preprocessing modules 513 and 514 are absent, the calculation module 511 can be used directly to process the input data.

[0112] During the preprocessing of input data by the execution device 510, or during the calculation module 511 of the execution device 510 performing calculations and other related processes, the execution device 510 can call data, code, etc. in the data storage system 550 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 550.

[0113] Finally, the I / O interface 512 provides the processing result to the client device 540, thereby providing it to the user.

[0114] exist Figure 2In the illustrated scenario, the user can manually provide input data, which can be done through the interface provided by I / O interface 512. Alternatively, the client device 540 can automatically send input data to I / O interface 512. If user authorization is required for the client device 540 to automatically send input data, the user can set the corresponding permissions in the client device 540. The user can view the output results of the execution device 510 on the client device 540, which can be presented in various forms such as display, sound, or animation. The client device 540 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530. Alternatively, data can be collected directly from the I / O interface 512 without going through the client device 540, using the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530.

[0115] It is worth noting that, Figure 2 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 2 In this context, the data storage system 550 is an external storage device relative to the execution device 510. However, in other cases, the data storage system 550 may also be placed within the execution device 510. It should be understood that the aforementioned execution device 510 may be deployed within the client device 540.

[0116] From the inference side of the model:

[0117] In this embodiment, the computing module 511 of the execution device 510 can obtain the code stored in the data storage system 550 to implement the steps related to the model reasoning process in this embodiment.

[0118] In this embodiment of the application, the computing module 511 of the execution device 510 may include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the training device 520 may be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0119] Specifically, the computing module 511 of the execution device 510 can be a hardware system with the function of executing instructions. The steps related to the model inference process provided in this application embodiment can be software code stored in the memory. The computing module 511 of the execution device 510 can obtain the software code from the memory and execute the obtained software code to implement the steps related to the model inference process provided in this application embodiment.

[0120] It should be understood that the computing module 511 of the execution device 510 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps related to the model reasoning process provided in the embodiments of this application can also be implemented by the hardware system in the computing module 511 of the execution device 510 without the function of executing instructions, which is not limited here.

[0121] From the training side of the model:

[0122] In this embodiment of the application, the training device 520 can access the memory ( Figure 2 (Not shown in the diagram, but can be integrated into the training device 520 or deployed separately from the training device 520) The code stored in the diagram can be used to implement the steps related to model training in the embodiments of this application.

[0123] In this embodiment of the application, the training device 520 may include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the training device 520 may be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0124] It should be understood that the training device 520 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps related to the training of the neutralization model provided in the embodiments of this application can also be implemented by the hardware system in the training device 520 without the function of executing instructions, which is not limited here.

[0125] In this embodiment, the forward propagation process of the model is involved, which can be executed by the execution device 510 or the training device 520 described in the above embodiments.

[0126] Furthermore, the execution device 510 or training device 520 can process the input data using a machine learning model. This machine learning model may include an attention layer that performs attention calculations on the input tokens. During these attention calculations, the attention layer can obtain intermediate results that can be reused in subsequent attention calculations on the same tokens. For example, these intermediate results could be key-value (KV) data. In this process, when processing a new token, reusable intermediate results can be stored in memory. These intermediate results can then be retrieved from memory and used as the basis for attention calculations on other tokens. However, the amount of reusable intermediate results that need to be stored grows rapidly as inference progresses, leading to a large storage requirement. Furthermore, excessively large intermediate results can severely slow down the inference process; therefore, compressing reusable intermediate results is crucial.

[0127] In one implementation, the compression process can be performed by a compression module, which can be centrally deployed with the execution device 510 or training device 520, for example, belonging to the same chip or other granular computing units, or it can be separately deployed, for example, belonging to different chips. For example, the execution device 510 or training device 520 can be an AI chip, and the compression module can belong to the CPU.

[0128] For example, refer to Figure 3 and Figure 4 , Figure 3 and Figure 4 The following are schematic diagrams of the architecture of embodiments of this application. The model running module can obtain intermediate results by running a machine learning model, the compression module can compress the intermediate results and write the compressed data into the memory, the compression module can read the compressed data from the storage area and decompress it to obtain the decompression result and transmit it to the model running module. Figure 3 The compression module and model execution module are deployed separately on different chips. Figure 4 The compression module and model execution module are centrally deployed on the same chip.

[0129] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0130] (1) Neural Network

[0131] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0132]

[0133] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0134] (2) A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer refers to the layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron can be connected to only some of the neurons in its neighboring layers. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way features are extracted being independent of their location. The convolutional kernel can be formalized as a matrix of random size, and during the training process of the CNN, the kernel can learn reasonable weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers in the CNN, while also reducing the risk of overfitting.

[0135] CNN is a very common type of neural network. As mentioned in the basic concept introduction above, a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture, which refers to learning at multiple levels of different abstraction using machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, in which each neuron can respond to the input image.

[0136] (3) Deep Neural Networks

[0137] Deep Neural Networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0138] In summary: the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows:

[0139] It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0140] (4) Loss Function

[0141] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0142] (5) Backpropagation algorithm

[0143] Backpropagation (BP) can be used during training to correct the parameters in the initial model, thereby reducing the model's error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters in the initial model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining optimal model parameters, such as the weight matrix.

[0144] (6) Large Language Model: A large language model is a natural language processing model trained on large-scale data, typically with billions or tens of billions of parameters. These models learn the general features of language by studying a large amount of text data during the pre-training stage, and can then be fine-tuned on downstream tasks to adapt to the needs of specific tasks.

[0145] (7) Transformer: The transformer is a deep learning model architecture originally used for sequence-to-sequence tasks, such as machine translation. It uses a self-attention mechanism to process input sequences and has achieved great success in the field of natural language processing. Most large language models, such as BERT, GPT, and T5, are based on the Transformer architecture.

[0146] (8) Key-Value Cache: A key-value cache is a cache structure that stores key-value pairs. In large language models, key-value caches are often used to store intermediate results or other useful information that the model is processing text in order to improve efficiency. By using a key-value cache, the model can avoid redundant calculations when processing text.

[0147] (9) Key-Value Cache Quantization: Key-value cache quantization refers to quantizing the values ​​in the key-value cache to reduce storage space and computational overhead. In some large language models, to adapt the model to limited resources, the values ​​in the key-value cache can be quantized to reduce the model's storage and computational costs.

[0148] (10) PPL (Perplexity): PPL is a metric used to evaluate the performance of a language model, representing the model's ability to predict a given text sequence. PPL is a positive real number, which can be understood as the average difficulty the model has in predicting the observed data sequence. The lower the PPL, the better the model performance.

[0149] (11) Token: In natural language processing, a "token" is the basic unit for segmenting a text string. This can be a word, a character, or a fragment of a word. Large language models typically need to segment the input text into tokens and then convert these tokens into numerical representations (such as word vectors) that the model can understand.

[0150] (12) Sequence: In the context of a large language model, "sequence" refers to a sequence of elements with a certain order relationship. Multiple tokens make up a sequence.

[0151] (13) Incremental Inference: Incremental inference allows the model to process only newly added parts of the input, rather than reprocessing the entire sequence each time. This is achieved by maintaining contextual information in the model's internal state, allowing the model to respond quickly when it receives new input. Incremental inference is particularly useful in interactive applications, such as chatbots or real-time translation, as it can significantly reduce latency and computational resource usage.

[0152] Today, large language models have achieved great success in general and various vertical fields, and are widely used in applications such as dialogue systems, document summarization and translation, code generation, and question-answering assistants. The ever-increasing expectations for model capabilities have led to greater attention being paid to the long-sequence capabilities of models. As a result, long-sequence large language models, implemented through techniques such as long text fine-tuning, positional encoding expansion, and improvements, have emerged.

[0153] However, the architectural characteristics of large language models dictate that their computational and memory consumption are quadratically related to the sequence length. With increasingly larger deployments and access volumes in real-world scenarios, this issue poses a severe challenge to the inference efficiency and computational timeliness of long-sequence large language models. To alleviate this problem, a common practice is to employ caching, a traditional optimization method called key-value caching, in the attention layer of large language models, thereby reducing the computational burden in autoregressive inference. However, the cache size occupied by the KV cache is still proportional to the sequence length and does not reduce the memory consumption of large language models. Especially in long-sequence inference, the memory occupied by the KV cache can easily exceed the model's parameter size, leading not only to significant memory overhead but also greatly increasing the system's input / output (I / O) time.

[0154] Against this backdrop, memory module methods based on key-value cache eviction retrieval have been proposed. These memory module methods can discard non-critical parts of the current cache while minimizing the impact of cache discarding. These methods do not require modification to the large language model itself, enabling plug-and-play application and thus applicability to almost all large language models. Most of these methods retrieve the top K cache blocks from the past key-value cache as key cache pairs by using the query value of the next token and the attention score obtained through a dot product with the key value.

[0155] However, these methods have shortcomings in retrieval accuracy.

[0156] This application provides a data processing method. The data processing method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0157] Reference Figure 5 , Figure 5 This is a flowchart illustrating a data processing method provided in an embodiment of this application, such as... Figure 5 As shown in the embodiment of this application, a data processing method may include steps 501 to 504, which are described in detail below.

[0158] 501. Obtain first KV data and second KV data, wherein the first KV data and the second KV data are generated by a full inference process on the input data through a machine learning model, the first KV data and the second KV data are obtained based on different tokens, and the first KV data and the second KV data are obtained by dividing the KV data continuously generated by the machine learning model, wherein the division is performed by the semantic similarity between the tokens on which the KV data are based;

[0159] The full-data server (also known as the server used for full-data inference) performs full-data inference on the input data to obtain key-value (KV) data for each input data. The machine learning model may include an attention layer. When the attention layer performs attention calculations on the tokens, it obtains intermediate results (i.e., the first and second KV data) that need to be reused in subsequent attention calculations on the tokens. For example, these intermediate results can be KV data (also called key and value data) or KV pairs.

[0160] When performing full inference, the full inference server obtains the key-value data of each token in the input data and stores the key-value data. During storage, the key-value data can be divided into multiple key-value units, and each key-value unit can calculate the corresponding index.

[0161] For example, the server can receive a user request, combine the request with the contextual knowledge base, apply the system prompt, and then pass it to the full inference server. The full server begins the full inference prefill phase. It performs window-by-window inference using a sliding window prefill sequence and generates the corresponding KV cache(K,V), where K = {k...} i |i=0,1,…,n}V={V i |i=0,1,…,n}. When the key-value data of the token in the local window satisfies the block segmentation condition (for example, the first key-value data is a block that satisfies the block segmentation condition, the second key-value data is a block that satisfies the block segmentation condition, and a block can also be called a cache block, and each cache block can be used as a unit to build the corresponding index), a vector representation vector will be constructed for the cache block. (Also known as an index), and the cache block index - index vector (K) block V block The vector database D and cache block data inserted into the incremental server are passed to the incremental server host memory M. decodeHost Where f(·) is the index representation vector construction method. After prefilling is complete, the full server predicts the first token:

[0162]

[0163] And the task context (window) M context =(I,D[g(W) q X,D),L) are transmitted to the incremental server. Where I represents the initial token of the sequence and the corresponding key-value pairs of all transformer layers, X is the embedding input corresponding to the last token, and W... q In the attention mechanism, the Query matrix is ​​defined as follows: g(·) represents the query operation in the vector database, obtaining the cache block index corresponding to the query. D[·] represents obtaining the KV cache corresponding to the index in the Host memory; L is the local context, representing the most recent part of the tokens in the sequence and the KV cache.

[0164] In one possible implementation, the semantic similarity between tokens of different KV data in the first KV data satisfies a first preset condition, the semantic similarity between tokens of different KV data in the second KV data satisfies the first preset condition, and the semantic similarity between tokens of the first KV data and tokens of the second KV data does not satisfy the first preset condition.

[0165] For example, the first preset condition is: semantic similarity is greater than a threshold.

[0166] In this embodiment of the application, when determining the cache block, the KV data is divided based on semantic similarity, which can improve the semantic consistency of the cache within the block, thereby improving the retrieval accuracy in the subsequent incremental inference stage.

[0167] In one possible implementation, the first target KV data is the KV data initially generated by the machine learning model in the second KV data. Based on the token corresponding to the first KV data, the token corresponding to the first target KV data can be predicted, obtaining the perplexity of the prediction result. When the perplexity abruptly changes compared to the perplexity when predicting the token corresponding to the first KV data, it is determined that the semantic similarity between the token of the first KV data and the token corresponding to the first target KV data does not satisfy the preset condition. That is, the first target KV data can be used as a block location (or adjusted subsequently based on this location) for block division, resulting in two blocks (i.e., cache blocks).

[0168] Based on the perplexity of the current sequence, it is segmented when the perplexity jumps significantly. It should be understood that after segmentation, the segmentation boundary can be fine-tuned based on the modulus of the similarity graph between tokens within the block.

[0169] Furthermore, regardless of the sequence partitioning method used, a correlation graph can be constructed for the intra-block cache after segmentation. Within this correlation graph, the modulus is maximized by adjusting the segment boundaries, thereby maximizing the semantic similarity of tokens within the block.

[0170] For example, if the next token in the current sequence undergoes a semantic shift, then using the current sequence to predict the next token will result in a high degree of perplexity for the correct answer. Therefore, this can be used to determine whether to perform sequence segmentation. The specific steps are as follows: In the prefill stage, after inputting a sequence, the final logits corresponding to the last token are obtained. i =Attention(W Q X,W K X,W V X). At this time, the entire server will generate a KV cache (W). Q X,W K X) is transmitted to the incremental server. The next token t in the prefetch sequence. i+1 The t here i+1 This represents the token ID after the token is encoded. The conditional probability of this token can be calculated in the following way in this embodiment:

[0171] p(t i+1|t0t1…t i ;θ)=softmax(Linear(logit i ))[t i+1 ];

[0172] The perplexity can be calculated using the following formula:

[0173] In the implementation, p(t) can also be used directly. i+1 |t0t1…t i ;θ) is used as a segmentation index. It can maintain the sequence moving average of the surprise index, and the mutation point exceeding α times the mean is used as the segmentation point.

[0174] The full inference process can be a sliding window process. In one possible implementation, the token corresponding to the first KV data includes a first token. Based on the first token, multiple KV data that meet the second preset condition in semantic similarity with the first token can be selected from the generated KV data (that is, the data included in the time window for KV data inference of the first token). Based on the multiple KV data, the KV data of the first token is generated.

[0175] In one possible implementation, based on the Q vector of the first token, multiple KV data that satisfy the second preset condition in semantic similarity with the first token can be selected from the generated KV data by comparing their similarity with the K data in the generated KV data.

[0176] Specifically, during the prefill phase, the sliding window is implemented as follows during the full server inference process: The model prefill context is divided into context = (I, E, L), where I is the first token of the sequence, whose value remains constant throughout the single task; E is the portion of the sliding window that needs to unload the KV cache, which changes as the sequence is updated; and L is the local window, which always contains the latest suffix of the sequence. Before inference, the number of tokens, num, to slide in each iteration needs to be set. evict .

[0177] The most common and fastest implementation is to stream new tokens into the key-value pair and then unload the old key-value cache in the order they entered the context window. The steps are as follows:

[0178] When the context window length is less than the maximum window size, inference with the maximum window size is performed normally; when the NPU / GPU memory for inference deployment is insufficient, inference can also be performed in blocks: Next, NumPy will be evicted. evict An old token. This can be used to uninstall the previous num from E.evict One token, then

[0179] Let the current sequence have completed reasoning for the first k tokens, and then add the new num. evict Each token is added to the end of the context, i.e., Concatenate{E,L} -i =Sequence[k+i],for i∈{1,…,num evict}

[0180] For LLM inference, for a large model with causal decoding, the input attention mask is:

[0181] AttentionMask = {a ij}i=1,…,num evict j = 1, ..., k

[0182]

[0183] Where 1 represents the valid part of the attention matrix, and 0 represents the part assigned -inf. In this example, the mask can be:

[0184] 1 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1

[0185] For the newly added KV cache after calculation, if it meets the block criterion, the update method of the memory module is called to construct the index vector and insertion vector database, and the KV buffer is transferred to the incremental server. All of the above operations are asynchronous with the large language model inference, and the model returns to step 3 until the prefill phase is completed.

[0186] In scenarios where inference accuracy is critical, a Topk eviction approach can be used. This approach doesn't prioritize queries entering the context window in order, but instead uses the last query as a representative vector. The old KV cache is then unloaded based on its qk score, which determines the relevance of the KV cache in E to the semantics of the current sequence. The steps are as follows:

[0187] When the context window length is less than the maximum window size, inference with the maximum window size is performed normally; when the NPU / GPU memory for inference deployment is insufficient, inference can also be performed in blocks.

[0188] Next, num will be expelled. evict The old token will be unloaded from the previous num in E. evict For each token, the last q is cached for all K pairs. e ={k i Find the qk scores for |i∈E+L} and select the lowest num.evict Each token constitutes the set of indices to be evicted, Index. evict ={i|i∈topK({-qk)} j |for j∈K e},num evict )}.

[0189] Then Where |concate{E,L}| represents the total size of sets E and L. If a positional encoding recalculation scheme is used at this point, sets E and L need to be sorted according to the sequence order of the tokens.

[0190] Let the current sequence have completed reasoning for the first k tokens, and then add the new num. evict Each token is added to the end of the context, i.e., Concatenate{E,L} -i =Sequence[k+i],for i∈{1,…,num evict}. For large models using causal decoding, the input attention mask during LLM inference is:

[0191] AttentionMask = {a ij}i=1,…,num evict j = 1, ..., k

[0192]

[0193] Where 1 represents the valid portion of the attention matrix, and 0 represents the portion assigned -inf. For example, the mask could be:

[0194]

[0195] For newly added KV cache after calculation, if the block criterion is met, the update method of the memory module is called to build the index vector and insertion vector database, and the KV buffer is transferred to the incremental server.

[0196] 502. Based on the first KV data, construct the first index;

[0197] 503. Based on the second KV data, construct the second index;

[0198] In one possible implementation, the embedding vector of each cache block (e.g., first KV data, second KV data, etc.) is calculated and stored in a vector database using that vector as an index.

[0199] In one possible implementation, a first index can be constructed based on the numerical relationships between multiple K data points included in the first KV data or the fusion of multiple K data points.

[0200] In one possible implementation, a second index can be constructed based on the numerical relationships between multiple K data included in the second KV data or the fusion of multiple K data.

[0201] For the index representation vector construction method required to store cache blocks into the vector database, the following two implementations can be exemplified:

[0202] Option 1: Perform a pooling operation on the query vector within the block.

[0203] In this approach, the scheme assumes that the semantic importance of all key-value caches corresponding to all tokens is similar. This is particularly applicable to the proposed scheme of segmenting by sequence astonishment. The specific vector construction method is as follows:

[0204] For the input X of the attention layer i ,i=1,…,n, where n is the block size, to obtain K i =W k X i For the index of this layer:

[0205]

[0206] Option 2: Select the minimum value at each position within the buffer block to construct a vector, and select the maximum value to construct another vector. These two vectors can be used as indices for the buffer block.

[0207] Existing solutions suffer from semantic loss when only a subset of queries is selected as the index, especially when cached blocks have multiple semantic meanings. In this application, an improved index construction method (incorporating information from each key-value pair during index construction) enhances recall during the retrieval phase.

[0208] 504. Store the first mapping relationship between the first index and the first KV data, and the second mapping relationship between the second index and the second KV data.

[0209] For the storage and management of KV cache, depending on the size of the large language model sequence window used, there are two possible implementations:

[0210] Option 1: Do not recalculate the position encoding; the cache blocks in the vector database contain position information.

[0211] Taking the decoding stage as an example, if no recalculation is performed, the attention process in the model inference flow is as follows:

[0212] For each transformer layer, calculate the QKV corresponding to the current token:

[0213] Q = W Q X

[0214] K = W K X

[0215] V = W V X

[0216] Where X is the token embedding or the latent variable input of the previous layer.

[0217] If ROPE rotational position encoding is used, position encoding information is injected into QKV here; for absolute position encoding, no calculation is required.

[0218] The key-value pair corresponding to the token is submitted to the memory module for insertion. At this point, the memory module begins asynchronous tasks such as index vector construction, key-value buffer storage, and vector database insertion. The key-value pair (Q) is passed to the memory module to request the corresponding key-value pair. The key-value cache in the HBM is updated to the key-value pair value retrieved from the buffer by the memory module. The core attention calculation process then begins, calculating the attention score.

[0219]

[0220] The final logits are obtained or passed to the hidden variables of the next level.

[0221] Option 2: Recalculate the position encoding; the cache blocks in the vector database do not contain position information.

[0222] Reference Figure 6 The embodiments of this application can realize PD separation based on memory module scheme in the scenario where the full and incremental servers are each a single machine, thereby improving the utilization rate of each component and increasing the batch size to improve throughput.

[0223] Reference Figure 7A , Figure 7A This is a schematic diagram of an application architecture according to an embodiment of this application.

[0224] In one possible implementation, the first key-value data includes second target key-value data and third target key-value data, wherein the second target key-value data and the third target key-value data are key-value data obtained from different network layers in the machine learning model. A first mapping relationship between the first index and the first key-value data can be stored on a first server, and a second mapping relationship between the second index and the second key-value data can be stored on a second server. The first server and the second server are different servers responsible for incremental inference; or, the first server is a server responsible for full inference, and the second server is a server responsible for incremental inference; or, the second server is a server responsible for full inference, and the first server is a server responsible for incremental inference.

[0225] Compared to the full-inference scheme with separate PD (Programmable and Data Encryption) operations, the distributed KV (Key-Value) vector database scheme in this application makes full use of the CPU computing resources of each server. For an architecture with multiple full-inference servers and decoding servers, this invention proposes dividing the insertion and retrieval tasks of vector data by layer, thereby fully utilizing the CPU computing resources of all servers.

[0226] For example, full inference prefill service can be provided by one or more servers containing NPUs and CPUs, providing partial vector database computation load in layers; incremental inference decoding service can be provided by one or more servers containing NPUs and CPUs, providing partial vector database computation load in layers, as well as KV Cache Block storage.

[0227] Existing solutions only decouple the prefill and decode stages, thus providing corresponding computing resources for the different load requirements of the two stages. However, they do not consider further management of the KV cache, and the memory and computing consumption for long sequence inference is not reduced. The embodiments of this application integrate KV memory module management into PD separation, thereby achieving a high degree of integration of the two technologies. In addition, this solution addresses the characteristics of computation and bandwidth bottlenecks in prefill and decode, respectively, by designing a distributed vector database, effectively utilizing the CPU computing power of multiple servers and masking the latency caused by database queries and insertions.

[0228] For example, during incremental inference, the incremental server can use the appropriate retrieval scheme to obtain the required query vector and query the corresponding cache index in the vector database. Use cache index to retrieve cache blocks in the host's memory. Load HBM. Perform normal Transformer calculations. Similar to (c), unload and store the cached block for the predicted token until the predicted token is...<eos>.

[0229] Reference Figure 7B In multi-tasking scenarios, a distributed vector database based on language model hierarchical partitioning is used. When the entire server employs multiple GPUs / clusters and utilizes multi-GPU acceleration solutions such as Ring attention, the embodiments of this application can be implemented in the following ways:

[0230] Alternately split the Transformer layers (for example, this can be applied to scenarios where block-wise attention is used in parallel, such as Ring attention).

[0231] Let the number of full servers be k (numbered 0,…,k-1), and the vector databases corresponding to the full servers be denoted as VDB0,…,VDB. k-1 The current layer is layer=0, and the total number of layers is layermax (numbered 0, ..., layermax-1). The calculation process here takes the Ringattention scenario as an example.

[0232] For all servers, calculate their respective local QKV:

[0233] Q bi =W bi X

[0234] K bi =W bi X

[0235] V bi =W bi X

[0236] Let the current target server be denoted as target = layer mod k. After calculation, the key-value pairs are asynchronously transmitted to the host memory of the full target server and merged into:

[0237]

[0238] Create an index vector and insert V into VDB. target Simultaneously, all servers calculate their respective local attention and the required values ​​for online softmax; local attention is merged according to the original multi-GPU scheme, and the global attention value is calculated. After the calculation is completed, the linear layer FFN is calculated to obtain X. output .

[0239] X output As input to the lower layer:

[0240] layer = layer + 1

[0241] X = X output

[0242] If it is already the last layer, layer = layermax, then set layer = 0 and X = nextToken;

[0243] Segmented splitting based on Transformer (suitable for multi-GPU / multi-server parallel splitting models).

[0244] Let the number of full servers be k (numbered 0,…,k-1), and the vector databases corresponding to the full servers be denoted as VDB0,…,VDB. k-1 The current model has 0 layers, a maximum number of layers (layermax), 0 shards, and a total of k shards, which is the same as the number of servers / number of computing cards.

[0245] For the current layer, calculate QKV across all servers:

[0246] Let the current target server be denoted as target = layer mod k. After calculation, the key-value pairs are asynchronously transferred to the host memory of the full target server, and an index vector is created and inserted into the VDB. k .

[0247] The remaining attention component and linear FFN are calculated using the normal method to obtain X. output .

[0248] X output As input to the lower layer:

[0249] layer = layer + 1

[0250] X = X output

[0251] If it is already the last level of the current segment Then set layer = 0, and set the hidden variable X... output Transmit to the next full server and return to a). If it is already the last full server, then the prefilling of this token ends, return to the first full server, and let X = nextToken;

[0252] This application proposes an acceleration scheme that uses multiple servers as full servers, thereby increasing the system's batch size and the total throughput of large language models when prefill becomes a bottleneck.

[0253] Furthermore, cache block budgets can be dynamically allocated hierarchically. For example, more cache blocks can be allocated to shallow attention layers, forming an inverted pyramid-shaped cache block structure. Existing solutions retrieve the same number of cache blocks for each transformer layer during the decoding phase. This application's embodiments can leverage the attention characteristics of the transformer to increase the number of searches for shallower layers with more dispersed attention, and reduce searches for lower layers, thereby improving the utilization rate of the search budget.

[0254] Reference Figure 7C , Figure 7C To illustrate a scenario where multiple full servers operate in parallel under multi-tasking conditions, num rank The size is related to the incremental server memory size, the ratio of full and incremental server NPU computing power, and the nature of the sequence task. Here, we take num as an example. rank Taking a time ratio of 4 as an example, the results are A, B, C, and D respectively. Here, we take a 3:1 ratio of prefill to decode task time as an example. For both full and incremental servers, the pipelining of prefill and decode under multi-tasking is explained in detail below:

[0255] Formalize the task sequence Q to be inferred task ={Task1,Task2,…,}, the sequence to be decoded Vector database group VDBs = {VDB A VDB B VDB C VDB D '}、Memory group pair Mems={Mem A Mem B Mem C Mem D Each vector database is divided into multiple Ranks based on the total number of servers, i.e., VDB in this example. Group ={VDB1,VDB2,VDB3},Group=A,B,C,D,Available Vector Database - Memory Pairs IDLE={A,B,C,D};

[0256] Reference Figures 7D to 7F , Figures 7D to 7F This is a flowchart illustrating an embodiment of this application.

[0257] All server worker threads:

[0258] The server checks for available vector database memory pairs and ensures the task queue is not empty. If the conditions are not met, the system will hibernate and wait for new tasks to arrive, the vector database to be accessed, or memory pairs to be released before waking up. If the conditions are met, resources and tasks will be acquired.

[0259] Resource = front(IDLE)

[0260]

[0261] Mem = Mems[Resource]

[0262] The full server task begins, as described in the aforementioned technical solution process. This service thread will process the Task sequence, and the KV buffer will be transferred to Mem in the selected incremental server. Based on task distribution rules, such as hierarchical round-robin distribution, it is determined whether to insert the index into the vector database Rank i. That is, the index is inserted into the VDB in the incremental server. i The insertion and transmission processes are asynchronous with the server's prefill process.

[0263] After the server prefill is complete, the build will be transferred to the incremental server's Decode task:

[0264] Task decode =(Task, I, L, Resource)

[0265] Where I represents the beginning of the context window, i.e., the start of the sequence; L represents the local window, which is the end of the context window, i.e., the end of the sequence. The sizes of I and L are defined within the frame.

[0266] This worker thread is released and returns to step a). Simultaneously, a child thread is created to asynchronously wait for all asynchronous transfer and insertion processes in step b) to complete. Once insertion is complete, the constructed decode task is inserted into the decode task queue.

[0267] Push_back(Q decode Task decode )

[0268] After insertion, attempt to wake up the incremental server thread.

[0269] Incremental server worker threads:

[0270] The server checks that the Decode task queue is not empty. If the conditions are not met, the system will hibernate and wait to be awakened when a new Decode task arrives. If the conditions are met, resources and tasks will be acquired.

[0271] Task decode =front(Q Decode )

[0272] Task, I, L, Resource = Task decode

[0273] PopFront(Q Decode )

[0274] VDB = VDBs[Resource]

[0275] Mem = Mems[Resource]

[0276] Based on the vector database distribution rules, calculate the server i where the Rank of this VDB query resides. For example, in this case, it is obtained by i = layer mod 3. (This is done through the VDB...) i In the process, the query corresponding to the current hidden layer vector is used to retrieve the relevant KV buffer index, and the corresponding KV is obtained from Mem and loaded into HBM to construct the initial context window for the Decode stage, i.e.:

[0277] M context =(I,Mem[g(W q X,VDB)],L)

[0278] The aforementioned technical solution is used for inference and decoding to obtain the output sequence Output.

[0279] Inference <eos>That is, after all inference is completed, the VDB and Mem corresponding to the current sequence are released, the HBM occupied in the NPU / GPU is unloaded, and all server worker threads are woken up after the resources are released.

[0280] Delete VDB Mem M context

[0281] insert(IDLE,Resource)

[0282] By using the above methods, the utilization rate of each component participating in prefilling by multiple servers can be further improved in multi-tasking scenarios, thereby increasing the system throughput.

[0283] In one possible implementation, during the incremental inference process of the first input data and the second input data on the incremental server, attention calculation can be performed by retrieving the key-value data required for incremental inference of the first input data and the key-value data required for incremental inference of the second input data from the database. The retrieval and the attention calculation are performed in parallel. The first input data and the second input data are data from different requests.

[0284] In one possible implementation, during the incremental inference process of the first input data and the second input data on the incremental server, the key-value data required for the incremental inference of the first input data can be retrieved from the database, and the key-value data required for the incremental inference of the second input data can be obtained from the database. The retrieval and the acquisition are performed in parallel. The first input data and the second input data are data from different requests.

[0285] In one possible implementation, during the incremental inference process of the first input data and the second input data on the incremental server, attention calculation can be performed on the KV data required for incremental inference of the first input data and the KV data required for incremental inference of the second input data obtained from the database. The acquisition and attention calculation are performed in parallel. The first input data and the second input data are data from different requests.

[0286] When multiple sequence tasks exist, insertion and retrieval can be implemented in a ping-pong pipeline for each split vector database instance, thereby improving the throughput of large language model inference.

[0287] For example, it can be done based on the following process:

[0288] 1. When (multiple) servers receive a user request, they combine the request with the context knowledge base, apply the system prompt, and then pass it to the full server. The label of this sequence is α.

[0289] 2. (Multiple) full servers begin the prefill phase. Window-by-window inference is performed using a sliding window prefill sequence, generating corresponding KV caches (K,V), where K = {k...} i |i=0,1,…,n}V={V i |i=0,1,…,n}.

[0290] 3. When the token of a local window meets the block segmentation condition, a vector representation vector will be constructed for the cache block. And cache block index-index vector (K) block V block ) For the vector database D inserted into the incremental server α Cache block data is transferred to the incremental server Host memory M decodeHost Where f(·) is the method for constructing index representation vectors.

[0291] 4. After prefilling is complete, all servers predict the first token:

[0292]

[0293] 5. Set the task context (window) The requests are then forwarded to the incremental server. For ease of explanation, we number the user requests arriving at the full server as 0, ..., m-1 in order of arrival. Where I... i The sequence consists of the initial token and the corresponding key-value pairs (KV) of all transformer layers, X. i W is the embedding input for the last token. q Let D be the Query matrix in the attention mechanism, where g(·) represents the query operation in the vector database, obtaining the cache block index corresponding to the query. i [·] indicates that the key-value cache corresponding to the index is obtained in the host memory; L i This is the local context, representing the most recent portion of the tokens in the sequence and the key-value cache. After prefilling is complete, all servers begin prefilling tasks for other task sequences.

[0294] 6. The incremental server uses the corresponding retrieval scheme to obtain the required query vector and queries the corresponding cache index in the vector database. At this point, the sequence number i+1mod m is being calculated.

[0295] 7. Use cache indexes to retrieve cache blocks from the host's memory. Load HBM. At this point, sequence number i+1 mod m is performing a vector database search.

[0296] 8. Perform normal Transformer calculations. Similar to (c), unload and store the cached block of the token from the prediction phase. At this time, the i+1mod m sequence is transmitting the retrieved cached block.

[0297] Repeat steps 5 through 8 until the predicted token is... <eos>.

[0298] In detail, in a multi-tasking scenario, this application proposes a two-part pipeline: pipelined prefilling and decoding for the full server and incremental server under multi-tasking conditions; and pipelined retrieval and computation of multiple sequence tasks within the incremental server. Under the requirement of pipelined operation, the vector database and KV cache memory in the incremental server are expanded into a vector database group and a cache block group, with a size of num. rank indivual.

[0299] num rank The size is related to the incremental server memory size, the ratio of full and incremental server NPU computing power, and the nature of the sequence task. Here, we take num as an example. rank For example, if the value is 4, the corresponding numbers are A, B, C, and D.

[0300] The following are two examples of streamlined processes:

[0301] Streamlining Method 1:

[0302] For full and incremental servers, the pipelining of prefill and decode under multi-tasking is explained in detail below:

[0303] Formalize the task sequence Q to be inferred task ={Task1,Task2,…,}, the sequence to be decoded Vector database group VDBs = {VDB A VDB B VDB C VDB D '}、Memory group pair Mems={Mem A Mem B Mem C Mem D }, can be used with vector database-memory pairs IDLE = {A, B, C, D};

[0304] Resource = front(IDLE)

[0305] Task = front(Q task )

[0306] PopFront(Q task )

[0307] PopFront(IDLE)

[0308] VDB = VDBs[Resource]

[0309] Mem = Mems[Resource]

[0310] Full server worker threads: The server checks for available vector database-memory pairs and ensures the task queue is not empty. If the conditions are not met, the system will hibernate and wait for new tasks to arrive, the vector database to be accessed, or memory pairs to be released before waking up. If the conditions are met, resources and tasks will be acquired.

[0311] The full server task begins, as described in the aforementioned technical solution process. This service thread will process the Task sequence, transfer the KV buffer to the Mem in the selected incremental server, and insert the index into the VDB in the incremental server. The insertion and transfer processes are asynchronous to the server prefill process.

[0312] After the server prefill is complete, the build will be transferred to the Decode task on both servers:

[0313] Task decode =(Task, I, L, Resource)

[0314] Where I represents the beginning of the context window, i.e., the start of the sequence; L represents the local window, which is the end of the context window, i.e., the end of the sequence. The sizes of I and L are defined within the frame.

[0315] This worker thread is released and returns to step a). Simultaneously, a child thread is created to asynchronously wait for all asynchronous transfer and insertion processes in step b) to complete. Once insertion is complete, the constructed decode task is inserted into the decode task queue.

[0316]

[0317] After insertion, attempt to wake up the incremental server thread.

[0318] Incremental server worker threads:

[0319] The server checks that the Decode task queue is not empty. If the conditions are not met, the system will hibernate and wait to be awakened when a new Decode task arrives. If the conditions are met, resources and tasks will be acquired.

[0320] Task decode =front(Q Decode )

[0321] Task, I, L, Resource = Task decode

[0322] PopFront(Q Decode )

[0323] VDB = VDBs[Resource]

[0324] Mem = Mems[Resource]

[0325] By querying the relevant key-value buffer index in the VDB using the query corresponding to the current hidden layer vector, retrieving the corresponding key-value pair from the Mem, and loading it into the HBM, the initial context window for the Decode stage is constructed, i.e.:

[0326] M context =(I,Mem[g(W q X,VDB)],L)

[0327] The aforementioned technical solution is used for inference and decoding to obtain the output sequence Output.

[0328] Inference <eos>That is, after all inference is completed, the VDB and Mem corresponding to the current sequence are released, the HBM occupied in the NPU / GPU is unloaded, and all server worker threads are woken up after the resources are released.

[0329] Delete VDB Mem M context

[0330] insert(IDLE,Resource)

[0331] In this diagram, the incremental server has four vector databases and a key-value cache memory, and three parallel worker threads based on the NPU's computing power. The full server has one worker thread. In the first scenario, the full server is performing the prefill task for sequence 4, inserting an index into vector database D and transferring the key-value cache to memory block D via cross-machine communication. Meanwhile, sequences 1-3, which have already been prefilled, are being decoded on the incremental server. The three worker threads independently search vector databases A, B, and C, and read memory blocks A, B, and C based on the retrieved sequence numbers.

[0332] Next, the prefill task for sequence 4 is completed, and all the data required for decoding is transferred to the vector database D and memory D. Sequence 1 is completed on the incremental server, so the vector database A and memory A resources are released. The full server then acquires the free resource A and begins the prefill task for sequence 5, inserting the key-value pairs of sequence 5 into memory A and the indexes into the vector database A. Simultaneously, the incremental worker thread that completed sequence 1 also acquires the relevant resource D for sequence 4 and begins the decoding task, retrieving data from the vector database D and reading data from memory D.

[0333] Streamlined process 2.

[0334] For pipelined processing within the incremental server, the main focus is on addressing the memory-PCIe bandwidth bottleneck, as well as potential computational bottlenecks in single-card multi-tasking scenarios. We can break down the decode task within the incremental server into the following steps:

[0335]

[0336] Since insertion operations are minimal in the incremental server's decoding process, their impact can be ignored. In a multi-GPU / NPU environment, the CPU and GPU / NPU loads can accommodate multiple threads simultaneously, while the PCIe bus and memory access speeds can easily become bottlenecks. Under these conditions, the several worker threads of this invention will form the following pipeline:

[0337]

[0338] In one possible implementation, during the incremental inference process, attention operations can be performed on the incremental inference of the second token based on the key-value data of the first token. The first token is the token predicted before the second token, and the key-value data of the first token is obtained by attention operations based on the Q data of the first token.

[0339] In one possible implementation, the key-value data corresponding to the first token can be verified; if the verification fails, attention operation for incremental reasoning of the second token is performed based on the key-value data of the second token, wherein the key-value data of the second token is obtained by attention operation based on the Q data of the second token.

[0340] For the Decode phase, compared to the original model's inference process, the memory-based approach introduces additional inference latency: the retrieval and transmission of the KV cache for the current query. Therefore, this invention proposes a speculative and verification implementation scheme. It approximates the KV corresponding to the current query by using the KV corresponding to the query before n tokens, thus masking the latency bubbles caused by retrieval and transmission. Furthermore, to address the accuracy loss due to approximation, this invention uses delayed verification, i.e., verification with the actual KV passed after n tokens (which is also the approximate KV for the query after n tokens), thereby essentially maintaining the original accuracy of the model.

[0341] In existing solutions, using the current token query for retrieval creates data dependency, preventing the retrieval and computation processes from being parallelized. This application's embodiment can use historical queries (e.g., the previous token query) for retrieval, essentially pre-fetching key-value data and accelerating the inference process. However, this approach can lead to accuracy loss in situations with semantic shifts. By validating the key-value context corresponding to the actual query, a balance between speed and accuracy can be achieved.

[0342] For example, once prefilling is complete, the full server can predict the first token.

[0343]

[0344] 1. And set the task context (window) M context =(I,D[g(W) q The IL in [X,D),L) is passed to the incremental server. Here, I represents the initial token of the sequence and the corresponding key-value pairs of all transformer layers, X is the embedding input corresponding to the last token, and W... q In the attention mechanism, the Query matrix is ​​defined as follows: g(·) represents the query operation in the vector database, obtaining the cache block index corresponding to the query. D[·] represents obtaining the KV cache corresponding to the index in the Host memory; L is the local context, representing the most recent part of the tokens in the sequence and the KV cache.

[0345] 2. The incremental server uses the corresponding retrieval scheme to obtain the required query vector, and asynchronously queries the corresponding cache index in the vector database. The query is cached in the host memory after completion. The retrieved index will be used after the S token; the current token actually uses the query result before the S token. After starting the query, proceed to the next step.

[0346] 3. After the asynchronous query is completed, the cached block D[g(q,D)] in the host memory is asynchronously loaded into HBM using the retrieved cache index, and is used after the S token is obtained. When in the initial Decode stage, we do not have a cached D[g(q,D)], so we need to wait for asynchronous steps (2) and (3) to complete. If a cache already exists, then according to the scheme of this application embodiment, the cache is used... That is, D[g(q,D)] before the S-step token, or D[g(q,D)] of the first token in the current token group.

[0347] 4. Perform normal Transformer calculations. Similar to (3), unload and store the cached block of the token during the prediction phase. However, at this time, if If the token is still needed for subsequent use, it will only be evicted from the context window of this model, but will still attempt to remain in HBM. If there is insufficient video memory, it can be unloaded to host memory for later loading.

[0348] Repeat steps (2) to (4) until the predicted token is... <eos>.

[0349] Reference Figure 7F , Figure 7F This is a flowchart.

[0350] For the verification part, taking S=1, i.e., the speculative retrieval step, as an example, the first step is to perform S steps of normal model inference. The detailed steps are explained below:

[0351] Entering the Decode phase, let t denote the currently decoded token (the nth token). n .

[0352] Calculate QKV, asynchronously using Q to query K from the vector database. n V n Insert a new k n v n Using K at the same time n-1 V n-1 Calculate the current attention. Obtain the next token, and calculate t. n At the same time, using K n-1 V n-1 The attention matrix calculation results mask t n The attention mask at the location is calculated. Verification t n-1 If the decode result is successfully verified, then t will be processed normally. n Add to sequence, enter t n+1 The prediction, otherwise use the new one. Replace t n-1 Recalculate t n .

[0353] The verification section includes the following schematic diagrams:

[0354] Option 1: Verify the proportion of attention

[0355] This scheme checks the hit portion of speculative retrieval during the attention calculation phase and verifies whether the sum of the attention scores in the (true) key-value (KV) attention matrix reaches a threshold. attn :

[0356]

[0357] Where qK represents the true q and the K obtained in the verification, and i is the intersection of the K obtained in the speculation and the verification respectively. This formula requires that the KV pair obtained in the speculation covers the vast majority of the attention threshold in the verification. attn ).

[0358] Option 2: Verify the probability distribution of the vocabulary

[0359] This approach only validates the logits output by the model, which has low computational cost and aims to ensure that the vocabulary probabilities output by the model under speculative retrieval are similar to the probability distribution under the real key-value pairs. A pre-set threshold is given. KL ,Require:

[0360]

[0361] Where p(·) represents the final vocabulary probability of the model under the true key-value pairs. The final vocabulary probability of the KV model for speculative retrieval

[0362] Option 3: Establish consistent top-k coverage of word probabilities

[0363] This scheme only performs validation on the final logits output by the model, resulting in low computational cost and making it more suitable for topk / topk-p decoding and sampling methods. It requires that the coverage of the final topk tokens exceed a threshold. topk

[0364]

[0365] The foregoing embodiments proposed schemes using speculative retrieval and using a single head for speculative retrieval. This embodiment combines these with existing speculative reasoning methods, specifically including:

[0366] 1. Perform prefilling as described in the previous embodiments to process the KV Cache.

[0367] 2. During the prefill process, type analysis is performed on each head.

[0368] 3. Perform attention relationship analysis and grouping on the attention between different layers and heads. For KVCache management, add a new attention head type.

[0369] 4. After entering the decoding stage, a lightweight model is used for speculative inference. [For example, a small model using the same KV Cache management pattern] [For example, using streamingllm] to obtain the next N candidate tokens. During the speculative model's inference, the sparsity and confidence of the attention corresponding to each token are analyzed to determine whether the model should perform KVCache Retrieve (optional, evict & retrieve can be skipped and attention & FFN can be performed directly).

[0370] 5. If a KV cache retrieval is required, the calculation should be based on the currently set transmission capacity, the attention mode of each head group, and N.

[0371] The attributes of each token, and the transmission capacity allocated to each group for each token.

[0372] 6. The main model decodes the N words. Simultaneously, the speculative inference for the next batch of tokens in step 6 and the speculative retrieval in step 7 begin asynchronously. If the main model does not accept the decoding results of the N words, i.e., the speculative inference fails, then the incorrect word...

[0373] The i-th word uses the prediction results of the main model, and the i+1 word starts from step 6.

[0374] 7. Continue until all sequence reasoning is completed and the main model is verified.

[0375] For detailed procedures, please refer to... Figure 7G As shown.

[0376] In the foregoing embodiments, the context window proposed by the present invention consists of the following three parts:

[0377] I,Mem[g(W q X,VDB)],L

[0378] Where I represents the key-value pairs (KV) corresponding to the tokens in the first few blocks of the sequence, and L represents the dynamic KV at the end of the sequence. This part corresponds to Type A attention mode, which is directly maintained in HBM memory. For example, refer to... Figure 7H As shown:

[0379] Mem[g(W q [X,VDB)] corresponds to dynamically unloaded and loaded block-based key-value pairs, which correspond to block-based attention patterns, such as those mentioned above. Figure 7I As shown:

[0380] The aforementioned approach is essentially a form of attention sparsity, which has significantly reduced the required HBM space and greatly improved inference speed compared to full attention.

[0381] In this embodiment, the invention extends the sparsity of attention to the entire prefill and decode stages, and improves the inference speed across all stages by optimizing the grouping of attention heads. Additionally, this invention considers another attention pattern: the diagonal-vertical line type. A diagram of this attention pattern is shown below. Figure 7J As shown:

[0382] In this context, vertical lines often represent semantic key points or abrupt changes in a text sequence, while diagonal lines typically summarize preceding paragraphs. It's worth noting that this attention pattern can be covered by block-based attention patterns, but the computational cost and bandwidth of block-based attention are limited by architecture and hardware. This invention, in long sequence scenarios, leverages the continuity of the diagonal-vertical attention pattern to estimate the diagonal-vertical attention across the entire sequence by analyzing the attention patterns of the last few tokens. Specifically:

[0383] For diagonal attention points, this invention accelerates the calculation process using the following steps:

[0384] 1. Calculate the attention matrix for the last few tokens using the last few queries (lastq queries).

[0385]

[0386] 2. In this lastq-row attention matrix, calculate the top k columns by summing the summations. v Column, summed along a 45° diagonal, top k s Diagonal line:

[0387]

[0388] 3. Using the obtained index vertical index slash This is extended to the complete attention matrix, and the relevant KVcache is loaded.

[0389] Mask sv = BUILD_VERTICAL_SLASH(index) vertial ,index slash )

[0390] 4. Call the vertical slash related operators of flash attention to calculate sparse attention:

[0391]

[0392] Note that different attention head types require different transmission capacity. Type A attention heads, except for prefill which requires a certain amount of bandwidth and decoding which requires a constant level of bandwidth, always maintain the required context within the context window. The other two types of attention require unloading and retrieval operations within this invention framework for maintenance, thus occupying a certain amount of transmission capacity. Furthermore, at the same token level, shallower attention layers require more capacity, while deeper layers require less.

[0393] The formalized capacity of each transmission is as follows:

[0394] max F(C B C S ;θ)

[0395] stC A =Constant P else 1If In prefill Stage

[0396] C A +C B +C S ≤C H2D C A C B C S ≥0

[0397] This means that for a specific model and parameters θ, the bandwidth allocation of the model for different attention heads needs to be optimized through experimental evaluation index F.

[0398] Reference Figure 8 , Figure 8 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application, such as... Figure 8 As shown in the embodiment of this application, a data processing apparatus 800 is provided, the apparatus comprising:

[0399] The full-scale reasoning module 801 is used to acquire first KV data and second KV data. The first KV data and the second KV data are generated by a full-scale reasoning process on the input data through a machine learning model. The first KV data and the second KV data are obtained based on different tokens. The first KV data and the second KV data are obtained by dividing the KV data continuously generated by the machine learning model. The division is based on the semantic similarity between the tokens on which the KV data is generated.

[0400] For a detailed description of the full inference module 801, please refer to the description of step 501 in the above embodiments, which will not be repeated here.

[0401] The index building module 802 is used to build a first index based on the first KV data and to build a second index based on the second KV data.

[0402] For a detailed description of the index building module 802, please refer to the description of step 502 in the above embodiment, which will not be repeated here.

[0403] Storage module 803 is used to store a first mapping relationship between the first index and the first KV data, and a second mapping relationship between the second index and the second KV data.

[0404] For a detailed description of the storage module 803, please refer to the description of step 503 in the above embodiments, which will not be repeated here.

[0405] In one possible implementation, the semantic similarity between tokens of different KV data in the first KV data satisfies a first preset condition, the semantic similarity between tokens of different KV data in the second KV data satisfies the first preset condition, and the semantic similarity between tokens of the first KV data and tokens of the second KV data does not satisfy the first preset condition.

[0406] In one possible implementation, the first target key-value data is the key-value data initially generated by the machine learning model in the second key-value data, and the full inference module is further used for:

[0407] Based on the token corresponding to the first KV data, predict the token corresponding to the first target KV data, and obtain the perplexity of the prediction result;

[0408] When the perplexity changes abruptly compared to the perplexity when predicting the token corresponding to the first KV data, it is determined that the semantic similarity between the token of the first KV data and the token corresponding to the first target KV data does not meet the preset condition.

[0409] In one possible implementation, the token corresponding to the first KV data includes a first token, and the full inference module is further configured to:

[0410] Based on the first token, select multiple key-value pairs from the generated key-value pairs that have a semantic similarity to the first token that meets the second preset condition;

[0411] Based on the multiple key-value data, generate the key-value data for the first token.

[0412] In one possible implementation, the full inference module is specifically used for:

[0413] Based on the Q vector of the first token, multiple KV data that meet the second preset condition in semantic similarity with the first token are selected from the generated KV data by comparing their similarity with the K data in the generated KV data.

[0414] In one possible implementation, the device further includes:

[0415] The incremental inference module is used to retrieve the key-value (KV) data required for incremental inference of the first input data and the KV data required for incremental inference of the second input data from the database during the incremental inference process of the first input data and the second input data in the incremental server, and to perform attention calculation on the KV data. The retrieval and the attention calculation are performed in parallel. The first input data and the second input data are data from different requests.

[0416] In one possible implementation, the incremental inference module is further configured to:

[0417] During the incremental inference process of the first input data and the second input data on the incremental server, the key-value data required for the incremental inference of the first input data is retrieved from the database, and the key-value data required for the incremental inference of the second input data is obtained from the database. The retrieval and the acquisition are performed in parallel. The first input data and the second input data are data from different requests.

[0418] In one possible implementation, the incremental inference module is further configured to:

[0419] During the incremental inference process of the first input data and the second input data in the incremental server, attention calculation is performed on the key-value data required for incremental inference of the first input data and the key-value data required for incremental inference of the second input data obtained from the database. The acquisition and attention calculation are performed in parallel. The first input data and the second input data are data from different requests.

[0420] In one possible implementation, the incremental inference module is further configured to:

[0421] During the incremental inference process, attention operations are performed on the incremental inference of the second token based on the key-value data of the first token. The first token is the token predicted before the second token, and the key-value data of the first token is obtained by attention operations based on the Q data of the first token.

[0422] In one possible implementation, the incremental inference module is further configured to:

[0423] Verify the key-value data corresponding to the first token;

[0424] If the verification fails, attention operation is performed on the incremental reasoning of the second token based on the KV data of the second token. The KV data of the second token is obtained by attention operation based on the Q data of the second token.

[0425] In one possible implementation, the first key-value data includes second target key-value data and third target key-value data, wherein the second target key-value data and the third target key-value data are key-value data obtained from different network layers in the machine learning model, and the storage module is specifically used for:

[0426] The first mapping relationship between the first index and the first KV data is stored on the first server, and the second mapping relationship between the second index and the second KV data is stored on the second server; wherein...

[0427] The first server and the second server are different servers responsible for incremental inference; or,

[0428] The first server is responsible for full inference, and the second server is responsible for incremental inference; or,

[0429] The second server is responsible for full inference, and the first server is responsible for incremental inference.

[0430] In one possible implementation,

[0431] The index building module is specifically used to: build a first index based on the numerical relationship between multiple K data included in the first KV data or the fusion of multiple K data; or to build a second index based on the numerical relationship between multiple K data included in the second KV data or the fusion of multiple K data.

[0432] The following describes a terminal device provided in an embodiment of this application. Please refer to [link to relevant documentation]. Figure 9 , Figure 9 This is a schematic diagram of a terminal device provided in an embodiment of this application. The terminal device 900 can specifically be a virtual reality (VR) device, a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., and is not limited thereto. Specifically, the terminal device 900 includes: a receiver 901, a transmitter 902, a processor 903, and a memory 904 (wherein the terminal device 900 may have one or more processors 903). Figure 9 (Taking a processor as an example), processor 903 may include application processor 9031 and communication processor 9032. In some embodiments of this application, receiver 901, transmitter 902, processor 903 and memory 904 may be connected via a bus or other means.

[0433] Memory 904 may include read-only memory and random access memory, and provides instructions and data to processor 903. A portion of memory 904 may also include non-volatile random access memory (NVRAM). Memory 904 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0434] Processor 903 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus but also power buses, control buses, and status signal buses. However, for clarity, all buses in the diagram are referred to as the bus system.

[0435] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 903. The processor 903 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 903 or by instructions in software form. The processor 903 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 903 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 904, and processor 903 reads the information from memory 904 and, in conjunction with its hardware, completes the steps of the above method.

[0436] Receiver 901 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 902 can be used to output digital or character information through the first interface; transmitter 902 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 902 may also include a display device such as a display screen.

[0437] This application also provides a server; please refer to [link / reference]. Figure 10 , Figure 10 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1000 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1010 (e.g., one or more processors) and memory 1032, and one or more storage media 1030 (e.g., one or more mass storage devices) for storing application programs 1042 or data 1044. The memory 1032 and storage media 1030 can be temporary or persistent storage. The program stored in the storage media 1030 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 1010 may be configured to communicate with the storage media 1030 and execute the series of instruction operations in the storage media 1030 on the server 1000.

[0438] Server 1000 may also include one or more power supplies 1026, one or more wired or wireless network interfaces 1050, one or more input / output interfaces 1058; or, one or more operating systems 1041, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0439] In this embodiment, the central processing unit 1010 is used to execute the actions described in the above embodiments.

[0440] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0441] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0442] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0443] For details, please refer to Figure 11 , Figure 11 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1100. The NPU 1100 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1103, which is controlled by the controller 1104 to extract matrix data from the memory and perform multiplication operations.

[0444] In some implementations, the arithmetic circuit 1103 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1103 is a two-dimensional pulsating array. The arithmetic circuit 1103 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1103 is a general-purpose matrix processor.

[0445] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1102 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1101 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1108.

[0446] Unified memory 1106 is used to store input and output data. Weight data is directly transferred to weight memory 1102 via Direct Memory Access Controller (DMAC) 1105. Input data is also transferred to unified memory 1106 via DMAC.

[0447] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1109.

[0448] The Bus Interface Unit (BIU) 1110 is used by the instruction fetch memory 1109 to fetch instructions from external memory, and also by the memory access controller 1105 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0449] The DMAC is mainly used to move input data from external memory DDR to unified memory 1106, or to weight data to weight memory 1102, or to input data to input memory 1101.

[0450] The vector computation unit 1107 includes multiple processing units that, when needed, further process the output of the computation circuit 1103, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0451] In some implementations, vector computation unit 1107 can store the processed output vector in unified memory 1106. For example, vector computation unit 1107 can apply a linear function, or a nonlinear function, to the output of computation circuit 1103, such as linear interpolation of feature planes extracted by convolutional layers, or, for example, a vector of accumulated values, to generate activation values. In some implementations, vector computation unit 1107 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to computation circuit 1103, for example, for use in subsequent layers of the neural network.

[0452] The instruction fetch buffer 1109 connected to the controller 1104 is used to store the instructions used by the controller 1104;

[0453] Unified memory 1106, input memory 1101, weight memory 1102, and instruction fetch memory 1109 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0454] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0455] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0456] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0457] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0458] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).< / eos> < / eos> < / eos> < / eos> < / eos>

Claims

1. A data processing method, characterized in that, The method includes: First KV data and second KV data are obtained. The first KV data and second KV data are generated by a machine learning model through a full inference process on the input data. The first KV data and second KV data are obtained based on different tokens. The first KV data and second KV data are obtained by dividing the KV data continuously generated by the machine learning model. The division is performed by the semantic similarity between the tokens on which the generation is based. Based on the first KV data, construct the first index; Based on the second KV data, construct a second index; The system stores a first mapping relationship between the first index and the first KV data, and a second mapping relationship between the second index and the second KV data.

2. The method according to claim 1, characterized in that, The semantic similarity between tokens of different KV data in the first KV data satisfies the first preset condition, the semantic similarity between tokens of different KV data in the second KV data satisfies the first preset condition, and the semantic similarity between tokens of the first KV data and tokens of the second KV data does not satisfy the first preset condition.

3. The method according to claim 1 or 2, characterized in that, The first target key-value data is the key-value data initially generated by the machine learning model in the second key-value data, and the method further includes: Based on the token corresponding to the first KV data, predict the token corresponding to the first target KV data, and obtain the perplexity of the prediction result; When the perplexity changes abruptly compared to the perplexity when predicting the token corresponding to the first KV data, it is determined that the semantic similarity between the token of the first KV data and the token corresponding to the first target KV data does not meet the preset condition.

4. The method according to any one of claims 1 to 3, characterized in that, The token corresponding to the first KV data includes the first token, and the method further includes: Based on the first token, select multiple key-value pairs from the generated key-value pairs that have a semantic similarity to the first token that meets the second preset condition; Based on the multiple key-value data, generate the key-value data for the first token.

5. The method according to any one of claims 1 to 3, characterized in that, The step of selecting multiple key-value data points from the generated key-value data that satisfy a second preset condition in semantic similarity to the first token, based on the first token, includes: Based on the Q vector of the first token, multiple KV data that meet the second preset condition in semantic similarity with the first token are selected from the generated KV data by comparing their similarity with the K data in the generated KV data.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: During the incremental inference process of the first input data and the second input data on the incremental server, attention calculation is performed on the KV data required for incremental inference of the first input data and the KV data required for incremental inference of the second input data in the database. The retrieval and the attention calculation are performed in parallel. The first input data and the second input data are data from different requests.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: During the incremental inference process of the first input data and the second input data on the incremental server, the key-value data required for the incremental inference of the first input data is retrieved from the database, and the key-value data required for the incremental inference of the second input data is obtained from the database. The retrieval and the acquisition are performed in parallel. The first input data and the second input data are data from different requests.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: During the incremental inference process of the first input data and the second input data in the incremental server, attention calculation is performed on the key-value data required for incremental inference of the first input data and the key-value data required for incremental inference of the second input data obtained from the database. The acquisition and attention calculation are performed in parallel. The first input data and the second input data are data from different requests.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: During the incremental inference process, attention operations are performed on the incremental inference of the second token based on the key-value data of the first token. The first token is the token predicted before the second token, and the key-value data of the first token is obtained by attention operations based on the Q data of the first token.

10. The method according to claim 9, characterized in that, The method further includes: Verify the key-value data corresponding to the first token; If the verification fails, attention operation is performed on the incremental reasoning of the second token based on the KV data of the second token. The KV data of the second token is obtained by attention operation based on the Q data of the second token.

11. The method according to any one of claims 1 to 10, characterized in that, The first key-value data includes second target key-value data and third target key-value data, wherein the second target key-value data and the third target key-value data are key-value data obtained from different network layers in the machine learning model. The storage of the first mapping relationship between the first index and the first key-value data, and the second mapping relationship between the second index and the second key-value data, includes: The first mapping relationship between the first index and the first KV data is stored on the first server, and the second mapping relationship between the second index and the second KV data is stored on the second server; wherein... The first server and the second server are different servers responsible for incremental inference; or, The first server is responsible for full inference, and the second server is responsible for incremental inference; or, The second server is responsible for full inference, and the first server is responsible for incremental inference.

12. The method according to any one of claims 1 to 11, characterized in that, The step of constructing a first index based on the first KV data includes: constructing a first index based on the numerical relationships between multiple K data points included in the first KV data or the fusion of multiple K data points; or, The step of constructing a second index based on the second KV data includes: constructing a second index based on the numerical relationship between multiple K data included in the second KV data or the fusion of multiple K data.

13. A data processing apparatus, characterized in that, The device includes: The full-scale reasoning module is used to acquire first key-value (KV) data and second key-value (KV) data. The first KV data and the second KV data are generated by performing a full-scale reasoning process on the input data through a machine learning model. The first KV data and the second KV data are obtained based on different tokens. Furthermore, the first KV data and the second KV data are obtained by dividing the KV data continuously generated by the machine learning model. The division is performed based on the semantic similarity between the tokens on which the KV data is generated. An index building module is used to build a first index based on the first KV data and to build a second index based on the second KV data. The storage module is used to store the first mapping relationship between the first index and the first KV data, and the second mapping relationship between the second index and the second KV data.

14. The apparatus according to claim 13, characterized in that, The semantic similarity between tokens of different KV data in the first KV data satisfies the first preset condition, the semantic similarity between tokens of different KV data in the second KV data satisfies the first preset condition, and the semantic similarity between tokens of the first KV data and tokens of the second KV data does not satisfy the first preset condition.

15. The apparatus according to claim 13 or 14, characterized in that, The first target key-value data is the key-value data initially generated by the machine learning model in the second key-value data. The full inference module is also used for: Based on the token corresponding to the first KV data, predict the token corresponding to the first target KV data, and obtain the perplexity of the prediction result; When the perplexity changes abruptly compared to the perplexity when predicting the token corresponding to the first KV data, it is determined that the semantic similarity between the token of the first KV data and the token corresponding to the first target KV data does not meet the preset condition.

16. The apparatus according to any one of claims 13 to 15, characterized in that, The token corresponding to the first KV data includes the first token. The full inference module is further used for: Based on the first token, select multiple key-value pairs from the generated key-value pairs that have a semantic similarity to the first token that meets the second preset condition; Based on the multiple key-value data, generate the key-value data for the first token.

17. The apparatus according to any one of claims 13 to 16, characterized in that, The full-scale inference module is specifically used for: Based on the Q vector of the first token, multiple KV data that meet the second preset condition in semantic similarity with the first token are selected from the generated KV data by comparing their similarity with the K data in the generated KV data.

18. The apparatus according to any one of claims 13 to 17, characterized in that, The device further includes: The incremental inference module is used to retrieve the key-value (KV) data required for incremental inference of the first input data and the KV data required for incremental inference of the second input data from the database during the incremental inference process of the first input data and the second input data in the incremental server, and to perform attention calculation on the KV data. The retrieval and the attention calculation are performed in parallel. The first input data and the second input data are data from different requests.

19. The apparatus according to any one of claims 13 to 18, characterized in that, The incremental inference module is also used for: During the incremental inference process of the first input data and the second input data on the incremental server, the key-value data required for the incremental inference of the first input data is retrieved from the database, and the key-value data required for the incremental inference of the second input data is obtained from the database. The retrieval and the acquisition are performed in parallel. The first input data and the second input data are data from different requests.

20. The apparatus according to any one of claims 13 to 19, characterized in that, The incremental inference module is also used for: During the incremental inference process of the first input data and the second input data in the incremental server, attention calculation is performed on the key-value data required for incremental inference of the first input data and the key-value data required for incremental inference of the second input data obtained from the database. The acquisition and attention calculation are performed in parallel. The first input data and the second input data are data from different requests.

21. The apparatus according to any one of claims 13 to 20, characterized in that, The incremental inference module is also used for: During the incremental inference process, attention operations are performed on the incremental inference of the second token based on the key-value data of the first token. The first token is the token predicted before the second token, and the key-value data of the first token is obtained by attention operations based on the Q data of the first token.

22. The apparatus according to claim 21, characterized in that, The incremental inference module is also used for: Verify the key-value data corresponding to the first token; If the verification fails, attention operation is performed on the incremental reasoning of the second token based on the KV data of the second token. The KV data of the second token is obtained by attention operation based on the Q data of the second token.

23. The apparatus according to any one of claims 13 to 22, characterized in that, The first key-value data includes second target key-value data and third target key-value data. The second target key-value data and the third target key-value data are key-value data obtained from different network layers in the machine learning model. The storage module is specifically used for: The first mapping relationship between the first index and the first KV data is stored on the first server, and the second mapping relationship between the second index and the second KV data is stored on the second server; wherein... The first server and the second server are different servers responsible for incremental inference; or, The first server is responsible for full inference, and the second server is responsible for incremental inference; or, The second server is responsible for full inference, and the first server is responsible for incremental inference.

24. The apparatus according to any one of claims 13 to 23, characterized in that, The index building module is specifically used to: build a first index based on the numerical relationship between multiple K data included in the first KV data or the fusion of multiple K data; or to build a second index based on the numerical relationship between multiple K data included in the second KV data or the fusion of multiple K data.

25. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions, which, when executed by one or more computers or processors, cause the one or more computers or processors to perform the method of any one of claims 1 to 12.

26. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on a computer device or processor, cause the computer device or processor to perform the method as described in any one of claims 1 to 12.

27. A system comprising at least one processor and at least one memory; The at least one processor and the at least one memory are connected via a communication bus; The at least one memory is used to store code; The at least one processor is used to execute the code to perform the method as described in any one of claims 1 to 12.

28. A chip, comprising a processor, characterized in that, The processor is used to support the data processing device in implementing the method as described in any one of claims 1 to 12.

Images