Data processing method and device based on processing flow in memory, and medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a data processing method and device based on an in-memory processing flow, and a medium. The method comprises the following steps: in a PIM mode of a first DRAM, calculating a first Q vector, a first K vector and a first V vector of a self-attention layer by using the first DRAM; in a non-PIM mode of the second DRAM, storing the first K vector, the first V vector and the first Q vector into the second DRAM; in a PIM mode of the first DRAM, performing self-attention calculation based on the first Q vector and a second K vector and a second V vector stored in the second DRAM to obtain second output information corresponding to a self-attention layer; and inputting the second output information to a subsequent network. According to the technical scheme, the problem that self-attention calculation in a neural network model is difficult to realize by an in-memory processing calculation process of an in-memory processing technology based on a dynamic random access memory at present can be solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, device, and medium based on in-memory processing flow. Background Technology

[0002] Neural network models are widely used in text classification, generation, and information retrieval. Among them, neural network models with self-attention layers typically exhibit higher performance when processing sequential data. However, for neural network models with self-attention layers, during the inference process, generating each token requires reading all the model's weights from dynamic random access memory (DRAM). For neural network models with many parameters, especially large language models, the bandwidth of DRAM limits the speed at which the neural network model can read weights.

[0003] To address the aforementioned issues, DRAM-PIM (Dynamic Random Access Memory-based Processing in Memory) technology can be employed. This technology utilizes DRAM to execute PIM calculations, thereby eliminating the bandwidth limitation on the speed at which neural network models can retrieve weights. In the DRAM-PIM computation process, the DRAM is placed in in-memory processing mode. However, in this mode, the DRAM only supports vector-matrix multiplication with static parameters and does not support calculations of dynamic parameters. In other words, dynamic parameter calculations cannot be performed in the DRAM-PIM computation process.

[0004] However, during the inference process of a neural network model, the K and V values ​​required for self-attention computation are calculated in real time for each token, and therefore cannot be pre-written as static parameters into the dynamic random access memory. Consequently, the current in-memory processing flow of DRAM-PIM technology struggles to implement self-attention computation in large models. Summary of the Invention

[0005] This application provides a data processing method, device, medium, and program product based on in-memory processing flow to solve the problem that the in-memory processing computation flow based on current dynamic random access memory (DRAM) is difficult to implement self-attention computation in large models.

[0006] To achieve the above objectives, in a first aspect, embodiments of this application provide an in-memory processing data processing method based on an in-memory processing mode, comprising: in the in-memory processing mode of a first dynamic random access memory (DRAM), calculating a first Q-vector, a first K-vector, and a first V-vector of a self-attention layer in a neural network model based on the first output information of a pre-order network in the DRAM, wherein the first DRAM is a storage area in a memory system, the pre-order network is a network located before the self-attention layer, the first Q-vector is used to identify the query vector corresponding to the first output information, and the first K-vector and the first V-vector are used to identify the feature vector corresponding to the first output information; in a second... In the non-in-memory processing mode of the dynamic random access memory, the first K vector, the first V vector, and the first Q vector are stored in the second dynamic random access memory. In the in-memory processing mode of the first dynamic random access memory, self-attention calculation is performed based on the first Q vector and the second K vector and the second V vector stored in the second dynamic random access memory to obtain the second output information corresponding to the self-attention layer, wherein the second K vector contains the first K vector and the second V vector contains the first V vector. The second output information is input into the subsequent network in the neural network, wherein the subsequent network is the network located after the self-attention layer, and the subsequent network is used to obtain the target output information of the neural network model based on the second output information.

[0007] The data processing method based on in-memory processing mode disclosed in this application utilizes a first dynamic random access memory (DRAM) in in-memory processing mode for self-attention computation, thereby improving computational efficiency. Furthermore, the first DRAM in in-memory processing mode can also be used for computation in both the pre-sequence network preceding the self-attention layer and the post-sequence network following the self-attention layer, further improving the inference speed of large models. Building upon this, considering that the first DRAM in in-memory processing mode cannot write dynamic parameters, this application stores the real-time computed first Q vector, first K vector, and first V vector in a second DRAM in non-in-memory processing mode. This solves the problem that the in-memory processing computation flow based on DRAM in-memory processing mode cannot store dynamic parameters, making it difficult to implement self-attention computation in neural network models.

[0008] In one implementation, the second dynamic random access memory (DRAM) and the first DRAM are the same DRAM, with the second DRAM located in the in-memory processing area of ​​the memory system. Using this implementation, since the second and first DRAMs are the same DRAM located in the in-memory processing area, different modes are switched for different operations. Computation is performed in the in-memory processing mode, while the storage of the first Q vector, first K vector, and first V vector is performed in the non-in-memory processing mode. This eliminates the need for two DRAMs; only one DRAM is used for operation mode switching. This solves the problem that the in-memory processing flow cannot perform self-attention calculations for neural network models because the DRAM cannot store dynamic parameters in the in-memory processing mode.

[0009] In one implementation, the second dynamic random access memory (DRAM) and the first DRAM are different DRAMs, with the second DRAM located in the non-in-memory processing area of ​​the memory system. In this implementation, the first DRAM is located in the in-memory processing area of ​​the memory system, and the second DRAM is located in the non-in-memory processing area; they are different DRAMs. The first DRAM can be set to in-memory processing mode, and the second DRAM to non-in-memory processing mode. This achieves the technical effect of having the first DRAM perform computational operations and the second DRAM perform storage operations without switching DRAM operating modes. This solves the problem of DRAM's inability to store dynamic parameters in in-memory processing mode and also reduces the time overhead during switching.

[0010] In one implementation, self-attention calculation is performed based on a first Q-vector and a second K-vector and a second V-vector stored in a second dynamic random access memory (DRAM). This includes: using a neural processing unit or graphics processing unit integrated on a system-on-a-chip (SoC) to read the second K-vector and the second V-vector from the second DRAM, wherein there is a communication connection between the SoC and the second DRAM; and using the neural processing unit or graphics processing unit to perform self-attention calculation based on the first Q-vector and the read second K-vector and the second V-vector. This implementation addresses the issue of the second DRAM, located outside of memory processing areas, being unable to perform calculations. Thus, the SoC, connected to the second DRAM, performs the self-attention calculation, while the first DRAM performs other calculations, allowing the first DRAM to still execute in-memory processing normally.

[0011] In one implementation, self-attention calculation is performed based on a first Q-vector and a second K-vector and a second V-vector stored in a second dynamic random access memory (DRAM) to obtain the second output information corresponding to the self-attention layer. This includes: reading the second K-vector from the second DRAM and obtaining the self-attention score corresponding to the second K-vector based on the first Q-vector and the second K-vector; and reading the second V-vector from the second DRAM and obtaining the second output information corresponding to the self-attention layer based on the self-attention score and the second V-vector. Using this implementation, the self-attention score is first calculated, and then a weighted calculation is performed using the self-attention score to obtain the output of the self-attention layer, i.e., the second output information. This enables self-attention calculation operations, improving the performance and expressive power of the neural network model.

[0012] In one implementation, storing the first K vector, the first V vector, and the first Q vector in a second dynamic random access memory (DRAM) includes: storing the first K vector and the first V vector in a key-value cache, wherein the key-value cache is located in the second DRAM and is a data structure that stores the first K vector and the first V vector in key-value pair format. This implementation utilizes a key-value cache to store the first K vector and the first V vector, avoiding the need to repeatedly calculate all K and V vectors during each inference iteration, thus improving computational efficiency and accelerating the inference process. Furthermore, when processing real-time data or long text tasks, the vectors stored in the key-value cache can be updated incrementally, eliminating the need to update the entire vector sequence at once, thereby improving inference efficiency and real-time performance.

[0013] In one implementation, the first dynamic random access memory (DRAM) and the second DRAM are the same DRAM. The second DRAM includes a first sub-DRAM and a second sub-DRAM, located in the non-in-memory processing area of ​​the memory system. Storing the first K vector, the first V vector, and the Q vector into the second DRAM includes: storing the first K vector and the first V vector into the second sub-DRAM in the non-in-memory processing mode of the second sub-DRAM. In this implementation, the first sub-DRAM in the second DRAM is located in the in-memory processing area and can be used for computation operations in in-memory processing mode, while the second sub-DRAM is located in the non-in-memory processing area and can be used for storage operations in non-in-memory processing mode. Thus, in subsequent self-attention calculations, the second K vector and the second V vector are read from the second DRAM, which means reading the vectors stored in the first and second sub-DRAMs for calculation. With this design, if the KV vector is also stored in the first sub-dynamic random access memory located in the processing area of ​​the main memory, the KV vector stored in the first sub-dynamic random access memory can also be read out and used in the calculation.

[0014] In one implementation, after performing self-attention calculation based on the first Q vector and the second K vector and second V vector stored in the second dynamic random access memory (DRAM), the method further includes: based on the second DRAM satisfying a preset stopping condition, in the non-in-memory processing mode of the first DRAM, storing the third K vector and third V vector stored in the second DRAM into the first DRAM, wherein the third K vector is all K vectors stored in the second DRAM, and the third V vector is all V vectors stored in the second DRAM; deleting the third K vector and third V vector stored in the second DRAM, and the memory system switching the first DRAM back to in-memory processing mode. Using this implementation, under certain stopping conditions, the operation mode of the first DRAM is switched, and the KV vectors (i.e., the third K vector and third V vector) stored in the second DRAM are moved to the first DRAM. In this way, the moved third K-vector and third V-vector can be used for self-attention calculation in the first sub-DRAM, avoiding the storage of too many third K-vectors and third V-vectors in the second sub-DRAM. This prevents the on-chip system from reading a large number of third K-vectors and third V-vectors from the second sub-DRAM, which would reduce the inference performance of the neural network model. In addition, this implementation clears the second sub-DRAM, providing storage space for the new first K-vector and first V-vector.

[0015] In one implementation, the preset stopping condition is that the number of third K vectors and / or the number of third V vectors in the second sub-dynamic random access memory (DRAM) reaches a preset threshold. By using the number of third K vectors and third V vectors in the second sub-DRAM as the criterion for determining whether the stopping condition has been met, this implementation ensures that the number of third K vectors and third V vectors stored in the second sub-DRAM is not excessive, thus preventing the on-chip system from reading a large number of third K vectors and third V vectors from the second sub-DRAM, which could lead to a decrease in the inference performance of the neural network model.

[0016] In one implementation, self-attention calculation is performed based on a first Q-vector and a second K-vector and a second V-vector stored in a second dynamic random access memory (DRAM) to obtain the second output information corresponding to the self-attention layer. This includes: reading the second K-vector from both the first and second DRAMs; obtaining the self-attention score corresponding to the second K-vector based on the first Q-vector and the second K-vector; reading the second V-vector from both the first and second DRAMs; and obtaining the second output information corresponding to the self-attention layer based on the self-attention score and the second V-vector. This implementation first calculates the self-attention score using the second K-vector stored in both DRAMs, and then performs a weighted calculation using the self-attention score and the second V-vector stored in both DRAMs to obtain the output of the self-attention layer, i.e., the second output information. This enables self-attention calculation, improving the performance and expressive power of the neural network model.

[0017] In one implementation, reading a second K-vector from both a first sub-DRAM and a second sub-DRAM includes: in the in-memory processing mode of the first sub-DRAM, using a computing unit to read a fourth K-vector stored in the first sub-DRAM, wherein the computing unit is integrated into the first sub-DRAM; and in the non-in-memory processing mode of the second sub-DRAM, using a neural processing unit or graphics processing unit integrated on a system-on-a-chip (SoC) to read a fifth K-vector stored in the second sub-DRAM, wherein there is a communication connection between the SoC and the second sub-DRAM; the second K-vector is the combination of the fourth K-vector and the fifth K-vector. This implementation, considering that both the first and second sub-DRAMs can store KV vectors, allows reading the fourth K-vector stored in the first sub-DRAM using a computing unit and reading the fifth K-vector stored in the second sub-DRAM using an SoC. This enables data reading for both different types of first and second sub-DRAMs. Furthermore, by combining the read data, the second K vector can be obtained.

[0018] In one implementation, obtaining the self-attention score corresponding to the second K vector based on the first Q vector and the second K vector includes: processing the first Q vector and the fourth K vector using the first sub-DRAM in in-memory processing mode to obtain the first sub-self-attention score corresponding to the fourth K vector; processing the first Q vector and the fifth K vector using a neural processing unit or a graphics processing unit in non-in-memory processing mode of the second sub-DRAM to obtain the second sub-self-attention score corresponding to the fifth K vector; and concatenating the first sub-self-attention score and the second sub-self-attention score to obtain the self-attention score corresponding to the second K vector. This implementation, considering that both the first and second sub-DRAMs can store K vectors, allows for the calculation of the self-attention score based on the fourth K vector using the first sub-DRAM, and the calculation of the self-attention score based on the fifth K vector using the neural processing unit or graphics processing unit integrated in the system-on-a-chip. Thus, self-attention score calculation can be implemented for different types of first and second sub-DRAMs. Furthermore, by combining the calculated first and second sub-self-attention scores, the final self-attention score can be obtained.

[0019] In one implementation, reading the second V vector from both the first and second sub-DRAMs includes: in the in-memory processing mode of the first sub-DRAM, using a computing unit to read the fourth V vector stored in the first sub-DRAM; and in the non-in-memory processing mode of the second sub-DRAM, using a neural processing unit or a graphics processing unit to read the fifth V vector stored in the second sub-DRAM. The second V vector is the combination of the fourth and fifth V vectors. This implementation, considering that both the first and second sub-DRAMs can store KV vectors, allows the PU to read the fourth V vector stored in the first sub-DRAM and the on-chip system to read the fifth V vector stored in the second sub-DRAM. This enables data reading for both different types of first and second sub-DRAMs. Furthermore, the read data can be combined to obtain the second V vector.

[0020] In one implementation, the second output information corresponding to the self-attention layer is obtained based on the self-attention score and the second V vector. This includes: processing the self-attention score and the fourth V vector using the first sub-DRAM in in-memory processing mode to obtain the first sub-output information; and processing the second sub-DRAM in non-in-memory processing mode using a neural processing unit or a graphics processing unit to obtain the second sub-output information based on the self-attention score and the fifth V vector. The first and second sub-output information are then concatenated to obtain the second output information corresponding to the self-attention layer. This implementation, considering that both the first and second sub-DRAMs can store V vectors, allows for weighted calculation of the self-attention score based on the fourth V vector using the first sub-DRAM, and weighted calculation of the self-attention score based on the fifth V vector using the neural processing unit or graphics processing unit integrated in the system-on-a-chip. This allows for weighted calculation of the self-attention score for different types of first and second sub-DRAMs. Furthermore, by combining the calculated first sub-output information and the second sub-output information, the final second output information can be obtained.

[0021] In one implementation, the first sub-dynamic random access memory includes a first key-value cache, which stores the fourth K vector and the fourth V vector using a key-value pair data structure. The second sub-dynamic random access memory includes a second key-value cache, which stores the fifth K vector and the fifth V vector using a key-value pair data structure. This implementation utilizes the first key-value cache to store the fourth K vector and the fourth V vector, and the second key-value cache to store the fifth K vector and the fifth V vector, avoiding the need to repeatedly calculate all K and V vectors during each inference, thus improving computational efficiency and accelerating the inference process. Furthermore, when processing real-time data or long text tasks, the vectors stored in the second key-value cache can be updated incrementally, eliminating the need to update the entire vector sequence at once, thereby improving inference efficiency and real-time performance.

[0022] In one implementation, the first key-value cache includes a first key cache and a first value cache, wherein the first key cache stores the fourth K vector and the first value cache stores the fourth V vector; the second key-value cache includes a second key cache and a second value cache, wherein the second key cache stores the fifth K vector and the second value cache stores the fifth V vector. This implementation further subdivides the first key-value cache into a first key cache and a first value cache, facilitating finer-grained control. Different update frequencies can be applied to the fourth K vector and the fourth V vector, and memory usage can be optimized. The beneficial effects of the second key cache and the second value cache are the same as those of the first key cache and the first value cache, and will not be elaborated further here.

[0023] In one implementation, the first sub-dynamic random access memory (DRAM) is also used to store the weight parameters of the neural network model. By storing the weight parameters of the neural network model in the first sub-DRAM, the first sub-DRAM can directly read the weight parameters during computation. This reduces the latency of reading the weight parameters of the neural network model and improves the inference efficiency of the neural network model.

[0024] In one implementation, after inputting the second output information into the subsequent network, the method further includes: processing the second output information using the subsequent network; and determining the processed second output information as the target output information of the neural network model based on the inference termination condition being met. This implementation utilizes the subsequent network to continue processing the second output information and uses the inference termination condition verification to determine the output of the subsequent network that satisfies the inference termination condition as the target output information of the neural network model. In this way, the quality of the target output information can be guaranteed through the inference termination condition.

[0025] In one implementation, the method further includes: determining new input information based on the processed second output information since the inference termination condition has not been met; inputting the new input information into the preceding network to obtain new first output information; and returning to the step of calculating the first Q vector, first K vector, and first V vector of the self-attention layer in the neural network based on the first output information of the preceding network. Using this implementation, if the inference termination condition is not met, the next round of inference is executed again until the inference termination condition is met. Thus, through multiple rounds of inference, the quality of the target output information of the neural network model can be improved.

[0026] Secondly, this application also provides an electronic device, including a memory and a processor; the memory and the processor are coupled; wherein the memory is used to store computer program code, the computer program code including computer instructions, and when the processor executes the computer instructions, it causes the electronic device to perform the data processing method based on the in-memory processing flow as described in the first aspect and any implementation thereof.

[0027] Thirdly, this application also provides a chip system, which includes a processor; the processor is coupled to a memory, the memory being used to store computer program code, the computer program code including computer instructions, and when the processor executes the computer instructions, the data processing method based on in-memory processing flow as described in the first aspect and any implementation thereof.

[0028] Fourthly, this application also provides a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform a data processing method based on an in-memory processing flow as described in the first aspect and any implementation thereof.

[0029] Fifthly, this application also provides a computer program product, which includes: a computer program or instructions that, when run on a computer, cause the computer to perform the data processing method based on in-memory processing flow as described in the first aspect and any implementation thereof.

[0030] Understandably, the beneficial effects that the technical solutions provided in the second to fifth aspects described above can be achieved by referring to the beneficial effects of the first aspect and any of its optional implementation methods, which will not be repeated here. Attached Figure Description

[0031] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a schematic diagram of data storage and computation in a large-scale model inference scenario;

[0033] Figure 2 This is a schematic diagram of data storage and computation in a large-scale model inference scenario based on DRAM-PIM technology;

[0034] Figure 3 It is a network structure diagram of a large model;

[0035] Figure 4 yes Figure 3Schematic diagram of the Transformer Block Layer structure of the medium-sized model;

[0036] Figure 5 yes Figure 4 Schematic diagram of the self-attention computation structure;

[0037] Figure 6 yes Figure 5 A schematic diagram of parameter storage in the self-attention computation structure during the self-attention computation process;

[0038] Figure 7 This is a flowchart illustrating an embodiment of the data processing method based on in-memory processing mode provided in this application.

[0039] Figure 8 This is a flowchart of another embodiment of the data processing method based on in-memory processing mode provided in this application;

[0040] Figure 9 Is adopted Figure 8 The illustrated embodiment is a flowchart illustrating the execution of large model inference operations;

[0041] Figure 10 yes Figure 9 Corresponding data storage diagram;

[0042] Figure 11 This is a flowchart of yet another embodiment of the data processing method based on in-memory processing mode provided in this application;

[0043] Figure 12 Is adopted Figure 11 The illustrated embodiment is a flowchart illustrating the execution of large model inference operations;

[0044] Figure 13 yes Figure 12 Corresponding data storage diagram;

[0045] Figure 14 This is a flowchart of yet another embodiment of the data processing method based on in-memory processing mode provided in this application;

[0046] Figure 15 Is adopted Figure 14 The illustrated embodiment is a flowchart illustrating the execution of large model inference operations;

[0047] Figure 16 yes Figure 15 Corresponding data storage diagram;

[0048] Figure 17 This is a schematic diagram of a data processing device based on an in-memory processing mode provided in an embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are all within the protection scope of this application.

[0050] In the following description, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0051] Furthermore, in this application, directional terms such as "upper," "lower," "inner," and "outer" are defined relative to the indicated placement of the components in the accompanying drawings. It should be understood that these directional terms are relative concepts, used for relative description and clarification, and can change accordingly depending on the placement of the components in the accompanying drawings.

[0052] The following explanations of the technical terms mentioned in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0053] A large model, also known as a large language model (LLM), refers to a machine learning model with a large number of parameters and a complex computational structure. Large models are typically composed of deep neural networks and have billions or even hundreds of billions of parameters. Compared to smaller models with fewer parameters, large models have stronger expressive power and higher accuracy. The data processing method in this application embodiment can be implemented using a large model.

[0054] Dynamic Random Access Memory (DRAM) is a type of memory widely used in personal computers, servers, graphics processing units, and embedded systems, providing fast data access and storage. In the embodiments of this application, DRAM can be used to perform computational operations and data storage operations during the inference process of a neural network model.

[0055] A processing unit (PU) refers to a unit used to perform computational tasks. In the embodiments of this application, a processing unit refers to a unit integrated on DRAM to enable DRAM to have computational capabilities.

[0056] A memory bank refers to a basic unit used for storing data. In the embodiments of this application, a memory bank refers to a unit integrated on DRAM to enable DRAM to have data storage capabilities.

[0057] Processing-in-Memory (PIM) technology refers to a technique that integrates computational operations directly into memory. Compared to traditional techniques that read data from memory and then perform calculations on the processor, PIM technology performs computational operations within memory, thus reducing data transfer latency between the processor and memory and improving computational efficiency.

[0058] Dynamic Random Access Memory-Processing-In-Memory (DRAM-PIM) technology refers to the application of PIM technology to DRAM. In the embodiments of this application, DRAM-PIM technology can be used to perform PIM calculations in DRAM, achieving the same technical effect as performing calculations in DRAM.

[0059] In-Memory (PIM) processing mode refers to an operating mode of DRAM. PIM mode integrates computing units within the DRAM chip, allowing computational tasks to be performed within the DRAM without transferring data to the processor. Therefore, PIM mode reduces data transfer latency between the processor and memory, thereby improving computational efficiency. In this embodiment, switching the DRAM to PIM mode enables the use of DRAM-PIM technology, where the DRAM performs the PIM computation process.

[0060] Standard mode, also known as Normal mode, refers to another operating mode of DRAM. In Normal mode, DRAM acts as a traditional memory module, responsible for storing data for access by the CPU or other processors. DRAM in Normal mode has no computational function and does not perform computational tasks; therefore, data needs to be transferred to the processor, and the processor returns the calculation result to the DRAM after completing the computation. In this embodiment, the DRAM is switched to Normal mode, meaning that the DRAM performs the storage operation.

[0061] A System-on-Chip (SoC) is an integrated circuit that integrates all or most of the components of an electronic system onto a single chip. These components may include a central processing unit, a graphics processing unit, a memory controller, input / output interfaces, network interfaces, digital signal processors, and various other dedicated function modules. In the embodiments of this application, the processor (or processing unit) integrated in the SoC can be used to perform computational operations.

[0062] A Neural Processing Unit (NPU), typically integrated on a System-on-a-Chip (SoC), can be used to accelerate machine learning and deep learning tasks, such as image recognition, speech recognition, and natural language processing. In the embodiments of this application, the NPU can be used to perform computational operations.

[0063] A graphics processing unit (GPU), typically integrated on a system-on-a-chip (SoC), can be used for image processing, scientific computing, data analysis, and other applications. In this embodiment, the GPU can be used to perform computational operations.

[0064] Double Data Rate (DDR) is a memory technology that enables double the data transfer rate. In this application's embodiments, DDR refers to DDR memory chips. A DDR memory chip is a memory chip based on DDR technology that can be used for temporary data storage for fast access by processors such as the CPU. DDR memory chips are a form of DRAM and are widely used in computer systems such as desktops, laptops, and servers.

[0065] Universal Flash Storage (UFS) refers to a high-performance storage technology with high transfer rates and low power consumption.

[0066] A token refers to the basic unit used by a large model when processing text data; it can be a word, a subword, or a character.

[0067] QKV (Key-Query-Value) is a key-value pair where Q (Query) is the representation of the token being processed, used to compare with the keys of other tokens to determine which tokens are most relevant to the current token. K (Key) is the representation of all tokens, used to compare with the query to calculate an attention score. The attention score represents the importance of each token to the current query. V (Value) is another representation of all tokens, used to perform a weighted sum based on the attention scores to generate a new, context-aware representation of the current token.

[0068] A key-value cache (KV-Cache) is a cache area used to store key-value pairs. By storing frequently accessed data in high-speed memory, the KV-Cache reduces access to slower memory, thereby improving data retrieval speed. In this embodiment, the KV-Cache can be used to store the K and V values ​​in the aforementioned QKV structure.

[0069] The application scenarios of the embodiments of this application will be described below with reference to the accompanying drawings.

[0070] Neural network models are widely used in text classification, generation, and information retrieval. Among them, neural network models with self-attention layers typically exhibit higher performance when processing sequential data. When processing text data, these models segment the text into tokens, allowing the model to understand the input text and generate output text by processing the token sequence. Currently, the weight parameters of neural network models are pre-stored in DRAM. Therefore, for each token generated, the model needs to read all weight parameters from DRAM. For neural network models with many parameters, especially large language models, DRAM bandwidth limits the speed at which the model can read weight parameters, thus limiting the model's inference speed.

[0071] Figure 1 This is a schematic diagram of data storage and computation in a large-scale model inference scenario.

[0072] like Figure 1 As shown, the processor integrated in the SoC receives input data and sends the data to the DRAM (e.g., Figure 1 The system uses DDR memory chips as DRAM. Based on data sent by the SoC, the DDR memory chip reads the corresponding model data from the UFS and feeds it back to the SoC. The SoC performs calculations based on the data fed back from the DDR memory chip. During the calculation process, the SoC and the DDR memory chip exchange data multiple times to finally obtain the output.

[0073] In the aforementioned large-scale model inference scenarios, the overall inference speed of the model is limited by bandwidth because the data transfer rate between the SoC and the DDR memory chip is affected by the bandwidth of the DDR memory chip. When bandwidth is low, the numerous data interactions between the SoC and the DDR memory chip significantly reduce the model's inference speed. For example, for the Llama2 7B model, even with the LPDDR5X (Low Power Double Data Rate 5X) chip's full bandwidth of 68GB / s, it can only provide an inference speed of approximately 16 tokens / s.

[0074] Considering the bandwidth limitation on the inference speed of large models, current solutions employ DRAM-PIM technology, utilizing DRAM to perform PIM computation processes, thereby improving model inference speed. Unlike conventional DRAM, which is used solely for data storage while computational operations are performed by processors such as CPUs, DRAM-PIM technology integrates computational functions into DRAM. This means both storage and computational operations are performed by DRAM, reducing data transfer between memory and processor. Therefore, it eliminates the impact of bandwidth on the inference speed of large models, improves inference efficiency, and accelerates large model performance.

[0075] Figure 2 This is a schematic diagram of data storage and computation in a large-scale model inference scenario based on DRAM-PIM technology.

[0076] like Figure 2 As shown in (a), the SoC and DRAM form a PIM system, in which the DRAM uses LPDDR memory chips. In this scenario, the DRAM has computing capabilities and can perform computing operations in PIM mode.

[0077] like Figure 2 As shown in (b), the DRAM contains multiple computing units (PUs) and multiple memory banks (such as...). Figure 2 The technology involves using BankA and BankB in a database, where each bank can be read and written independently. This allows for parallel access to multiple banks, improving data processing efficiency.

[0078] like Figure 2 As shown in (c), the processor integrated in the SoC receives input data and sends the data to the DRAM (e.g., Figure 2 The system uses DDR memory chips as DRAM. The DDR memory chips read corresponding model data from UFS based on data sent by the SoC. This differs from... Figure 1In the scenario shown, the DDR memory chip feeds data back to the SoC, which then performs calculations. In this scenario, the DDR memory chip, in PIM mode, can execute the PIM calculation process, specifically by using the PU to read data stored in Bank A and Bank B and perform calculations. After the calculation is complete, the DDR memory chip feeds the result back to the SoC, which then outputs the result.

[0079] like Figure 2 As shown in (d), due to Figure 2 The computational operations shown are performed within the DDR memory chip, therefore they can be omitted. Figure 1 The scenario shown illustrates multiple data interactions between the SoC and DDR. Therefore, employing DRAM-PIM technology can reduce the impact of DRAM bandwidth on model inference speed. Compared to baseline performance and power consumption, DRAM-PIM technology can effectively improve the performance of large models and reduce power consumption.

[0080] However, DRAM-PIM technology still has the following problems when used for large model acceleration.

[0081] First, combine Figure 3-6 The network structure of the large model is explained.

[0082] Figure 3 It is a network structure diagram of a large model.

[0083] like Figure 3 As shown, the large model consists of sequentially connected input embedding layers, several layers of the Transformer block (Transformer Block Layer 1 to Transformer Block Layer 32), root mean square normalization (RMSNorm) layers, linear transformation layers, and softmax layers. The principles and functions of each layer will not be elaborated here.

[0084] Figure 4 yes Figure 3 A schematic diagram of the Transformer Block Layer structure of the medium-to-large model.

[0085] like Figure 4As shown, in Transformer Block Layers 1 to 32, each layer contains a sequentially connected self-attention layer and a feedforward network. The feedforward network includes a mean squared error normalization layer, a linear transformation layer, and a Sigmoid Linear Unit (SiLU). The self-attention layer includes a mean squared error normalization layer, a linearization layer, and a self-attention computation structure, where the self-attention computation structure is used to perform self-attention computation operations.

[0086] For ease of description, this application refers to the network structure before the self-attention layer in the large model as the pre-sequence network, and the network after the self-attention layer as the subsequent network.

[0087] Figure 5 yes Figure 4 Schematic diagram of the self-attention computation structure; Figure 6 yes Figure 5 A schematic diagram of parameter storage in the self-attention computation structure during the self-attention computation process.

[0088] like Figure 5 As shown, in the self-attention layer, the token output by the preceding network (such as...) is received. Figure 6 The NewToken shown undergoes mean squared error normalization and linearization before entering the self-attention computation structure. It should be noted that the self-attention structure in this application is a partial network structure contained within the self-attention layer; the relationship between the self-attention layer and the self-attention structure is explained above. Figure 4 This will not be elaborated further here. In subsequent embodiments of this application, for ease of description, the mean squared error normalization layer and linearization layer in the self-attention layer are omitted. In other words, in this application... Figure 7-16 In the illustrated embodiment, the self-attention layer is equivalent to the self-attention structure.

[0089] exist Figure 5 In the self-attention computation structure, three vectors are obtained: Q vector, K vector, and V vector (e.g., ...). Figure 6(See the Query, Key, and Value shown). The Q vector can be stored in memory or temporary storage, while the K and V vectors are stored in the KV-Cache. Since the KV-Cache stores the K and V vectors generated over multiple rounds of inference, all element values ​​in the K and V vectors can be quickly retrieved during the current inference process for self-attention calculation. The Q and K vectors are rotated position embedding (RoPE) to capture the positional information of each element. Self-attention calculation is then performed based on the Q, K, and V vectors to obtain the output of the self-attention layer. During the self-attention calculation, the self-attention score is first calculated using the position-encoded Q and K vectors. It can be understood that during the calculation of the self-attention score, if there are positions that need to be ignored, their values ​​can be set to 0 or negative infinity (-INF) to mask them. In subsequent flexibility maximization, the weight of that position will be equal to 0 or close to zero, effectively ignoring the information at that position. The self-attention score is then passed through a flexible maximization layer, and the V vector is weighted to obtain the output vector. Finally, the output vector is linearized and used as the output of the self-attention layer.

[0090] In this process, it can be seen that the K vector and V vector are calculated in real time based on the token during the large model inference process. In other words, the K vector and V vector are dynamic parameters.

[0091] However, when using DRAM-PIM technology to perform PIM calculations, the DRAM needs to be set to PIM mode. In PIM mode, DRAM only supports vector matrix multiplication where the matrix is ​​a static parameter. The K and V vectors required for self-attention calculations cannot be pre-written into the DRAM as static parameters. Therefore, current DRAM-PIM technology cannot achieve self-attention calculations in large models, meaning it cannot accelerate self-attention calculations.

[0092] To address the issue that the current PIM calculation process of DRAM-PIM technology cannot be applied to self-attention calculation in large models, this application proposes a data processing method based on in-memory processing mode.

[0093] In the data processing method based on in-memory processing mode proposed in this application, DRAM-PIM technology is still used to perform self-attention calculation in DRAM. Considering that when performing PIM calculation using DRAM-PIM technology, DRAM in PIM mode does not support the pre-writing of K vectors and V vectors as static parameters into DRAM, this application, after obtaining K vectors and V vectors based on tokens, writes K vectors and V vectors into non-PIM mode DRAM to solve the problem that PIM mode DRAM cannot write K vectors and V vectors, which makes it difficult to implement self-attention calculation in large models in the PIM calculation process.

[0094] The following is combined with Figures 7-16 The technical solution proposed in this application will be further described below.

[0095] Figure 7 This is a flowchart illustrating one embodiment of the data processing method based on in-memory processing mode provided in this application.

[0096] like Figure 7 As shown, this embodiment includes the following steps S11-S14.

[0097] S11: In the in-memory processing mode of the first dynamic random access memory, the first Q vector, the first K vector, and the first V vector of the self-attention layer in the neural network model are calculated based on the first output information of the preorder network in the neural network model using the first dynamic random access memory.

[0098] This application applies to a neural network model, which includes a pre-sequence network, a self-attention layer, and a post-sequence network. The pre-sequence network is the network located before the self-attention layer, and typically includes an input embedding layer, a position encoding layer, a feedforward neural network, etc. The post-sequence network is the network located after the self-attention layer, and typically includes a feedforward neural network, residual connections and normalization layers, an output layer, etc. The specific structure of the pre-sequence network and the post-sequence network is not limited here.

[0099] This application embodiment targets the preceding network before the self-attention layer, and can employ DRAM-PIM technology to execute the PIM process. Specifically, in the PIM mode of the first DRAM (first dynamic random access memory), computation is performed using the first DRAM to obtain the first output information. It can be understood that the first DRAM is a storage area in the memory system. In PIM mode, the first DRAM, in addition to its storage function, also has computational functions. Therefore, the memory system of this application embodiment adjusts the first DRAM to PIM mode and utilizes the computational function of the first DRAM to perform computations on the preceding network, thereby improving the inference speed of the neural network model.

[0100] Subsequently, for the self-attention layer, the DRAM-PIM technique can be continued, utilizing the first DRAM based on the first output information of the preceding network to perform QKV vector calculation, thereby improving computation speed. For example, the first output information can be processed through three different fully connected layers to generate three vectors: a Q vector, a K vector, and a V vector, denoted as the first Q vector, the first K vector, and the first V vector, respectively. Here, Q stands for Query, and the first Q vector identifies the query vector corresponding to the current first output information. K stands for Key, and V stands for Value; the first K vector is paired with the first V vector to identify the feature vector corresponding to the first output information. For detailed concepts of Q vector, K vector, and V vector, please refer to the aforementioned terminology description, which will not be repeated here.

[0101] S12: In the non-in-memory processing mode of the second dynamic random access memory, the first K vector, the first V vector, and the first Q vector are stored in the second dynamic random access memory.

[0102] In this embodiment, after obtaining the first K vector, first V vector, and first Q vector, they can be stored for subsequent steps to retrieve and compute. The first Q vector can be stored in memory or a temporary cache, such as in a second DRAM (second dynamic random access memory). The first K vector and first V vector are also stored in the second DRAM as key-value pairs. It is understood that the second DRAM is a storage area in the memory system. In non-PIM mode, the second DRAM has storage functionality but not computational functionality. Therefore, in this embodiment, the memory system adjusts the second DRAM to non-PIM mode, utilizing its storage function to store the first K vector, first V vector, and first Q vector calculated in the aforementioned steps. This solves the problem that the PIM computation process is difficult to implement self-attention computation in large models because the DRAM in PIM mode cannot write K and V vectors.

[0103] It is understood that non-PIM mode refers to modes other than PIM mode, such as Write-Through mode, Write-Back mode, etc. In some embodiments, non-PIM mode refers to Normal mode.

[0104] In some embodiments, step S12 includes the following steps:

[0105] S121: Store the first K vector and the first V vector into the key-value cache.

[0106] The KV-Cache is located in the second DRAM. The KV-Cache is a storage area that stores the first K vector and the first V vector using a key-value pair data structure.

[0107] In this embodiment, the first K vector and the first V vector are stored in a KV-Cache (key-value cache) in the form of key-value pairs. It can be understood that a KV-Cache is a storage area used to store key-value pair data. Due to the special nature of the key-value pair data structure, keys and values ​​can be quickly retrieved and stored in data stored using this structure. In this embodiment, a KV-Cache is provided in the second DRAM. After each calculation of the first K vector and the first V vector, they can be written into the KV-Cache to improve the efficiency of subsequently searching for the stored K vector and V vector in the second DRAM.

[0108] S13: In the in-memory processing mode of the first dynamic random access memory, self-attention calculation is performed based on the first Q vector and the second K vector and the second V vector stored in the second dynamic random access memory to obtain the second output information corresponding to the self-attention layer.

[0109] In this embodiment, self-attention calculation is performed using Q-vectors, K-vectors, and V-vectors for the self-attention layer. Specifically, DRAM-PIM technology can be continued, and the calculation is performed using the first DRAM in PIM mode to improve the calculation speed.

[0110] Furthermore, it can be understood that the Q vector used in the self-attention calculation process is the Q vector calculated based on the first output information in the current round of inference, which is also the first Q vector in the aforementioned step S11.

[0111] The K-vector used in the self-attention calculation process includes not only the K-vector calculated based on the first output information in the current inference round, but also the K-vectors obtained in previous inference rounds. Therefore, the first DRAM reads the second K-vector stored in the second DRAM for self-attention calculation. Since the K-vector is calculated using the first output information and stored in the second DRAM in each inference round, the second K-vector in the second DRAM includes the first K-vector from step S11 and the K-vectors stored in previous inference rounds.

[0112] Similarly, the first DRAM reads the second V vector stored in the second DRAM for self-attention calculation. The second V vector in the second DRAM includes the first V vector from step S11 above, as well as the V vector stored in previous rounds of inference.

[0113] In some embodiments, step S13 includes the following steps:

[0114] S131: Read the second K vector from the second dynamic random access memory, and obtain the self-attention score corresponding to the second K vector based on the first Q vector and the second K vector.

[0115] In this embodiment, self-attention is calculated using the first Q vector and the second K vector stored in the second DRAM to obtain the corresponding self-attention score. For example, the first Q vector and the second K vector can be dot-producted to obtain a self-attention score vector, which can represent the importance of each K vector in the second K vector.

[0116] Furthermore, after calculating the self-attention score using the first Q vector, the first Q vector stored in the first DRAM can be deleted.

[0117] S132: Read the second V vector from the second dynamic random access memory, and obtain the second output information corresponding to the self-attention layer based on the self-attention score and the second V vector.

[0118] In this embodiment, self-attention calculation is performed using the self-attention score and the second V vector stored in the second DRAM to obtain the corresponding second output information. For example, each self-attention score element in the self-attention score vector can be used to perform a weighted summation of the second V vector to obtain the final output representation, which serves as the second output information.

[0119] S14: Input the second output information into the subsequent network of the neural network model.

[0120] The post-processor network is the network located after the self-attention layer in the neural network model. The post-processor network is used to obtain the target output information of the neural network model based on the second output information.

[0121] In this embodiment of the application, after performing self-attention calculation on the self-attention layer and obtaining the second output information based on the first output information, the second output information is input into the subsequent network so that subsequent operations are performed on the subsequent network after the self-attention layer based on the second output information.

[0122] In one implementation, the embodiments of this application may further include the following steps:

[0123] S15: Process the second output information using the subsequent network.

[0124] In this embodiment, for the subsequent network after the self-attention layer, DRAM-PIM technology can be used to perform the PIM calculation process. Specifically, in the PIM mode of the first DRAM, the first DRAM is used to perform calculations on the second output information to obtain the processed second output information. The memory system of this embodiment adjusts the first DRAM to PIM mode and uses the computing function of the first DRAM to perform calculations on the subsequent network, which can improve the inference speed of the neural network model.

[0125] S16: Since the processed second output information does not meet the reasoning termination condition, new input information is determined based on the processed second output information.

[0126] S17: Input the new input information into the preceding network to obtain the new first output information, and return to the aforementioned step S11.

[0127] S18: Based on the processed second output information reaching the inference termination condition, the processed second output information is determined to be the target output information of the neural network model. After this, the inference operation of the neural network model can be terminated.

[0128] In this embodiment of the application, considering that better results can be obtained through multiple rounds of reasoning, a reasoning termination condition can be set, and a new round of reasoning operation can be performed if the reasoning termination condition is not met, thereby improving the quality of the target output information by increasing the number of reasoning rounds.

[0129] Based on this, if the reasoning termination condition is met, the next round of reasoning will not be executed. Instead, the reasoning operation of the neural network model will be terminated directly, and the processed second output information of the subsequent network output will be used as the target output information of the neural network model.

[0130] If the inference termination condition is not met, new input information can be determined based on the processed second output information from the subsequent network. This new input information is then used as input to the neural network model and fed into the preceding network to obtain new first output information. Through multiple rounds of inference by the neural network model, the quality of the target output information can be improved.

[0131] The reasoning termination condition can be designed according to the specific reasoning task and neural network model structure, such as reaching the maximum number of reasoning rounds or reaching the preset computing resource limit.

[0132] Figure 8 This is a flowchart of another embodiment of the data processing method based on in-memory processing mode provided in this application.

[0133] like Figure 8 As shown, this embodiment includes the following steps S21-S24.

[0134] S21: In the in-memory processing mode of the first dynamic random access memory, the first Q vector, the first K vector, and the first V vector of the self-attention layer in the neural network model are calculated based on the first output information of the preorder network in the neural network model using the first dynamic random access memory.

[0135] For an explanation of step S21, please refer to step S11 above, which will not be repeated here.

[0136] S22: In the non-in-memory processing mode of the first dynamic random access memory, the first K vector, the first V vector, and the first Q vector are stored in the first dynamic random access memory.

[0137] The embodiments of this application can be regarded as Figure 4 In the illustrated embodiment, the second DRAM and the first DRAM are the same DRAM. In this case, to enable the first DRAM to have computing capabilities, the first DRAM should be located in the PIM area of ​​the memory system, where the PIM area refers to the area where computing units are integrated. In other words, the first DRAM in the PIM area integrates computing units, ensuring that the first DRAM has computing capabilities in addition to its conventional storage capabilities, thus enabling storage and computing operations to be performed through the first DRAM.

[0138] In this embodiment, different operations can be performed using the first DRAM by switching it to different modes. For example, in step S21, the first DRAM is in PIM mode to perform computation operations, while in step S22, the first DRAM is switched to non-PIM mode to perform storage operations on the first K vector, the first V vector, and the first Q vector.

[0139] It is understood that non-PIM mode refers to modes other than PIM mode. For details, please refer to the introduction of non-PIM mode in step S12, which will not be repeated here.

[0140] In some embodiments, step S22 includes the following steps:

[0141] S221: Store the first K vector and the first V vector into the key-value cache.

[0142] The KV-Cache is located in the first DRAM. For an explanation of the KV-Cache, please refer to the aforementioned embodiments, which will not be repeated here.

[0143] For an explanation of step S221, please refer to step S121 above, which will not be repeated here.

[0144] S23: In the in-memory processing mode of the first dynamic random access memory, self-attention calculation is performed based on the first Q vector and the second K vector and the second V vector stored in the first dynamic random access memory to obtain the second output information corresponding to the self-attention layer.

[0145] S24: Input the second output information into the subsequent network.

[0146] For an explanation of steps S23-S24, please refer to steps S13-S14 above, which will not be repeated here.

[0147] In one implementation, the embodiments of this application may further include the following steps:

[0148] S25: Process the second output information using a subsequent network.

[0149] S26: Based on the processed second output information reaching the inference termination condition, the processed second output information is determined to be the target output information of the large model. After this, the inference operation of the large model can be terminated.

[0150] S27: Since the processed second output information does not meet the reasoning termination condition, new input information is determined based on the processed second output information.

[0151] S28: Input the new input information into the preceding network to obtain the new first output information. After that, you can return to the previous step S21.

[0152] For an explanation of steps S25-S28, please refer to steps S15-S18 above, which will not be repeated here.

[0153] Figure 8 In the illustrated embodiment, the first DRAM is set to PIM mode during the computation step to ensure its computational capabilities. During the data storage step, the first DRAM is set to non-PIM mode to ensure successful writing of the first K vector and the first V vector. By switching the first DRAM's operating mode, the problem of the PIM computation process being unable to perform self-attention calculations when the first DRAM is in PIM mode and cannot write dynamic data such as the first K vector and the first V vector during the current DRAM-PIM technology is implemented can be solved.

[0154] The following is combined with Figure 9-10 ,right Figure 8 The illustrated embodiment will be further explained.

[0155] Figure 9 Is adopted Figure 8 The illustrated embodiment is a flowchart illustrating the execution of large model inference operations; Figure 10 yes Figure 9The corresponding data storage diagram.

[0156] In this embodiment, the first DRAM is switched to Normal mode, so that the first DRAM is in a state where the computing function is not enabled.

[0157] In this embodiment, the first DRAM is the first DRAM located in the PIM area (e.g., Figure 10 The PIM-DRAM shown, also known as the first DRAM, has computing capabilities. In actual operation, the computing function can be controlled by switching the operating mode of the first DRAM. Switching the first DRAM to PIM mode enables its computing function; switching it to other modes disables its computing function.

[0158] like Figure 10 As shown, before performing the inference operation of the neural network model using this embodiment, the model's weight parameters are pre-stored in the first DRAM. During model inference, the weight parameters can be directly read from the first DRAM.

[0159] like Figure 9 As shown, when performing inference operations using this embodiment, the first DRAM is first switched to PIM mode, and DRAM-PIM technology is used to perform pre-processing network calculations before the self-attention layer using the first DRAM. Subsequently, the self-attention layer calculates the latest Q-vector, K-vector, and V-vector based on the output of the pre-processing network, which are the first Q-vector, first K-vector, and first V-vector.

[0160] Since DRAM-PIM technology supports PIM calculation processes to be executed by DRAM, enabling dynamic random access memory to perform calculation operations, DRAM-PIM technology supports calculation operations within the first DRAM. Therefore, both the preceding network calculation and the QKV vector calculation can be performed within the first DRAM, which can reduce data interaction between the first DRAM and the processor and improve model inference efficiency.

[0161] Next, the first DRAM is switched to Normal mode, and the calculated first K vector and first V vector are written into the first DRAM. For example... Figure 10 As shown, the data can be written to the KV-Cache located in the first DRAM. This embodiment avoids the problem that the first DRAM cannot write the first K vector and the first V vector in PIM mode by switching the first DRAM to Normal mode.

[0162] Finally, the first DRAM is switched back to PIM mode. Using DRAM-PIM technology, self-attention calculation is performed by the first DRAM based on QKV vectors, and subsequent network operations after the self-attention layer are performed based on the self-attention calculation results. This embodiment uses DRAM-PIM technology to perform calculations within the first DRAM, which reduces data interaction between the first DRAM and the processor, improving model inference efficiency.

[0163] Furthermore, after the subsequent network operation, it can be determined whether the inference has ended. If it has ended, the current operation result of the subsequent network can be directly used as the output of the neural network model, i.e., the target output information. If it has not ended, it returns to the computation of the preceding network before the attention layer to perform the next round of inference operation until the inference ends.

[0164] In summary, in this embodiment, after each calculation of the first K vector and the first V vector, the first DRAM is switched to Normal mode. The first K vector and the first V vector are written in Normal mode. After writing is complete, the first DRAM is switched back to PIM mode for self-attention calculation. Therefore, it can be seen that, regarding the aforementioned... Figure 3-6 The large model shown contains 32 Transformer Block Layers in its network structure, and each Transformer Block Layer contains a self-attention layer. Therefore, the large model needs to go through 32 self-attention layers for inference for each token, which means it needs to perform 32 sets of first DRAM operation mode switching. Therefore, this embodiment is suitable for application scenarios with a small number of self-attention layers or where the switching speed between PIM mode and Normal mode is fast.

[0165] Figure 11 This is a flowchart of yet another embodiment of the data processing method based on in-memory processing mode provided in this application.

[0166] like Figure 11 As shown, this embodiment includes the following steps S31-S34.

[0167] S31: In the in-memory processing mode of the first dynamic random access memory, the first Q vector, the first K vector, and the first V vector are calculated using the first output information of the first dynamic random access memory based on the first output information of the preceding network.

[0168] For an explanation of step S31, please refer to step S11 above, which will not be repeated here.

[0169] S32: In the non-in-memory processing mode of the second dynamic random access memory, the first K vector, the first V vector, and the first Q vector are stored in the second dynamic random access memory.

[0170] The second DRAM is located in the non-PIM area of ​​the memory system.

[0171] The embodiments of this application can be regarded as Figure 4 In the illustrated embodiment, the second DRAM and the first DRAM are different DRAMs. In this case, to enable the second DRAM to store data, it can be located in the non-PIM area of ​​the memory system. The non-PIM area refers to the region where no computing units are integrated. In other words, the second DRAM in the non-PIM area does not integrate computing units, thus having conventional storage capabilities but no computing capabilities. This ensures that the second DRAM is not affected by the PIM calculation process and can write the first K vector, the first V vector, and the first Q vector.

[0172] In this embodiment, the second DRAM is placed in the non-PIM region, ensuring that the second DRAM always remains in non-PIM mode, thus enabling successful write operations to the first K vector and the first V vector. With this design, this embodiment eliminates the need to switch the first DRAM back to non-PIM mode, reducing the time overhead caused by DRAM operation mode switching.

[0173] It is understood that non-PIM mode refers to modes other than PIM mode. For details, please refer to the introduction of non-PIM mode in step S12, which will not be repeated here.

[0174] In some embodiments, step S32 includes the following steps:

[0175] S321: Store the first K vector and the first V vector into the key-value cache.

[0176] For an explanation of step S321, please refer to step S121 above, which will not be repeated here.

[0177] S33: In the in-memory processing mode of the first dynamic random access memory, self-attention calculation is performed based on the first Q vector and the second K vector and the second V vector stored in the second dynamic random access memory to obtain the second output information corresponding to the self-attention layer.

[0178] For an explanation of step S33, please refer to step S13 above, which will not be repeated here.

[0179] In some embodiments, step S33 includes the following steps:

[0180] S331: Using a neural processing unit or graphics processing unit integrated in a system-on-a-chip, read the second K vector and the second V vector from the second dynamic random access memory.

[0181] There is a communication connection between the on-chip system and the second dynamic random access memory.

[0182] In this embodiment, for the aforementioned scheme where the first DRAM and the second DRAM are different DRAMs, considering that the second DRAM is a non-PIM region DRAM and always remains in non-PIM mode, computational operations cannot be performed within the second DRAM. Therefore, the SoC can read the second K vector and the second V vector stored in the second DRAM and perform self-attention computation. The SoC in this implementation integrates an NPU or GPU, both of which are processors with computing capabilities. There is a communication connection between the SoC and the second DRAM, and the NPU or GPU can utilize this communication connection to read the second K vector and the second V vector from the second DRAM.

[0183] S332: Using a neural processing unit or a graphics processing unit, self-attention calculation is performed based on the first Q vector and the read second K vector and second V vector to obtain the second output information corresponding to the self-attention layer.

[0184] In this embodiment, self-attention calculation is performed on the self-attention layer. Specifically, since the second DRAM has no computational function, self-attention calculation can be performed on the second K vector and the second V vector stored in the second DRAM through the NPU or GPU integrated in the SoC. This solves the problem that the second DRAM cannot perform self-attention calculation operations because it is located in a non-PIM area.

[0185] Furthermore, for an explanation of the first Q vector, the second K vector, and the second V vector during the self-attention calculation process, please refer to step S13 above, which will not be repeated here.

[0186] It should be noted that, since the self-attention calculation in this embodiment is performed using an NPU or GPU integrated in the SoC, after the first Q vector is calculated in step S31, it can be directly stored in the SoC instead of the second DRAM. Furthermore, after performing self-attention calculation using the first Q vector, the first Q vector stored in the SoC can be deleted.

[0187] S34: Input the second output information into the subsequent network.

[0188] For an explanation of step S34, please refer to step S14 above, which will not be repeated here.

[0189] In one implementation, the embodiments of this application may further include the following steps:

[0190] S35: Process the second output information using a subsequent network.

[0191] S36: Based on the fact that the processed second output information meets the inference termination condition, the processed second output information is determined to be the target output information of the neural network model. After this, the inference operation of the neural network model can be terminated.

[0192] S37: Since the processed second output information does not meet the reasoning termination condition, new input information is determined based on the processed second output information.

[0193] S38: Input the new input information into the preceding network to obtain the new first output information. After that, you can return to the previous step S31.

[0194] For an explanation of steps S35-S38, please refer to steps S15-S18 above, which will not be repeated here.

[0195] Figure 11 The illustrated embodiment includes a first DRAM in the PIM region and a second DRAM in a non-PIM region. Therefore, the first DRAM can be kept in PIM mode for computational operations, while the second DRAM can be kept in non-PIM mode for storing the first K vector, first V vector, and first Q vector. With this design, the embodiments of this application do not require frequent switching between the operating modes of the first and second DRAMs. Therefore, this not only solves the problem of the inability to write the first K vector, first V vector, and first Q vector when the DRAM is in PIM mode during the PIM computation process, but also reduces the time overhead caused by DRAM operating mode switching.

[0196] The following is combined with Figure 12-13 ,right Figure 11 The illustrated embodiment will be further explained.

[0197] Figure 12 Is adopted Figure 11 The illustrated embodiment is a flowchart illustrating the execution of large model inference operations; Figure 13 yes Figure 12 The corresponding data storage diagram.

[0198] This embodiment uses a second DRAM in a non-PIM area to disable the PIM function.

[0199] In this embodiment, the DRAM includes a first DRAM located in the PIM region (as shown in Figures 12 and 13, PIM-DRAM) and a second DRAM located in the non-PIM region (as shown in Figures 12 and 13, Normal-DRAM). The Normal-DRAM does not have computational functionality, while the PIM-DRAM does. In actual operation, the computational function of the PIM-DRAM can be controlled by switching its operating mode. For example, switching the PIM-DRAM to PIM mode enables its computational function. In this embodiment, the PIM-DRAM can be set to PIM mode. In this scheme, the PIM-DRAM is used to store the weight parameters of the neural network model, and the Normal-DRAM is used to store the first K vector and the first V vector.

[0200] like Figure 13 As shown, before performing inference operations using this embodiment, the weight parameters of the neural network model are pre-stored in PIM-DRAM (i.e., the first DRAM). During model inference, the weight parameters can be directly read from the PIM-DRAM.

[0201] like Figure 12 As shown, when performing model inference operations using this embodiment, the PIM-DRAM is first switched to PIM mode, and DRAM-PIM technology is used to perform pre-processing network calculations before the self-attention layer. Subsequently, the self-attention layer calculates the latest Q vector, vector K, and V vectors based on the output of the pre-processing network, which are the first Q vector, the first K vector, and the first V vector.

[0202] Since DRAM-PIM technology executes the PIM calculation process through DRAM, enabling dynamic random access memory to perform calculation operations, DRAM-PIM technology supports calculation operations within PIM-DRAM. Therefore, the calculation of the preceding network and the calculation of the first Q vector, the first K vector, and the first V vector can all be performed within PIM-DRAM, which can reduce data interaction between PIM-DRAM and the processor and improve model inference efficiency.

[0203] Next, the calculated first K vector and first V vector are written into the Normal-DRAM (i.e., the second DRAM) in the non-PIM area, such as... Figure 13As shown, the data can be written to the KV-Cache located in the Normal-DRAM. This scheme uses the PIM-DRAM in the PIM area to perform calculation operations and uses the Normal-DRAM in the non-PIM area to perform storage operations on the first K vector and the first V vector, thus avoiding the problem that the PIM-DRAM cannot write dynamic parameters (such as the first K vector and the first V vector) in PIM mode during the PIM calculation process.

[0204] Finally, the second K vector and the second V vector stored in Normal-DRAM are read using the NPU or GPU integrated on the SoC. Self-attention calculation is performed based on the read vectors, and subsequent network operations after the self-attention layer are performed using PIM-DRAM based on the self-attention calculation results. This step utilizes the NPU or GPU to perform the self-attention calculation operation, thus solving the problem that Normal-DRAM lacks computational capabilities and cannot perform computational operations.

[0205] Furthermore, after subsequent network operations, it can be determined whether the inference has ended. If it has ended, the current operation result can be directly used as the output of the neural network model, i.e., the target output information. If it has not ended, it returns to the preceding network operation before the attention layer to perform the next round of inference operations until the inference ends.

[0206] In summary, in this embodiment, the PIM-DRAM is used for computational operations, and the Normal-DRAM is used for storage operations of the first K vector and the first V vector, without needing to write the first K vector and the first V vector into the PIM-DRAM. Therefore, the PIM-DRAM in this embodiment does not need to switch between PIM mode and Normal mode, thus solving the problem of... Figure 9-10 The illustrated embodiment suffers from the problem of excessive switching of the first DRAM's operating mode.

[0207] Furthermore, in this embodiment, the self-attention calculation is performed by the NPU or GPU. During the self-attention calculation process, the NPU or GPU needs to read the second K vector and the second V vector stored in the second DRAM. When the context length is too long (e.g., >4K), the number of second K vectors and second V vectors will also increase accordingly. At this time, the NPU or GPU needs to read a large amount of data from the second DRAM. It is understandable that when a large amount of data needs to be read from the second DRAM, a bandwidth bottleneck will occur, resulting in a slower data reading speed. In other words, in application scenarios with excessively long context lengths, this embodiment will affect the inference performance of large models due to the bandwidth limitation of the second DRAM. Therefore, this embodiment is more suitable for scenarios with relatively short context lengths.

[0208] Figure 14 This is a flowchart of another embodiment of the data processing method based on in-memory processing mode provided in this application.

[0209] like Figure 14 As shown, this embodiment includes the following steps S4a-S4i.

[0210] S4a: In the in-memory processing mode of the first dynamic random access memory, the first Q vector, the first K vector, and the first V vector of the self-attention layer in the neural network model are calculated using the first dynamic random access memory based on the first output information of the preceding network in the neural network model.

[0211] For an explanation of step S4a, please refer to step S11 above, which will not be repeated here.

[0212] S4b: In the non-in-memory processing mode of the second sub-dynamic random access memory, the first K vector and the first V vector are stored in the second sub-dynamic random access memory.

[0213] The first dynamic random access memory includes a first sub-dynamic random access memory and a second sub-dynamic random access memory, with the second sub-dynamic random access memory located in the non-memory processing area of ​​the memory system.

[0214] The embodiments of this application can be regarded as Figure 4 In the illustrated embodiment, the second DRAM and the first DRAM are the same DRAM. In this case, the first DRAM can be divided into two parts: a first sub-DRAM (first sub-Dynamic Random Access Memory) and a second sub-DRAM (second sub-Dynamic Random Access Memory). It is understood that the first DRAM, second DRAM, first sub-DRAM, and second sub-DRAM provided in this application embodiment can all be arbitrarily defined, rather than referring to a specific DRAM chip. For example, the first DRAM can be a single DRAM chip or composed of multiple DRAM chips. As another example, if the first DRAM includes DRAM chip a and DRAM chip b, then DRAM chip a can be divided into the first sub-DRAM, and DRAM chip b into the second sub-DRAM.

[0215] The first sub-DRAM is located in the PIM region; for an explanation of the PIM region, please refer to step S22 above. In PIM mode, the first sub-DRAM has computing capabilities; in non-PIM mode, the first sub-DRAM has storage capabilities. The second sub-DRAM is located in the non-PIM region, which is an area without integrated computing units; therefore, the second sub-DRAM does not have computing capabilities.

[0216] In this embodiment, the second sub-DRAM can be kept in non-PIM mode to perform storage operations on the first K vector, first V vector, and first Q vector. This design eliminates the need for frequent switching of the first sub-DRAM's operating mode, thus solving the problem that the PIM-mode DRAM cannot write the first K vector, first V vector, and first Q vector during the PIM calculation process, and also reducing the time overhead caused by DRAM operating mode switching.

[0217] In some embodiments, the first sub-DRAM is also used to store weight parameters of the neural network model.

[0218] In some embodiments, step S4b includes the following steps:

[0219] S4b1: Store the first K vector and the first V vector into the second key-value buffer.

[0220] The second KV-Cache is located in the second sub-dynamic random access memory.

[0221] For an explanation of step S4b1, please refer to step S121 above, which will not be repeated here.

[0222] S4c: In the in-memory processing mode of the first dynamic random access memory, the second K vector is read from the first sub-dynamic random access memory and the second sub-dynamic random access memory respectively.

[0223] In this embodiment, during the self-attention calculation for the self-attention layer, the second K vector is first read from the first DRAM to calculate the self-attention score. Furthermore, considering that in subsequent step S4g, when the stopping condition is met, the third K vector and third V vector stored in the second sub-DRAM are moved to the first sub-DRAM, and then new first K vectors and first V vectors are stored, both the first and second sub-DRAMs can store a portion of the second K vector. In other words, the K vectors stored in the first and second sub-DRAMs are combined to obtain the second K vector. Based on this, this embodiment reads the second K vector from both the first and second sub-DRAMs to ensure the integrity of the read second K vector.

[0224] S4d: In the in-memory processing mode of the first dynamic random access memory, the self-attention score corresponding to the second K vector is obtained based on the first Q vector and the second K vector.

[0225] For an explanation of step S4d, please refer to step S131 above, which will not be repeated here.

[0226] S4e: In the in-memory processing mode of the first dynamic random access memory, the second V vector is read from the first sub-dynamic random access memory and the second sub-dynamic random access memory respectively.

[0227] In this embodiment, during the self-attention calculation process for the self-attention layer, a second V vector is read from the first DRAM to perform a weighted calculation using the self-attention score, thereby obtaining the second output information of the self-attention layer. This embodiment reads the second V vector from both the first and second sub-DRAMs to ensure the integrity of the read second V vector; the principle is the same as step S4c and will not be repeated here.

[0228] S4f: In the in-memory processing mode of the first dynamic random access memory, the second output information corresponding to the self-attention layer is obtained based on the self-attention score and the second V vector.

[0229] For an explanation of step S4f, please refer to step S132 above, which will not be repeated here.

[0230] Furthermore, the aforementioned steps S4c-S4f are all performed in the PIM mode of the first DRAM, which can improve the computational efficiency of self-attention and increase the inference speed of large models.

[0231] S4g: Based on the second sub-dynamic random access memory meeting the preset stopping condition, in the non-in-memory processing mode of the first sub-dynamic random access memory, the third K vector and the third V vector stored in the second sub-dynamic random access memory are stored in the first sub-dynamic random access memory.

[0232] Among them, the third K vector is all the K vectors stored in the second sub-dynamic random access memory, and the third V vector is all the V vectors stored in the second sub-dynamic random access memory.

[0233] In this embodiment, considering that the second sub-DRAM lacks computing power, the NPU or GPU integrated on the SoC needs to read the data stored in the second sub-DRAM for computation. However, reading large amounts of data from the second sub-DRAM can cause a bandwidth bottleneck, affecting computational efficiency. Therefore, this application sets a stopping condition. When the stopping condition is met, all K vectors (i.e., the third K vector) and all V vectors (i.e., the third V vector) stored in the second sub-DRAM are moved to the first sub-DRAM so that the first sub-DRAM, which has computing power, can perform computations on the K and B vectors moved to the first sub-DRAM. It should be noted that, in order to ensure that the first sub-DRAM can write data smoothly, the first sub-DRAM can be adjusted to a non-PIM mode, such as adjusting it to Normal mode.

[0234] This application embodiment solves the problem that reading large amounts of data from the second sub-DRAM can reduce computational efficiency by moving the data to the first sub-DRAM for computation, and can maintain high inference performance even in application scenarios with long context lengths.

[0235] Furthermore, the stopping condition can be preset. For example, if the number of self-attention calculations reaches a preset number, the stopping condition can be considered met, the third K vector and the third V vector are moved to the first sub-DRAM, and the number of self-attention calculations is reset to zero and restarted. In some embodiments, the stopping condition can be that the number of third K vectors in the second sub-DRAM reaches a preset threshold; in other embodiments, the stopping condition can be that the number of third V vectors in the second sub-DRAM reaches a preset threshold; in still other embodiments, the numbers of the third K vector and the third V vector can be analyzed comprehensively, and the stopping condition can be considered met when both reach the threshold.

[0236] In some embodiments, step S4g includes the following steps:

[0237] S4g1: Store the third K vector and the third V vector into the first key-value buffer.

[0238] The first key-value cache is located in the first sub-dynamic random access memory.

[0239] For an explanation of step S4g1, please refer to step S421 above, which will not be repeated here.

[0240] S4h: Delete the third K vector and the third V vector stored in the second sub-dynamic random access memory, and the memory system switches the first sub-dynamic random access memory back to in-memory processing mode.

[0241] In this embodiment, after storing the third K vector and the third V vector in the first sub-DRAM, the third K vector and the third V vector stored in the second sub-DRAM can be deleted to provide storage space for the new first K vector and the first V vector. This also avoids the NPU or GPU integrated on the SoC reading too much data from the second sub-DRAM. Furthermore, the first sub-DRAM is switched back to PIM mode so that DRAM-PIM technology can be used in subsequent computational operations, utilizing the first sub-DRAM to perform computational operations.

[0242] S4i: Input the second output information into the subsequent network.

[0243] For an explanation of step S4i, please refer to step S14 above, which will not be repeated here.

[0244] In one implementation, the embodiments of this application may further include the following steps:

[0245] S4j: Uses a post-processing network to process the second output information.

[0246] S4k: Based on the processed second output information reaching the inference termination condition, the processed second output information is determined to be the target output information of the large model. After this, the inference operation of the neural network model can be terminated.

[0247] S4l: Since the processed second output information does not meet the reasoning termination condition, new input information is determined based on the processed second output information.

[0248] S4m: Input the new input information into the preceding network to obtain the new first output information. After that, you can return to the previous step S4a.

[0249] Please refer to steps S15-S18 above for steps S4j-S4m, which will not be repeated here.

[0250] Figure 14 The illustrated example sets up a first sub-DRAM in the PIM area and a second sub-DRAM in a non-PIM area. The calculated first K vector and first V vector are stored in the second sub-DRAM. Therefore, while successfully storing the first K vector and first V vector, it also avoids the first sub-DRAM frequently switching operating modes for writing data. Furthermore, in this embodiment, when the number of KV vectors in the second sub-DRAM is too large, the KV vectors are moved to the first sub-DRAM in non-PIM mode, which avoids the excessive number of KV vectors in the second sub-DRAM reducing the inference speed of the neural network model.

[0251] The implementation schemes for steps S4c-S4f will be further described below.

[0252] In some embodiments, step S4c may include the following steps S4c1-S4c2, step S4d may include the following steps S4d1-S4d3, step S4e may include the following steps S4e1-S4e2, and step S4f may include the following steps S4f1-S4f3.

[0253] S4c1: In the in-memory processing mode of the first sub-dynamic random access memory, the fourth K vector stored in the first sub-dynamic random access memory is read using the computing unit.

[0254] The computing unit is integrated in the first sub-dynamic random access memory.

[0255] In this embodiment, the fourth K-vector is read from the first sub-DRAM to perform self-attention calculation. Since the first sub-DRAM is located in the PIM region, a computing unit (PU) is integrated within it. In PIM mode, the PU integrated in the first sub-DRAM can be used to read and calculate the fourth K-vector.

[0256] In some embodiments, the fourth K vector is stored in the first key cache.

[0257] In this embodiment, the first sub-DRAM stores a first KV-Cache, and the fourth K vector and the fourth V vector are stored in the first KV-Cache. The principle and function of the first KV-Cache are similar to those of the KV-Cache in step S121 above, and will not be repeated here. Further, the first KV-Cache includes a first K-Cache (first key cache) and a first V-Cache (first value cache), and the fourth K vector is stored in the first K-Cache. This separate storage of the fourth K vector and the fourth V vector allows for fine-grained control of the fourth K vector and the fourth V vector, and is beneficial for optimizing memory usage.

[0258] S4c2: In the non-in-memory processing mode of the second sub-dynamic random access memory, the fifth K vector stored in the second sub-dynamic random access memory is read using the neural processing unit or graphics processing unit integrated in the system-on-chip.

[0259] There is a communication connection between the on-chip system and the second sub-dynamic random access memory.

[0260] In this embodiment, the fifth K-vector is read from the second sub-DRAM to perform self-attention calculation. Since the second sub-DRAM is located in the non-PIM region, it does not have computational capabilities; that is, the second sub-DRAM can only operate in non-PIM mode. In the non-PIM mode of the second sub-DRAM, a processor with computational capabilities can be used to read and calculate the fourth K-vector. This processor can be an NPU or GPU, and can be integrated into an integrated circuit such as a SoC. Furthermore, to facilitate the reading of the fifth K-vector from the second sub-DRAM, a communication connection can be established between the SoC and the second sub-DRAM.

[0261] In some embodiments, the fifth K vector is stored in the second key cache.

[0262] In this embodiment, the second sub-DRAM stores a second KV-Cache, and the fifth K vector and the fifth V vector are stored in the second KV-Cache. Further, the second KV-Cache includes a second K-Cache (second key cache) and a second V-Cache (second value cache), with the fifth K vector stored in the second K-Cache. Its principle and function are similar to those of storing the fourth K vector in the first K-Cache in step S4c1, and will not be repeated here.

[0263] It can be understood that the second K vector is the combination of the fourth K vector and the fifth K vector. Therefore, by reading the fourth K vector and the fifth K vector through the aforementioned steps, the second K vector is also read.

[0264] S4d1: In the in-memory processing mode of the first sub-dynamic random access memory, the first Q vector and the fourth K vector are processed using the first sub-dynamic random access memory to obtain the first sub-self-attention score corresponding to the fourth K vector.

[0265] In this embodiment, since the first sub-DRAM in PIM mode has computing capabilities, after the PU reads the fourth K vector from the first sub-DRAM, it directly uses the PU to calculate the self-attention score within the first sub-DRAM. Specifically, the first Q vector obtained in the aforementioned steps can be used to perform a dot product with the read fourth K vector to obtain the corresponding first sub-attention score.

[0266] S4d2: In the non-in-memory processing mode of the second sub-dynamic random access memory, the first Q vector and the fifth K vector are processed by the neural processing unit or the graphics processing unit to obtain the second sub-self-attention score corresponding to the fifth K vector.

[0267] In this embodiment, since the second sub-DRAM in non-PIM mode does not have computing power, self-attention calculation can be performed within the SoC after the NPU or GPU reads the fifth K vector from the second sub-DRAM. The calculation process is similar to the aforementioned step S4d1 and will not be described again here.

[0268] S4d3: Concatenate the first sub-self-attention score and the second sub-self-attention score to obtain the self-attention score corresponding to the second K vector.

[0269] In the embodiments of this application, it can be understood that since the second K vector is the combination of the fourth K vector and the fifth K vector, the first sub-attention score obtained based on the fourth K vector and the second sub-attention score obtained based on the fifth K vector are concatenated to obtain the total self-attention score corresponding to the second K vector.

[0270] S4e1: In the in-memory processing mode of the first sub-dynamic random access memory, the fourth V vector stored in the first sub-dynamic random access memory is read using the computing unit.

[0271] In this embodiment of the application, the fourth V vector in the first sub-DRAM is read to perform self-attention calculation. The principle and function are similar to the aforementioned step S4c1, and will not be repeated here.

[0272] In some embodiments, the fourth V vector is stored in the first value buffer.

[0273] In this embodiment, the principle and function of storing the fourth V vector in the first V-Cache are similar to those of storing the fourth K vector in the first K-Cache in the aforementioned step S4c1, and will not be repeated here.

[0274] S4e2: In the non-in-memory processing mode of the second sub-dynamic random access memory, the fifth V vector stored in the second sub-dynamic random access memory is read using the neural processing unit or the graphics processing unit.

[0275] In this embodiment of the application, the fifth V vector in the second sub-DRAM is read to perform self-attention calculation using the fifth V vector. Its principle and function are similar to the aforementioned step S4c2, and will not be repeated here.

[0276] In some embodiments, the fifth V vector is stored in the second value buffer.

[0277] In this embodiment, the principle and function of storing the fifth V vector in the second V-Cache are similar to those of storing the fifth K vector in the second K-Cache in the aforementioned step S4c2, and will not be repeated here.

[0278] It can be understood that the second V vector is the combination of the fourth V vector and the fifth V vector. Therefore, by reading the fourth V vector and the fifth V vector through the aforementioned steps, the second V vector is also read.

[0279] S4f1: In the in-memory processing mode of the first sub-dynamic random access memory, the self-attention score and the fourth V vector are processed using the first sub-dynamic random access memory to obtain the first sub-output information.

[0280] In this embodiment of the application, since the first sub-DRAM in PIM mode has computing capabilities, after the PU reads the fourth V vector from the first sub-DRAM, it directly uses the PU to perform weighted calculation of self-attention scores within the first sub-DRAM to obtain the corresponding first sub-output information.

[0281] S4f2: In the non-in-memory processing mode of the second sub-dynamic random access memory, the second sub-output information is obtained based on the self-attention score and the fifth V vector using a neural processing unit or a graphics processing unit.

[0282] In this embodiment, since the second sub-DRAM in non-PIM mode does not have computing power, self-attention calculation can be performed within the SoC after the NPU or GPU reads the fifth V vector from the second sub-DRAM. The calculation process is similar to the aforementioned step S4f1 and will not be repeated here.

[0283] S4f3: Concatenate the first sub-output information and the second sub-output information to obtain the second output information corresponding to the self-attention layer.

[0284] In the embodiments of this application, it can be understood that since the second V vector is the combination of the fourth V vector and the fifth V vector, the first sub-output information obtained based on the fourth V vector and the second sub-output information obtained based on the fifth V vector are concatenated to form the total second output information corresponding to the self-attention layer.

[0285] The following is combined with Figure 15-16 ,right Figure 14 The illustrated embodiment will be further explained.

[0286] Figure 15 Is adopted Figure 14 The illustrated embodiment is a flowchart illustrating the execution of large model inference operations; Figure 16 yes Figure 15 The corresponding data storage diagram.

[0287] In this embodiment, a first sub-DRAM and a second sub-DRAM are respectively set up, and the second sub-DRAM is a non-PIM area DRAM, that is, the second sub-DRAM does not have computing power.

[0288] In this embodiment, the DRAM includes a first sub-DRAM located in the PIM region (e.g., Figure 16 , 17 The PIM-DRAM shown) and the second sub-DRAM located in the non-PIM area (such as Figure 16 , 17The diagram shows the Normal-DRAM. The first sub-DRAM has computational capabilities, while the second sub-DRAM does not. In actual operation, the computational function can be enabled or disabled by switching the operating mode of the first sub-DRAM. For example, switching the first sub-DRAM to PIM mode enables its computational function. In this embodiment, the second sub-DRAM can be set to a non-PIM mode. In this embodiment, the first sub-DRAM stores model parameters and the third K-vector and third V-vector of the second sub-DRAM batch movement, while the second sub-DRAM stores the calculated first K-vector and first V-vector.

[0289] like Figure 16 As shown, before performing inference operations using this embodiment, the model's weight parameters are pre-stored in the first sub-DRAM (e.g., Figure 16 The weight parameters can be read directly from the first sub-DRAM during model inference.

[0290] like Figure 15 As shown, when performing inference operations on the neural network model using this embodiment, the first sub-DRAM is first switched to PIM mode, and DRAM-PIM technology is used to perform pre-processing network calculations before the self-attention layer using the first sub-DRAM. Subsequently, based on the output of the pre-processing network, the first sub-DRAM calculates the latest Q-vector, K-vector, and V-vector in the self-attention layer, which are the first Q-vector, first K-vector, and first V-vector.

[0291] Since DRAM-PIM technology supports PIM calculations being performed by DRAM, enabling computational operations to be executed within dynamic random access memory (DRAM), the preceding network computation and the calculations of the first Q vector, first K vector, and first V vector can all be performed within the first sub-DRAM. This embodiment utilizes the first sub-DRAM of the PIM region to perform computational operations, reducing data interaction between the first sub-DRAM and the processor and improving model inference efficiency.

[0292] Secondly, such as Figure 16 As shown in (b), the calculated first K vector and first V vector are written into the second sub-DRAM in the non-PIM area. Specifically, as... Figure 16 As shown in (a), the data can be written to the second K-Cache and the second K-Cache located in the second sub-DRAM, respectively. In this embodiment, the storage operations of the first K vector and the first V vector are performed using the second sub-DRAM in the non-PIM area, which can avoid the problem that the first DRAM cannot write dynamic parameters such as the first K vector and the first V vector in PIM mode.

[0293] After that, as Figure 15 and Figure 16 As shown in (a), the fifth K vector and the fifth V vector are read from the second sub-DRAM using an NPU or GPU integrated on the SoC. Self-attention calculation is then performed based on the read data to address the issue of the second sub-DRAM being unable to perform computational operations. Simultaneously, if the first sub-DRAM stores the fourth K vector and the fourth V vector, then... Figure 15 and Figure 16 As shown in (a), self-attention calculation is performed based on the stored fourth K vector and fourth V vector in the first sub-DRAM, and the self-attention calculation results (i.e., the first sub-output information) corresponding to the first sub-DRAM and the self-attention calculation results (i.e., the second sub-output information) corresponding to the second sub-DRAM are combined to obtain the total self-attention calculation result, i.e. the second output information.

[0294] After completing the self-attention calculation, the first sub-DRAM is used to perform subsequent network operations after the self-attention layer based on the self-attention calculation results.

[0295] Furthermore, after subsequent network operations, it can be determined whether the inference has ended. If it has ended, the current operation result can be directly used as the output of the neural network model; if it has not ended, the next round of inference operation needs to be performed until the inference ends.

[0296] Before the next round of inference, it is first determined whether the number of the third K vector and / or third V vector in the second sub-DRAM has reached a preset threshold n. If it has reached n, the third K vector and third V vector in the second sub-DRAM can be moved to the first sub-DRAM. Specifically, the first sub-DRAM is switched to Normal mode to ensure that data can be written to it. Then, the third K vector and third V vector stored in the second sub-DRAM are written to the first sub-DRAM, specifically to the first K-Cache and the first V-Cache, respectively. Afterward, the third K vector and third V vector stored in the second sub-DRAM are cleared so that the second sub-DRAM can store new first K vectors and first V vectors in the next round of inference. After the third K vector and third V vector are written to the first sub-DRAM, the first sub-DRAM is switched to PIM mode so that computational operations can be performed using the first sub-DRAM in the next round of inference.

[0297] In summary, in this embodiment, the first sub-DRAM is used for computational operations, and the second sub-DRAM is used for storing the first K vector and the first V vector. When the number of stored vectors reaches n, they are moved to the first sub-DRAM. Therefore, this embodiment eliminates the need to write the first K vector and the first V vector into the first sub-DRAM for each inference iteration, and also eliminates the need to switch the first sub-DRAM from PIM mode to Normal mode for each inference iteration, thus solving the problem of... Figure 9 The illustrated embodiment suffers from excessive DRAM mode switching. Furthermore, this embodiment utilizes the first sub-DRAM to perform self-attention calculations on the fourth K vector and fourth V vector moved to the first sub-DRAM, and utilizes the NPU or GPU to perform self-attention calculations on the fifth K vector and fifth V vector stored in the second sub-DRAM, thus solving the problem. Figure 12 The scheme shown requires the NPU or GPU to read a large number of second K vectors and second V vectors from the second DRAM to the SoC, which causes the inference performance to be affected by the DRAM bandwidth.

[0298] The following description uses an application scenario to illustrate the embodiments of this application.

[0299] Taking the inference process based on the Llama2 7B model as an example, this model is a large language model with 7 billion parameters and 32 transformer blocks, each with a self-attention layer. Assuming a scenario has 1000 tokens as input and 500 tokens as output, an update is performed every 10 tokens generated, moving the K and V vectors stored in the second KV-Cache to the first KV-Cache. First, the prefill stage is calculated, where a first K-Cache of 1000×4096 dimensions and a first V-Cache of 1000×4096 dimensions are calculated in the self-attention layer of each block, and the first output token is generated. Then, the generated token is used as input for model inference. The calculation process is illustrated using one block as an example. After calculation, the input vector yields a 4096-dimensional first Q vector, first K vector, and first V vector. Next, in the first sub-DRAM, the first Q vector is dot-producted with the first K-Cache to obtain a vector of dimension 1000 (i.e., the first sub-attention score). This is then dot-producted with the first V-Cache to obtain a 4096-dimensional vector (i.e., the first sub-output information). In the NPU, the first Q vector is dot-producted with the first K vector to obtain a vector of dimension 1 (i.e., the second sub-attention score). This is then multiplied with the first V vector to obtain a 4096-dimensional vector (i.e., the second sub-output information). The first sub-output information obtained from the first sub-DRAM is added to the second sub-output information obtained from the NPU to obtain the subsequent calculation result, i.e., the second output information. After inference is completed, the first K vector and the first V vector are updated in the second KV-Cache. At this point, both the second K-Cache and the second V-Cache have a dimension of 1×4096.

[0300] During the second token inference, the input vector is also calculated to obtain new 4096-dimensional first Q vector, first K vector, and first V vector. Then, in the first sub-DRAM, the new first Q vector is multiplied by the first K-Cache to obtain a vector of dimension 1000 (i.e., the new first sub-attention score), which is then multiplied by the first V-Cache to obtain a 4096-dimensional vector (i.e., the new first sub-output information). The second KV-Cache is read into the NPU, where the new first Q vector is multiplied by the 2×4096-dimensional matrix (concatenated with the new first K vector and the second K-Cache) to obtain a vector of dimension 2 (i.e., the new second sub-attention score), which is then multiplied by the 2×4096-dimensional matrix (concatenated with the new first V vector and the second V-Cache) to obtain a 4096-dimensional vector (i.e., the new second sub-output information). The new first sub-output information obtained from the first sub-DRAM is added to the new second sub-output information obtained from the NPU to obtain the subsequent calculation result, i.e., the new second output information. After the inference is completed, the new first K vector and the new first V vector are updated into the second KV-Cache. At this time, the dimensions of the second K-Cache and the second V-Cache become 2×4096.

[0301] After 10 rounds of inference, the dimensions of the second K-Cache and the second V-Cache become 10×4096. At this point, the first sub-DRAM is switched to Normal mode, and the second K-Cache and the second V-Cache are merged into the first K-Cache and the first V-Cache. The dimensions of the first K-Cache and the first V-Cache change from 1000×4096 to 1010×4096. The second K-Cache and the second V-Cache are cleared. Then, the first sub-DRAM is switched to PIM mode, and the subsequent inference process continues until the inference termination condition is met, obtaining the target output information of the large model.

[0302] The following is a summary of this application. Figure 9 , Figure 12 and Figure 15 Further comparisons are made with the embodiments shown.

[0303] For the aforementioned Llama2 7B model, assuming a context length of 8K, Llama2 7B parameters 4-bit quantization, KV-Cache 8-bit quantization, a maximum context length of 4K, a DRAM switching time overhead of 100µs between PIM and Normal modes, a DRAM bandwidth of 68GB / s, and a PIM internal bandwidth (i.e., the rate at which data is transferred from one memory bank to another or a compute unit PU within DRAM) that is 8 times the DRAM bandwidth, then, if the following is adopted... Figure 9 The embodiment shown has an inference speed of approximately 47.3 tokens / s for large models. If a method such as... Figure 12 The embodiment shown has an inference speed of approximately 68 tokens / s. If a method such as... Figure 15 In the embodiment shown, the second KV-Cache is updated to the first KV-Cache every 10 tokens inferred, and the inference speed can reach 120 tokens / s.

[0304] It should be noted that the embodiments of this application can be applied to electronic devices with computing functions, and corresponding behaviors exist during application. For example, a mobile phone is equipped with PIM-DRAM hardware. During large model inference, DRAM read behavior is captured and the DRAM read bandwidth is determined. If the following behaviors exist, it can be considered that the solution of the embodiments of this application has been adopted: the read bandwidth gradually increases, and DRAM mode switching actions occur periodically, and data write behaviors that conform to the PIM working mode occur periodically to write data to specific locations in the DRAM.

[0305] Other embodiments of this application provide a data processing apparatus based on an in-memory processing mode.

[0306] Figure 17 This is a schematic diagram of a data processing device based on an in-memory processing mode provided in an embodiment of this application.

[0307] like Figure 17As shown, the data processing device based on in-memory processing mode may include: a display screen 1001, a memory 1002, a processor 1003, and a communication module 1004. These devices can be connected via one or more communication buses 1005. The display screen 1001 may include a display panel 10011 and a touch sensor 10012. The display panel 10011 is used to display images, and the touch sensor 10012 can transmit detected touch operations to the application processor to determine the touch event type, providing visual output related to the touch operation through the display panel 10011. The processor 1003 may include one or more processing units, such as: an application processor, a modem processor, a graphics processor, an image signal processor, a controller, a video codec, a digital signal processor, a baseband processor, and / or a neural network processor. Different processing units may be independent devices or integrated into one or more processors. The memory 1002 is coupled to the processor 1003 and is used to store various software programs and / or computer instructions. The memory 1002 may include volatile memory and / or non-volatile memory. When the processor executes computer instructions, the data processing device based on the in-memory processing mode can perform the various functions or steps performed in the above method embodiments.

[0308] When the software program and / or multiple sets of instructions in the memory 1002 are executed by the processor 1003, the data processing device based on the in-memory processing flow implements the following method steps: In the in-memory processing mode of the first dynamic random access memory, the first Q vector, first K vector, and first V vector of the self-attention layer in the neural network model are calculated using the first dynamic random access memory based on the first output information of the preorder network in the neural network model. Here, the first dynamic random access memory is a storage area in the memory system, the preorder network is the network located before the self-attention layer, the first Q vector is used to identify the query vector corresponding to the first output information, and the first K vector and first V vector are used to identify the features corresponding to the first output information. Vectors; in the non-in-memory processing mode of the second dynamic random access memory, the first K vector, the first V vector, and the first Q vector are stored in the second dynamic random access memory; in the in-memory processing mode of the first dynamic random access memory, self-attention calculation is performed based on the first Q vector and the second K vector and the second V vector stored in the second dynamic random access memory to obtain the second output information corresponding to the self-attention layer, wherein the second K vector contains the first K vector, and the second V vector contains the first V vector; the second output information is input into the post-processing network in the neural network model, wherein the post-processing network is the network located after the self-attention layer, and the post-processing network is used to obtain the target output information of the large model based on the second output information.

[0309] This application also provides an electronic device, including: a processor and a memory; the processor may include one or more processing units, and the memory stores program instructions, which, when executed by the processor, cause the electronic device to perform a data processing method based on in-memory processing mode in any of the above embodiments.

[0310] This application also provides a chip system including at least one processor and at least one interface circuit. The processor and the interface circuit are interconnected via lines. For example, the interface circuit can be used to receive signals from other devices (e.g., the memory of an electronic device). Or, for example, the interface circuit can be used to send signals to other devices. Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application does not specifically limit this.

[0311] This application also provides a computer-readable storage medium including computer instructions that, when executed on the electronic device, cause the electronic device to perform the various functions or steps described in the method embodiments.

[0312] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps performed by the mobile phone in the above method embodiments.

[0313] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0314] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0315] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0316] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0317] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0318] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method based on in-memory processing flow, characterized in that, The method includes: In the in-memory processing mode of the first dynamic random access memory, the first Q vector, first K vector, and first V vector of the self-attention layer in the neural network model are calculated based on the first output information of the preceding network in the neural network model using the first dynamic random access memory. The first dynamic random access memory is a storage area in the memory system, the preceding network is a network located before the self-attention layer, the first Q vector is used to identify the query vector corresponding to the first output information, and the first K vector and the first V vector are used to identify the feature vector corresponding to the first output information. In the non-in-memory processing mode of the second dynamic random access memory, the first K vector, the first V vector, and the first Q vector are stored in the second dynamic random access memory; In the in-memory processing mode of the first dynamic random access memory, self-attention calculation is performed based on the first Q vector and the second K vector and the second V vector stored in the second dynamic random access memory to obtain the second output information corresponding to the self-attention layer, wherein the second K vector contains the first K vector and the second V vector contains the first V vector. The second output information is input into the subsequent network of the neural network model, wherein the subsequent network is located after the self-attention layer, and the subsequent network is used to obtain the target output information of the neural network model based on the second output information.

2. The method according to claim 1, characterized in that, The second dynamic random access memory and the first dynamic random access memory are the same dynamic random access memory, and the second dynamic random access memory is located in the memory processing area of ​​the memory system.

3. The method according to claim 1, characterized in that, The second dynamic random access memory and the first dynamic random access memory are different dynamic random access memories, and the second dynamic random access memory is located in the non-memory processing area of ​​the memory system.

4. The method according to claim 3, characterized in that, The self-attention calculation based on the first Q vector and the second K vector and second V vector stored in the second dynamic random access memory includes: Using a neural processing unit or graphics processing unit integrated on a system-on-a-chip, the second K vector and the second V vector are read from the second dynamic random access memory, wherein there is a communication connection between the system-on-a-chip and the second dynamic random access memory; Self-attention is calculated using the neural processing unit or the graphics processing unit based on the first Q vector and the read second K vector and second V vector.

5. The method according to any one of claims 1 to 4, characterized in that, The self-attention calculation based on the first Q vector and the second K vector and second V vector stored in the second dynamic random access memory to obtain the second output information corresponding to the self-attention layer includes: The second K vector is read from the second dynamic random access memory, and the self-attention score corresponding to the second K vector is obtained based on the first Q vector and the second K vector. The second V vector is read from the second dynamic random access memory, and the second output information corresponding to the self-attention layer is obtained based on the self-attention score and the second V vector.

6. The method according to any one of claims 1 to 4, characterized in that, The step of storing the first K vector, the first V vector, and the first Q vector into the second dynamic random access memory includes: The first K vector and the first V vector are stored in a key-value cache, wherein the key-value cache is located in the second dynamic random access memory, and the key-value cache is a storage area that stores the first K vector and the first V vector in a key-value pair data structure.

7. The method according to claim 1, characterized in that, The first dynamic random access memory and the second dynamic random access memory are the same dynamic random access memory. The second dynamic random access memory includes a first sub-dynamic random access memory and a second sub-dynamic random access memory. The second sub-dynamic random access memory is located in the non-memory processing area of ​​the memory system. The step of storing the first K vector, the first V vector, and the Q vector into the second dynamic random access memory includes: In the non-in-memory processing mode of the second sub-dynamic random access memory, the first K vector and the first V vector are stored in the second sub-dynamic random access memory.

8. The method according to claim 7, characterized in that, After performing self-attention calculation based on the first Q vector and the second K vector and the second V vector stored in the second dynamic random access memory, the method further includes: Based on the second sub-dynamic random access memory meeting the preset stopping condition, in the non-in-memory processing mode of the first sub-dynamic random access memory, the third K vector and the third V vector stored in the second sub-dynamic random access memory are stored in the first sub-dynamic random access memory, wherein the third K vector is all K vectors stored in the second sub-dynamic random access memory, and the third V vector is all V vectors stored in the second sub-dynamic random access memory; The third K vector and the third V vector stored in the second sub-dynamic random access memory are deleted, and the memory system switches the first sub-dynamic random access memory back to the in-memory processing mode.

9. The method according to claim 8, characterized in that, The preset stopping condition is: the number of the third K vector in the second sub-dynamic random access memory and / or the number of the third V vector in the second sub-dynamic random access memory reaches a preset number threshold.

10. The method according to claim 8, characterized in that, The step of performing self-attention calculation based on the first Q vector and the second K vector and the second V vector stored in the second dynamic random access memory to obtain the second output information corresponding to the self-attention layer includes: The second K vector is read from the first sub-dynamic random access memory and the second sub-dynamic random access memory, respectively; Based on the first Q vector and the second K vector, the self-attention score corresponding to the second K vector is obtained; The second V vector is read from the first sub-dynamic random access memory and the second sub-dynamic random access memory, respectively; The second output information corresponding to the self-attention layer is obtained based on the self-attention score and the second V vector.

11. The method according to claim 10, characterized in that, The step of reading the second K vector from the first sub-dynamic random access memory and the second sub-dynamic random access memory respectively includes: In the in-memory processing mode of the first sub-dynamic random access memory, the fourth K vector stored in the first sub-dynamic random access memory is read by the computing unit, wherein the computing unit is integrated in the first sub-dynamic random access memory. In the non-in-memory processing mode of the second sub-dynamic random access memory, the fifth K vector stored in the second sub-dynamic random access memory is read using a neural processing unit or graphics processing unit integrated on the system-on-a-chip, wherein there is a communication connection between the system-on-a-chip and the second sub-dynamic random access memory. The second K vector is the union of the fourth K vector and the fifth K vector.

12. The method according to claim 11, characterized in that, The step of obtaining the self-attention score corresponding to the second K vector based on the first Q vector and the second K vector includes: In the in-memory processing mode of the first sub-dynamic random access memory, the first Q vector and the fourth K vector are processed using the first sub-dynamic random access memory to obtain the first sub-self-attention score corresponding to the fourth K vector; In the non-in-memory processing mode of the second sub-dynamic random access memory, the first Q vector and the fifth K vector are processed by the neural processing unit or the graphics processing unit to obtain the second sub-self-attention score corresponding to the fifth K vector; By concatenating the first sub-self-attention score and the second sub-self-attention score, the self-attention score corresponding to the second K vector is obtained.

13. The method according to claim 12, characterized in that, Reading the second V vector from the first sub-dynamic random access memory and the second sub-dynamic random access memory respectively includes: In the in-memory processing mode of the first sub-dynamic random access memory, the computing unit reads the fourth V vector stored in the first sub-dynamic random access memory. In the non-in-memory processing mode of the second sub-dynamic random access memory, the fifth V vector stored in the second sub-dynamic random access memory is read using the neural processing unit or the graphics processing unit. The second V vector is the combination of the fourth V vector and the fifth V vector.

14. The method according to claim 13, characterized in that, The step of obtaining the second output information corresponding to the self-attention layer based on the self-attention score and the second V vector includes: In the in-memory processing mode of the first sub-dynamic random access memory, the self-attention score and the fourth V vector are processed using the first sub-dynamic random access memory to obtain the first sub-output information; In the non-in-memory processing mode of the second sub-dynamic random access memory, the second sub-output information is obtained based on the self-attention score and the fifth V vector using the neural processing unit or the graphics processing unit. By concatenating the first sub-output information and the second sub-output information, the second output information corresponding to the self-attention layer is obtained.

15. The method according to claim 13, characterized in that, The first sub-dynamic random access memory includes a first key-value cache area, which is a storage area that stores the fourth K vector and the fourth V vector in a key-value pair data structure. The second sub-dynamic random access memory includes a second key-value cache, which is a storage area that stores the fifth K vector and the fifth V vector in a key-value pair data structure.

16. The method according to claim 15, characterized in that, The first key-value cache includes a first key cache and a first value cache, wherein the first key cache is used to store the fourth K vector and the first value cache is used to store the fourth V vector; The second key-value cache includes a second key cache and a second value cache, wherein the second key cache is used to store the fifth K vector and the second value cache is used to store the fifth V vector.

17. The method according to claim 6, characterized in that, The first sub-dynamic random access memory is also used to store the weight parameters of the neural network model.

18. The method according to claim 1, characterized in that, After inputting the second output information into the subsequent network, the method further includes: The second output information is processed using the subsequent network; Based on the fact that the reasoning termination condition is met after processing the second output information, the processed second output information is determined to be the target output information of the neural network model.

19. The method according to claim 18, characterized in that, The method further includes: Since the processed second output information does not meet the reasoning termination condition, new input information is determined based on the processed second output information; The new input information is input into the preceding network to obtain the new first output information, and then the process returns to the step of calculating the first Q vector, the first K vector, and the first V vector of the self-attention layer in the neural network model based on the first output information of the preceding network in the neural network model.

20. An electronic device, characterized in that, The device includes a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, the computer program code including computer instructions, and when the processor executes the computer instructions, it causes the electronic device to perform a data processing method based on an in-memory processing flow as described in any one of claims 1-19.

21. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or instructions that, when executed on a computer, cause the computer to perform a data processing method based on an in-memory processing flow as described in any one of claims 1-19.

22. A computer program product, characterized in that, The computer program product includes: a computer program or instructions that, when executed on a computer, cause the computer to perform a data processing method based on an in-memory processing flow as described in any one of claims 1-19.

Images