Near-memory computing cascading method based on distributed parameters and cache storage

Abstract

The present application relates to the technical field of computers, and provides a near-memory computing cascading method based on distributed parameters and cache storage. The present application adopts a distributed method to store large model parameters and cache (KV-Cache) data in an inference process through multiple cascaded near-memory computing chips and connected local memories; each near-memory computing chip obtains corresponding large model layered parameters and cache data from the local memory connected thereto, uses the data and input data generated by the previous stage to complete layered calculation of the corresponding large model decoding stage, so that distributed cascading inference calculation of the large model is completed under cooperation of the multiple cascaded near-memory computing chips. This scheme solves the problem that, due to huge parameter quantity, huge calculation quantity and huge cache data quantity, and high cost of special chips, the large model cannot meet the demand for large capacity, large concurrent large model inference.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a near-memory computing cascade method based on distributed parameters and cache storage. Background Technology

[0002] Key-Value Cache (KV-Cache) is a technique used to optimize the inference efficiency of Large Language Models (LLMs). By storing intermediate computation results, it significantly reduces redundant calculations, allowing large models to generate text faster and with less resource consumption. For ease of description, "large model" will be used to refer to "large language model" in the following text, and will not be elaborated further. KV-Cache is specifically used to store the intermediate results of the self-attention mechanism in the Transformer architecture, namely key (i.e., key) vectors and value (i.e., value) vectors. When the large model generates text (i.e., tokens) word by word, each step requires calculating the association between the current word and all historical words (i.e., attention score). For example, when generating the 101st word, without KV-Cache, the large model needs to recalculate the key and value vectors of the previous 100 words; however, with KV-Cache, it only needs to read the cached historical key and value vectors, calculate the new key and value vectors of the current word, and append them to the cache to complete the generation.

[0003] However, deep learning, especially large models based on the transformer architecture, has a massive number of parameters, often ranging from tens to thousands of gigabytes, resulting in enormous computational demands. Simultaneously, the input and output of parameters during training and inference are also enormous. To address this issue, the industry has designed expensive, large-area dedicated chips such as Graphics Processing Units (GPUs), Tensor Processing Units (TPUs), and Neural Processing Units (NPUs), while also configuring high-bandwidth memory (HBM) and graphics double data rate (GDDR) storage chips. However, these chips are costly, and some foreign chips and technologies are subject to various restrictions, making it impossible to meet the demands of high-capacity, high-concurrency model inference.

[0004] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a near-memory computing cascade method based on distributed parameters and cache storage. Its purpose is to solve the problem that existing large models cannot meet the requirements of large-capacity, high-concurrency large model inference during training and inference due to the huge number of parameters, huge amount of computation, huge amount of cached data, and high cost of dedicated chips.

[0006] The present invention adopts the following technical solution: In a first aspect, the present invention provides a near-memory computing cascade method based on distributed parameters and cache storage, comprising: By using multiple cascaded near-memory computing chips and connected local memory, a distributed approach is adopted to store the parameters of large models and cached data during the inference process. Each of the near-memory computing chips obtains the corresponding large model layering parameters and cached data from its connected local memory. Using the large model layering parameters, the cached data, and the input data generated by the previous level, it completes the layering calculation of the corresponding large model decoding stage to obtain updated cached data and the next level input. If the output of the final-level near-memory computation does not generate a termination token, it returns to the first-level near-memory computation chip to continue inference computation, so that the distributed and cascaded inference computation of the large model can be completed through the cooperation of multiple cascaded near-memory computation chips.

[0007] Furthermore, the method also includes: The local memory stores model parameters of a preset number of layers in a distributed manner; The local memory corresponding to the local memory's near-memory computing chip broadcasts the model parameters to other near-memory computing chips.

[0008] Furthermore, the method also includes: Upon completion of each round of large model interaction, determine whether to cache the KV-Cache of this round of large model interaction; When it is determined that KV-cache needs to be stored, the user-input terms are stored on the front-end CPU, and the KV-Cache is cached by network nodes in the distributed network. When a user interacts with a large model, suggested terms are generated based on the cached KV-Cache; these terms are then compared with existing user-input terms stored in the CPU. If highly similar or identical terms are found, it is considered a cache hit. In the event of a cache hit, the CPU instructions retrieve the corresponding KV-Cache from the memory of the corresponding near-memory computing chip node based on the highly similar or identical terms hit; and use the retrieved KV-Cache to generate a dialogue response.

[0009] Furthermore, the method includes: When the network node has available resources, determine the key value to be stored corresponding to the KV-Cache; The network node analyzes the correlation between the key value to be stored and its own stored key values; When the correlation degree meets the second preset condition, the network node determines to cache the KV-Cache so as to cache the KV-Cache in local storage.

[0010] Furthermore, the resource information includes the remaining storage capacity of the corresponding network node and the number of stored terms; The method includes: The network node corresponding to the received storage message is selected as a candidate node; from all candidate nodes, available nodes with remaining storage capacity greater than a first preset value are determined. The product of the number of stored terms and the weight parameter is determined as the amount used, and the comprehensive score of the available nodes is determined based on the ratio of the remaining storage amount to the amount used; wherein, the weight parameter is greater than a second preset value; Send an acknowledgment message to the network node with the highest overall score among all available nodes to select the network node to cache the KV-Cache; When a first preset time has elapsed since the storage message was sent, the network node that has not received the confirmation information deletes the KV-Cache from its local storage.

[0011] Furthermore, the method includes: Establish a mapping relationship between the identifier of the network node and the key corresponding to the cached KV-Cache; Identify the key content involved in this interaction; use the key content to perform semantic similarity matching with all keywords to obtain multiple matching keywords; The matching keywords are identified as related terms; The step of obtaining KV-Cache from the corresponding network node based on the associated terms and generating a dialogue response using the obtained KV-Cache includes: The identifier corresponding to the associated term is determined based on the mapping relationship; Retrieve the KV-Cache corresponding to the associated term from the network node corresponding to the identifier; The KV-Cache is used as an intermediate calculation result to complete the calculation of the dialogue response.

[0012] Furthermore, the method includes: When one or more of the current IP segmentation, interactive content popularity, and / or account type classification involved in this round of large model interaction meet the corresponding first preset conditions, the KV-Cache is cached.

[0013] Secondly, the present invention also provides a near-memory computing cascade device based on distributed parameters and cache storage, for implementing the near-memory computing cascade method based on distributed parameters and cache storage described in the first aspect, the device comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the near-memory computing cascade method based on distributed parameters and cache storage as described in the first aspect.

[0014] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions, which are executed by one or more processors to perform the near-memory computing cascade method based on distributed parameters and cache storage described in the first aspect.

[0015] This invention employs a cascaded architecture for deep learning, large-scale model inference, and training using multiple chips. Each cascaded near-memory computing chip is equipped with its own local memory for storing key-value (KV) cache data. Each near-memory computing chip retrieves the necessary KV-Cache data from its connected local memory, uses the required model parameters, the retrieved KV-Cache data, and the input data to complete the large-scale model decoding stage, and then passes the data to the next near-memory computing chip. This enables cascading and synchronous computation of multiple near-memory computing chips. By utilizing multiple parallel cascaded near-memory computing chips, efficient KV-Cache storage and computation are achieved in the parallel decoding stage of inference computation using local memory. This allows for handling computational tasks with massive parameter counts, massive computational loads, and massive parameter input / output volumes, eliminating the need for expensive dedicated chips and meeting the demands of high-capacity, high-concurrency model inference. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0017] Figure 1 This is a schematic diagram of a computing architecture based on cascaded near-memory computing and parameter broadcasting provided by an embodiment of the present invention; Figure 2 This is a flowchart illustrating a near-memory computing cascade method based on distributed parameters and cache storage provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of a process for inference computation based on cascaded near-memory computing chips, provided by an embodiment of the present invention. Figure 4 This is a flowchart illustrating a specific example of a cascaded near-memory computing chip performing cascaded computing, as provided in an embodiment of the present invention. Figure 5 This is a schematic diagram of a specific example of an AI dialogue interface provided in an embodiment of the present invention; Figure 6 This is an overall schematic diagram of a specific example of a near-memory computing cascade method based on distributed parameters and cache storage provided in an embodiment of the present invention; Figure 7 This is a schematic diagram illustrating a specific example of a network node reporting stored messages, provided in an embodiment of the present invention. Figure 8 This is a schematic diagram illustrating a specific example of a processor selecting one of the network nodes according to an embodiment of the present invention; Figure 9 This is a flowchart illustrating step 60 provided in an embodiment of the present invention; Figure 10 This is a schematic diagram illustrating a specific example of generating a dialogue response based on associative terms, provided by an embodiment of the present invention. Figure 11 This is a flowchart illustrating step 70 provided in an embodiment of the present invention; Figure 12 This is a schematic diagram of the architecture of a near-memory computing cascade device based on distributed parameters and cache storage provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] In the description of this invention, the terms "inner", "outer", "longitudinal", "lateral", "upper", "lower", "top", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and do not require that this invention must be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0020] In this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0021] In this application, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium. Furthermore, the term "coupled" can refer to an electrical connection that enables signal transmission.

[0022] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0023] Example 1: To facilitate the explanation of the near-memory computation cascading method based on distributed parameters and cache storage in the embodiments of the present invention, a brief introduction to KV-Cache is given first: In the Transformer's self-attention mechanism, for each input token, three vectors are generated: a query vector, a key vector, and a value vector. The KV Cache (i.e., cached data) caches the key vector and the value vector.

[0024] For ease of description, the chat dialogue process (i.e., a complete round of interaction) in the prior art that can use KV-Cache to calculate the current dialogue response is defined as a continuous round of chat dialogue between the user and the large model, which will not be elaborated further below. During a continuous round of chat dialogue between the user and the large model, a corresponding query vector is generated based on each user input. By calculating the query vector with the corresponding key vector and value vector, a dialogue response is generated word by word (i.e., token).

[0025] Specifically, in the self-attention mechanism, the key vector can be viewed as an "identifier" or "summary" for each token in a sequence. When the large model processes a new user-input token, it uses the query vector corresponding to that new token to perform similarity calculations (i.e., dot product operations) with the key vectors of all tokens in the sequence to determine which parts of the sequence should be focused on. In other words, the key vector is used for retrieval by the query vector, thereby determining the attention weights. Each key vector represents a certain feature of the corresponding token, which is used to match it with the query vector.

[0026] The value vector is the "actual content" or "information" of each token. In the attention mechanism, after the large model calculates the attention weights based on the similarity between the query vector and the key vector, it uses these weights to perform a weighted sum of all the value vectors to obtain the output of the current token. The value vector carries the information extracted from each token, and this information will be aggregated into the representation of subsequent tokens according to the attention weights. In each layer of the Transformer, for each token, the input vector (i.e., typically the output of the previous layer in the neural network) is transformed into a query vector, a key vector, and a value vector through three different linear transformations (i.e., weight matrices). These transformations are learned by the larger model during training. Therefore, the key and value vectors are intermediate computations obtained from the input of the current layer; they are not directly used as the output of that layer but are used to compute the output after self-attention.

[0027] During the autoregressive generation process, when the large model generates tokens one by one, the key vectors and value vectors of previously generated historical tokens are reused when calculating subsequent tokens. Therefore, to avoid redundant calculations, these key vectors and value vectors are cached, i.e., a KV cache. In the process of generating dialogue responses, when generating a new token, the model only needs to calculate the query vector of the new token along with the corresponding key and value vectors. Then, it uses the query vector of the new token and all key vectors in the cache (including the key vector of the currently calculated new token) to calculate attention weights. These weights are then used to perform a weighted sum of all value vectors in the cache (including the value vector of the currently calculated new token) to obtain the output of the new token. After generating outputs for each new token input by the user, a complete dialogue response is obtained.

[0028] To address the aforementioned problems, embodiments of the present invention provide a near-memory computing cascade method based on distributed parameters and cache storage, such as... Figure 1The diagram illustrates a pre-filling stage computational architecture for a large deep learning model based on cascaded near-memory computing and parameter broadcasting, according to an embodiment of the present invention. Each near-memory computing chip is connected to a corresponding local memory; multiple near-memory computing chips are connected sequentially, forming a string with the connected local memories. Ellipses represent near-memory computing chips, rectangles represent local memories, and the local memory near each near-memory computing chip is defined as the local memory connected to that chip. Figure 1 In the middle, multiple near-memory computing chips, connected end-to-end by arrows in each horizontal row, along with the local memory connected to these near-memory computing chips, form a string, for example, Figure 1 It contains three strings. The specific types of the local memory and the near-memory computing chip are determined by those skilled in the art based on the specific application scenario. In one embodiment, a low-cost, general-purpose memory with moderate storage capacity can be used, such as Double Data Rate Synchronous Dynamic Random-Access Memory (DDR), Graphics Double Data Rate Synchronous Dynamic Random-Access Memory (GDDR), and Low Power Double Data Rate Synchronous Dynamic Random-Access Memory (LPDDR) as the local memory; where SDRAM stands for Synchronous Dynamic Random-Access Memory. The near-memory computing chip can be a near-memory computing chip integrating the controller intellectual property core (IP) corresponding to the local memory, programmable matrix and vector computing units, and cascading interfaces; where the controller intellectual property core corresponding to the local memory can be the controller IP of DDR, GDDR, or LPDDR.

[0029] Based on such Figure 1 The architecture shown is as follows: Figure 2 As shown, this invention provides a near-memory computing cascade method based on distributed parameters and cache storage, including: Step S1: Using multiple cascaded near-memory computing chips and connected local storage, a distributed method is employed to store the parameters of the large model and the cached data during inference. The cached data is KV-Cache data, as described above, and will not be repeated here.

[0030] Step S2: Each near-memory computing chip retrieves the corresponding large model layering parameters and cached data from its connected local memory. Using the large model layering parameters, the cached data, and the input data generated in the previous stage, it completes the layered calculation of the corresponding large model decoding stage, obtaining updated cached data and the next stage input. Here, the large model layering parameters are the parameters of the large model in step S1, referring to the network weight parameters that participate in the calculation and are iteratively trained during the training phase. Figure 1 As shown, each row of cascaded near-memory computing chips in the horizontal direction receives one input data (i.e., Figure 1 After the "input" in the text, follow the steps as follows: Figure 3 The process shown performs inference calculations. A specific example flowchart of a cascaded near-memory computing chip performing the cascaded calculation process is shown below. Figure 4 As shown, each near-memory computing chip is only responsible for calculating the parameters and input data of a large model at one layer or a specified number of layers.

[0031] Step S3: The output of the last-level near-memory computation, without generating a termination token, returns to the first-level near-memory computation chip to continue inference computation, so as to finally complete the distributed and cascaded inference computation of the large model with the cooperation of multiple cascaded near-memory computation chips.

[0032] In one embodiment, the method further includes: the local memory storing model parameters of a preset number of layers in a distributed manner; and the local memory-corresponding near-memory computing chip broadcasting the model parameters to other near-memory computing chips. The preset number of layers is determined by those skilled in the art based on the specific application scenario and is not limited here. Each near-memory computing chip obtains one or more layers of parameters from the horizontally cascaded broadcast parameters, while simultaneously reading cached data from the local memory, thereby completing the computation in the decoding stage. Parameter broadcasting also reduces the number of memory reads, and the reuse of cached data in the local memory reduces the number of computations. Figure 1 In practical applications, all the near-memory computing chips in the system perform inference calculations based on the same network model. Only one row of cascaded near-memory computing chips in the horizontal direction needs to read the corresponding model parameters and broadcast them to the corresponding near-memory computing chips in the vertical direction. This achieves the effect that other cascaded near-memory computing chips do not need to read the same model parameters repeatedly, which greatly reduces the number of read and write operations of other strings to read model parameters and the corresponding resource consumption. It also reduces the number of storage reads and storage bandwidth required in the pre-filling stage during a large number of parallel inference calculations, making it highly practical.

[0033] This invention employs a cascaded architecture for deep learning, large-scale model inference, and training using multiple chips. Each cascaded near-memory computing chip is equipped with its own local memory, which stores model parameters and cached data. Each near-memory computing chip retrieves the necessary model parameters and cached data from its connected local memory. Using the required model parameters, the retrieved cached data, and the input data, it completes the large-scale model decoding stage and passes the computation to the next near-memory computing chip, thus enabling cascading and synchronous computation of multiple near-memory computing chips. By using multiple parallel cascaded near-memory computing chips, efficient key-value cache storage and computation are performed using local memory during the parallel decoding stage of inference computation. This allows for handling computational tasks with massive parameter counts, massive computational loads, and massive parameter input / output volumes, eliminating the need for expensive dedicated chips and meeting the demands of high-capacity, high-concurrency model inference.

[0034] In one embodiment, the near-memory computing chip can also integrate a general-purpose processor as a preprocessing unit for deep learning and large models, and together with the cascaded near-memory computing chip, form a complete architecture for large model computing.

[0035] In another embodiment, the method includes: Step 10: Each of the local memories stores the KV-Cache data of the corresponding local memory computing chip.

[0036] Each local memory stores the KV-Cache data of the nearby computing chip connected to it.

[0037] The following example, a typical autoregressive generative language model based on a transformer structure, illustrates how the architecture of this invention works: Background: Large language models are based on the Transformer structure and generally consist of several parts: (1) input text embedding; (2) multiple layers of self-attention and forward connection layers with the same structure, each layer containing a large number of network model weight parameters; (3) output layer, which converts the vectors output by the previous multiple layers into token text.

[0038] The work on autoregressive generative formulas is divided into two stages: (1) In the pre-filling stage, the input text string is converted into L tokens, and then converted into a two-dimensional matrix with a length of L and a width of d. The matrix is ​​calculated with each self-attention layer and forward connection layer to output a new two-dimensional matrix with a length of L and a width of d. This process is called the pre-filling stage.

[0039] (2) Decoding stage: The aforementioned two-dimensional matrices of length L and width d are merged along the length dimension to obtain a one-dimensional vector of width d; this one-dimensional vector is then added to the aforementioned two-dimensional matrix of length L to form a two-dimensional matrix of length L+1. This new two-dimensional matrix is ​​then used as input to the first self-attention layer and the forward connection layer of the transformer structure for the next round of computation. Each time it passes through all layers, a new one-dimensional vector is formed until a terminator is generated.

[0040] In one embodiment, to reduce computational load, during the decoding stage, the calculation results of the key vector and value vector in each layer of the L-dimensional query vector, key vector, and value vector can be cached, i.e., KV-Cache (i.e., cached data). Each time, the calculation is not repeated, and only the latest (e.g., L+1 vector) calculation is required for each layer.

[0041] Step 20: Each near-memory computing chip obtains KV-Cache data from its connected local memory, and uses the required model parameters, the obtained KV-Cache data, and the input data to complete the calculation of the large model decoding stage and obtain intermediate quantities.

[0042] The specific method of using KV-Cache data in computation is determined by those skilled in the art based on the specific application scenario and is not limited here. In one embodiment, when each near-memory computing chip needs to perform computation during the large model decoding stage, it retrieves the KV-Cache data required for this computation from its connected local memory, and uses the model parameters to be computed, the retrieved KV-Cache data, and the input data to perform computation to obtain intermediate quantities. For example... Figure 1 As shown, each string processes its own input data separately. Figure 1 The three strings in the process are processed into three "inputs". For the first near-memory computing chip of each string (i.e., Figure 1 For the leftmost near-memory computing chip, the input data involved in the calculation in step 20 is... Figure 1 The "input" in the context refers to the intermediate value, which is the result of the current calculation by the near-memory computing chip and can be understood as the intermediate calculation result of the string in the current round of calculation.

[0043] Step 30: Transmit the intermediate quantity to the next near-memory computing chip and use it as input data to participate in the calculation, obtain the intermediate quantity of the next near-memory computing chip, and obtain the output result corresponding to the string to complete the response generation.

[0044] Here, the next near-memory computing chip refers to the next near-memory computing chip directly connected to the near-memory computing chip that performs the calculation in step 20; for example, when the execution entity of step 20 is... Figure 1 When the leftmost near-memory computing chip is selected, the next near-memory computing chip is... Figure 1 The second near-memory computing chip from left to right. The process of the next near-memory computing chip obtaining the intermediate value is the same as in step 20, except that this intermediate value is used as the input data for the current calculation. That is, the next near-memory computing chip uses the model parameters it needs to calculate, the KV-Cache data it has acquired, and the intermediate value to complete the large model decoding stage calculation, obtaining the intermediate value for this calculation. This will not be elaborated further. This process continues level by level, using the intermediate values ​​calculated by each near-memory computing chip on the same string as the input data for the next near-memory computing chip, and calculating sequentially. After the last near-memory computing chip of each string completes its calculation, the output result of that string in the current round of calculation is obtained. In other words, each level of near-memory computing chip uses the token input from the previous level, combined with the model parameters and cache of this level, to calculate the output of this level, which is then used as the input for the next level.

[0045] For example Figure 1 As shown, for each input, multiple rounds of computation are required to generate the final output for that input token by token. For example, in a scenario where the user provides an input and the model needs to generate a dialogue response, each string generates the output result of the current round after each round of computation. Then, the output result of the current round is used as the input for the next round of computation, and the iteration continues to generate the output result of the next round. After the final iteration, the statement generated token by token in each round is the final output, which is the complete response generated for the user input.

[0046] It should be noted that the near-memory computation cascade method based on distributed parameters and cache storage in this embodiment of the invention is applicable to model computation processes that can utilize KV-Cache technology, used to generate tokens, and is suitable for large model inference; the completion response generation in step 30 is only a specific example in a specific scenario provided by this embodiment of the invention, and is not intended to limit the invention.

[0047] It is worth noting that in large model training computations: gradient-based backpropagation computation can be achieved through a single forward inference process and a single backpropagation process across multiple cascaded near-memory computing chips. In large model fine-tuning computations: the implementation process is similar to that in large model training computations, with only the parameter changes for fine-tuning based on pre-trained parameters distributed across different near-memory computing chips. During the training phase, a single cascaded near-memory computing chip can support backpropagation training through distributed multi-level computation, or support training solely through parameter optimization methods using forward propagation.

[0048] In one embodiment, a central processor manages multiple strings; the near-memory computing chips between the different strings managed by the central processor are interconnected. In a specific example, such as Figure 1 As shown, a series consists of one or more central processors connected in series with multiple near-memory computing chips; one central processor can connect to and manage multiple series. Figure 1 The central processor connects and manages the three strings. The proximity computing chips between different strings can be used... Figure 1 Vertical bus or point-to-point connection.

[0049] Based on this, in one embodiment, to illustrate the process of obtaining the required model parameters, in step 20, before completing the large model decoding stage calculation using the required model parameters, the obtained KV-Cache data, and the input data to obtain the intermediate quantity, the following steps are also included: Step 201: The local memory corresponding to one of the strings managed by the central processor stores the model parameters of the preset number of layers in a distributed manner.

[0050] In this context, the local memory corresponding to one of the strings managed by the central processor refers to the local memory connected to all the near-memory computing chips in that string. The preset number of layers refers to the number of network layers of a model (e.g., a deep learning model) network structure that a near-memory computing chip needs to compute each time. This preset number of layers is determined by those skilled in the art based on the specific application scenario and is not limited here. Model parameters refer to the network weight parameters that participate in the computation and are iteratively trained during the training phase.

[0051] In one embodiment, such as Figure 1 As shown, the model parameters of each layer of the deep learning model used in the current computation are stored in a distributed manner. Figure 1 The model parameters for one or more layers can be stored in the local memory connected to a local memory chip, which is located at the bottommost level. Each local memory chip is responsible for calculating the model parameters and input data for only one or more layers.

[0052] In one embodiment, to support the computation of autoregressive generative models, after the last near-memory computing chip completes its computation, the vector of the output result obtained by the current string in the current round is returned to the first near-memory computing chip of the current string through an input feedback loop (e.g., Figure 1 The leftmost near-memory computing chip.

[0053] Step 202: The local memory corresponding to the local memory's near-memory computing chip broadcasts the model parameters to the near-memory computing chips corresponding to other strings.

[0054] In one embodiment, for Figure 1 All strings managed by this central processing unit are stored in a distributed manner as model parameters used in the current computation. Figure 1 When the model is in the lowest row of local memory, the local memory computing chips in the bottom row (or other rows) broadcast the model parameters needed by the large model during inference or training to the local memory computing chips in other rows.

[0055] In the pre-filling stage of large model inference computation, the scheme of reading and cascading broadcasting model parameters according to the embodiments of the present invention allows the same data to be reused among multiple strings, reducing the storage capacity required to store model parameters and reducing the number of memory reads. After the pre-filling stage of large model inference computation is completed, the local memory of the near-memory computing chips in different strings is used to store KV-Cache. Then, in the decoding stage of large model inference computation, each near-memory computing chip obtains one or more layers of parameters from the vertically broadcast parameters, and simultaneously reads KV-Cache from the local memory, thereby completing the computation in the decoding stage. In this process, the number of memory reads is also reduced through parameter broadcasting, and the number of computations is reduced through the KV-Cache in the local memory. Multiple parallel cascaded near-memory computing chips support the computation of the same model simultaneously through parameter broadcasting in the parallel pre-filling stage of inference computation. It is worth noting that, due to, as Figure 1 In the architecture shown, the central processing unit manages multiple strings using the same network model. Therefore, the model parameters required by the near-memory computing chips for computing the same network layer in each string are the same. Only one string needs to read the corresponding model parameters once, and broadcast them to the near-memory computing chips of other strings. This eliminates the need for other strings to repeatedly read the same model parameters, significantly reducing the number of read / write operations and resource consumption for other strings. The more inputs the central processing unit processes simultaneously, the more read operations can be reduced by broadcasting model parameters, thus enabling a low-cost, high-capacity, and high-bandwidth storage system to support the training and inference of large models. This invention achieves parameter reuse in the pre-filling stage of inference computation for large models through parameter broadcasting connection channels between multiple parallel cascaded near-memory computing chips. This reduces the number of storage reads and storage bandwidth required in the pre-filling stage during large-scale parallel inference computation, achieving efficient inference computation in the filling stage. Furthermore, by using the private local memory of multiple parallel cascaded near-memory computing chips for KV-Cache storage, parameter reuse in the decoding stage of inference computation for large models is achieved, reducing the required KV-Cache transmission, storage capacity, and KV-Cache storage bandwidth, and achieving efficient inference computation in the decoding stage.

[0056] In existing technologies, the size of the KV-Cache data grows linearly with the number of dialogue rounds. This leads to inefficiency in quickly locating relevant information from this massive historical context, resulting in increased computational complexity for attention and a significant rise in response latency. Consequently, the effectiveness of improving generation speed based on KV-Cache is limited. Furthermore, single-node storage and computing resources are limited, failing to meet the needs of multi-user concurrent long dialogue scenarios. Additionally, historical KV-Cache data cannot be shared and reused across different dialogue sessions, ultimately resulting in wasted storage resources.

[0057] To address the limitations of existing technologies in improving generation speed based on KV-Cache and inefficient utilization of storage resources, in one embodiment, the method further includes: Upon completion of each round of large model interaction, determine whether to cache the KV-Cache from this round of large model interaction.

[0058] When it is determined that KV-cache needs to be stored, the user input terms are stored in the front-end CPU, and the KV-Cache is cached using network nodes in the distributed network. Specifically, storing user input terms in the front-end Central Processing Unit (CPU) means that the CPU service responsible for request preprocessing receives user input, determines whether to cache it, how to cache it, and performs the initial storage operation.

[0059] When a user interacts with a large model, suggested terms are generated based on the cached KV-Cache; these terms are then compared with existing user-input terms stored in the CPU. If highly similar or identical terms are found, the cache (i.e., KV-Cache) is hit.

[0060] In the event of a cache hit, the CPU instructions retrieve the corresponding KV-Cache from the memory of the corresponding near-memory computing chip node based on the highly similar or identical terms hit; and use the retrieved KV-Cache to generate a dialogue response.

[0061] In one embodiment, the method further includes: Step 40: After each round of large model interaction is completed, determine whether to cache the KV-Cache of this round of large model interaction.

[0062] Here, KV-Cache refers to the intermediate calculation results generated during the calculation process, which has been introduced and explained in Embodiment 1 and will not be repeated here. Completing a round of large model interaction means that the user and the large model engage in a continuous chat conversation. In this embodiment of the invention, each time the user and the large model finish a continuous chat conversation, it determines whether to cache the KV-Cache of this round of large model interaction, that is, whether the intermediate calculation results of this round of interaction have storage value. A specific determination method will be provided below and will not be repeated here. The KV-Cache cache cached in step 40 is the KV-Cache corresponding to one or more tokens in each round of interaction, that is, the KV-Cache generated when calculating one or more tokens.

[0063] Step 50: Cache the KV-Cache using network nodes in the distributed network.

[0064] In this context, a distributed network refers to a distributed computing node network, which will be simply referred to as a "distributed network" for ease of description. A distributed network includes multiple network nodes, such as network node 1, network node 2, and so on, network node n. The processor of the distributed network is used to handle the computations required by the distributed network as a whole, and the memory of the distributed network is used to store the data required by the distributed network as a whole. Each network node includes a processor and a memory. The processor is used to handle the computations required by the network node, and the memory is used to store the data required by the network node. In one embodiment, the processor that actually performs the computation in each network node can be the local memory chip mentioned in steps 10 to 30 above, and the memory of the network node can be the local memory mentioned above. The processor of the distributed network can perform the functions of the central processor mentioned above.

[0065] Step 60: When a user interacts with a large model, generate associated terms based on the cached KV-Cache.

[0066] The cached KV-Cache refers to the KV-Cache stored on various network nodes in the distributed network during user interaction with the large model. In one embodiment, multiple associative terms are generated based on the cached KV-Cache, and these terms can be used to construct an associative term library. This library is used to generate associative terms in the interactive interface each time a dialogue response needs to be generated based on the KV-Cache during the entire large model interaction process, allowing the user to select from them. For example, such as... Figure 5The image shows an AI dialogue interface (i.e., an interactive interface) for a user to chat with a large model. In one embodiment, when the user inputs the first question "Current topic 1" into the large model, the large model first generates dialogue content according to existing technology and obtains "Initial reply 1". At the same time, since the large model has obtained the input "Current topic 1", it can use "Current topic 1" and the associated words corresponding to the cached KV-Cache and display them on the interface.

[0067] Step 70: Obtain KV-Cache from the corresponding network node based on the associated terms; use the obtained KV-Cache to generate a dialogue response.

[0068] For example, such as Figure 5 As shown, in one embodiment, after a user selects one or more of "associative terms 1" to "associative terms n", the interface will automatically add the text of the selected associated terms to the user input box. After the user adds a description based on the selected associated terms and fills in the user input, they click the send button to submit the user input. At this time, the associated terms selected by the user are obtained, and the network node storing this or these associated terms is determined from the distributed network. The corresponding KV-Cache is obtained from the corresponding network node. These KV-Caches are used as intermediate calculation results in history and participate in the calculation of the dialogue response, so as to achieve the effect of only needing to reuse the historical context each time and calculate the query vector and the new key vector and value vector. The specific method of using the obtained KV-Cache to generate the dialogue response is determined by those skilled in the art according to the specific use case. A specific example is described above and will not be repeated here.

[0069] This invention uses a distributed network to store the KV-Cache. Compared to a single high-bandwidth memory, the storage and computational pressure on each network node is minimal, avoiding the limitation of the KV-Cache being unable to fully cache due to local storage capacity constraints. By generating predictive entries, it enables rapid location of relevant information in the current user input from massive historical contexts, effectively alleviating the response latency of dialogue response generation and breaking through the limitations of existing technologies that rely on KV-Cache to improve generation speed. Furthermore, it not only meets the needs of multi-user concurrent long dialogue scenarios but also enables the sharing and reuse of historical KV-Cache across different dialogue sessions through predictive entries, making better use of storage resources and avoiding waste caused by the inability to reuse historical KV-Cache.

[0070] Because user inputs are likely to be highly similar in certain scenarios (e.g., enterprise LANs with low storage costs), this invention utilizes a historical key-value cache corresponding to the associative term index and leverages a distributed network to alleviate the access pressure on the historical key-value cache; it uses the historically computed key-value cache to generate new dialogue responses for similar user inputs. Figure 6 As shown, the near-memory computation cascade method based on distributed parameters and cache storage, according to an embodiment of the present invention, will be further described below: In one embodiment, step 40 includes: caching the KV-Cache when one or more of the current IP segmentation, interactive content popularity and / or account type classification involved in this round of large model interaction meet the corresponding first preset conditions.

[0071] The first preset condition is determined by those skilled in the art based on the specific use case. In one embodiment, the current IP segment involved in this round of large model interaction is obtained.

[0072] For example, if the IP segment is a campus IP, a designated enterprise IP, a specific research institute IP, etc., the interactive content includes currently popular online terms, and the account type is at a specific level or above (i.e., can be identified as a professional or expert personnel to a certain extent), then the corresponding KV-Cache will be considered to have reuse value.

[0073] In this round of large-scale model interaction, the current IP segment refers to the IP segment used by the user currently engaging in the large-scale model interaction. For the current IP segment, the first preset condition can be: the current IP segment is a campus IP, a designated enterprise IP, or a specific research institute IP.

[0074] The system pre-captures currently trending online terms within the current time period; the specific value of the current time period is determined by those skilled in the art based on the specific use case; in an optional embodiment, the current time period can be a preset number of hours before the start time of this round of large model interaction, up to the end time of this round of large model interaction; the current time period can also be a fixed value, such as 12 hours. In an optional embodiment, when the dialogue content (i.e., user input and large model output) of this round of large model interaction contains one or more of the currently captured currently trending online terms, the popularity of the interaction content satisfies the corresponding first preset condition.

[0075] In an optional embodiment, a large language model can also be used to perform semantic analysis and matching between the currently crawled popular web terms and the dialogue content (i.e., user input and large model output) of this round of large model interaction. The percentage of semantic analysis matching degree is determined as the popularity of the interaction content in this round of large model interaction. In this case, the first preset condition for the interaction content popularity can be: the interaction content popularity is greater than a preset percentage; wherein, the specific value of the preset percentage is determined by those skilled in the art according to the specific use case; in an optional embodiment, the preset percentage can be 80%.

[0076] Obtain the user accounts that participated in this round of large-scale model interaction. For account type classification, the first precondition can be: the user account is a specified type such as a professional account or an expert account. Accounts at a specific level or above can be considered to have reusability; for example, the interactive content involving an expert account is likely to be cited frequently in related fields, such as other users searching based on keywords in an expert's paper.

[0077] In one embodiment, step 50 includes: Step 501: Broadcast the KV-Cache to each network node in the distributed network.

[0078] Broadcast the KV-Cache to the distributed computing node network.

[0079] Step 502: The network node determines, based on its own circumstances, whether to cache the KV-Cache locally.

[0080] like Figure 6 As shown, each network node, upon receiving a broadcast, calculates and analyzes whether to cache the KV-Cache. Local storage means storing it in the network node's own memory. In one embodiment, the network node in step 50 can use external non-volatile memory to store the KV-Cache.

[0081] Step 503: When the network node caches the KV-Cache in local storage, it generates a storage message carrying its own resource information and reports the storage message to the distributed network.

[0082] If a network node caches this KV-Cache, it will report the cache status to the processors of the distributed network.

[0083] For example, such as Figure 7As shown, there are m network nodes that, according to step 502, determine that the KV-Cache can be cached locally and have already cached it. Each network node reports the corresponding storage message to the processor of the distributed network after caching. At this time, the processor of the distributed network will receive m storage messages. For example, network node 1 determines that the KV-Cache can be cached locally and has already cached it, and reports storage message 1 to the processor of the distributed network; network node 3 determines that the KV-Cache can be cached locally and has already cached it, and reports storage message 3 to the processor of the distributed network; network node n determines that the KV-Cache can be cached locally and has already cached it, and reports storage message m to the processor of the distributed network.

[0084] Step 504: The distributed network selects one of the network nodes according to the storage message; sends confirmation information to the selected network node to complete the caching of the KV-Cache.

[0085] For example, such as Figure 8 As shown, the processor of the distributed network selects network node 3 from the m network nodes to cache the KV-Cache based on the received m storage messages, so the processor sends an acknowledgment message to network node 3.

[0086] In one embodiment, after broadcasting, each network node will analyze the correlation between the K value (i.e., key value) in the KV-Cache and its own stored K value (i.e. key value) when idle. When the semantic analysis correlation meets the preset value, the KV-Cache will be cached locally and the caching result will be informed to the processor. At this time, the processor updates the associative dictionary and performs relation mapping on the node ID of the network node that caches the KV-Cache.

[0087] In one embodiment, step 502 includes: Step 5021: When the network node has available resources, determine the key value to be stored corresponding to the KV-Cache.

[0088] The specific definition of available resources is determined by those skilled in the art based on the specific use case. In one embodiment, it means that the network node is currently available for storage within a certain time range and has sufficient storage space for the size of the KV-Cache; the key value to be stored corresponding to the KV-Cache refers to the key value in the KV-Cache.

[0089] Step 5022: The network node analyzes the correlation between the key value to be stored and its own stored key values.

[0090] Among them, the stored key-value refers to the key-value that has been stored in the KV-Cache of the local storage of this network node.

[0091] Step 5023: When the correlation degree meets the second preset condition, the network node determines to cache the KV-Cache so as to cache the KV-Cache in local storage.

[0092] The second preset condition is determined by those skilled in the art based on the specific use case; in one embodiment, the second preset condition may be a correlation degree greater than 80%.

[0093] In one embodiment, after each network node caches its data, it reports to the processor by carrying resource information through a storage message. This resource information includes the remaining storage capacity and the number of stored terms for the corresponding network node. Step 504 includes: Step 5041: Select the network node corresponding to the received storage message as a candidate node; determine the available node with a remaining storage amount greater than the first preset value from all candidate nodes.

[0094] The first preset value is determined by those skilled in the art based on the specific usage scenario, and is not limited here.

[0095] Step 5042: Determine the used amount by multiplying the number of stored terms by the weight parameter, and determine the comprehensive score of the available nodes based on the ratio of the remaining storage amount to the used amount; wherein the weight parameter is greater than a second preset value.

[0096] The second preset value is determined by those skilled in the art based on the specific usage scenario. In one optional embodiment, the second preset value can be 1.

[0097] In one embodiment, both the remaining storage capacity and the number of stored terms can be normalized values. A method for calculating the overall score of each available node. The specific example expression is as follows:

[0098] in, Indicates the remaining storage capacity. Represents the weight parameters. This indicates the number of terms that have been stored. It is an adjustable weight parameter (default value can be set to 1) used to control the degree of penalty for the score based on the number of stored terms. The larger the value, the stronger the negative impact of the number of stored terms on the score. The "+1" in the denominator is used to avoid... When the denominator is zero, the formula is ensured to be mathematically valid. The larger the score, the higher the chance of being selected. The larger the value, the lower the score, to avoid too many entries affecting performance. Smaller terms result in lower scores and are less likely to be selected. Since storing too many terms can affect the speed of subsequent matching and searching, and when storing terms, it is necessary to prioritize network nodes with more remaining storage space, the above expression can be used to achieve this effect.

[0099] Step 5043: Send an acknowledgment message to the network node with the highest overall score among all available nodes to select the network node to cache the KV-Cache.

[0100] When a processor in a distributed network sends an acknowledgment to one of the network nodes, it means that the network node has been selected to cache the KV-Cache.

[0101] Step 5044: When a first preset time has elapsed since the storage message was sent, the network node that has not received the confirmation information deletes the KV-Cache from its local storage.

[0102] The first preset time is determined by those skilled in the art based on the specific use case. In one optional embodiment, the first preset time can be 5 minutes. In one embodiment, if no confirmation information is received, or if it is confirmed that the node was not selected, the corresponding cached KV-Cache will be locally deleted.

[0103] In one embodiment, during broadcasting, the KV-Cache is also transmitted to the processors of the distributed network. In one embodiment, such as Figure 6 As shown, step 504 further includes: when the distributed network does not receive a storage message within a second preset time, selecting one of the network nodes based on the resource information of each network node, and sending an acknowledgment message to the selected network node, so that the selected network node caches the KV-Cache. The second preset time is determined by those skilled in the art based on the specific usage scenario.

[0104] In one embodiment, such as Figure 9 and Figure 10 As shown, step 60 includes: Step 601: Establish a mapping relationship between the identifier of the network node and the key corresponding to the cached KV-Cache.

[0105] In one embodiment, while saving the KV-Cache, historical interaction output tokens (i.e., tokens for historical user input and output) can also be saved. Since the KV-Cache already contains all the semantic information and positional relationships of the output tokens within the larger model, the KV-Cache and the corresponding output tokens are saved as a single historical data unit. Figure 10 As shown, a mapping table is used to record the identifier (e.g., network node ID) of the network node storing each historical data unit. The keywords corresponding to the cached KV-Cache refer to the keywords of the output tokens of the historical data units that have been stored in the distributed network. Since the output tokens may be one or more sentences, in order to quickly index the network nodes from the associated terms and obtain the corresponding historical data units, after extracting the keywords in this embodiment of the invention, a mapping relationship between the keywords and the network nodes is first established, and then used for matching with key content.

[0106] Step 602: Determine the key content involved in this interaction; use the key content to perform semantic similarity matching with all keywords to obtain multiple matching keywords.

[0107] Here, all keywords refer to the keywords of the output tokens (i.e., the tokens) of all historical data units stored by all network nodes in the distributed network.

[0108] like Figure 5 As shown, in this new interaction, the user input "current topic 1" is obtained as the key content, or the key content is extracted from the user input according to semantic keywords. Since the structure of the output tokens in the historical data unit is more similar to the user input, the key content is directly used to match the keywords (or keyword sets) of all output tokens cached in the distributed network, and the keywords with similarity exceeding the preset matching value are determined as the matching keywords.

[0109] Step 603: Determine the matching keywords as associated terms.

[0110] The natural language of the matching keywords obtained in step 602 is determined as the associated terms; in one embodiment, such as Figure 5 As shown, the multiple suggested terms generated this time (e.g., "Suggested term 1", "Suggested term 2", "Suggested term 3" and "Suggested term n") are all displayed in the suggested term recommendation box for users to click and select.

[0111] Based on this, once the user input is generated from the suggested terms provided by the large model interactive interface, it will directly jump to the corresponding network node through the processor's mapping relationship to perform the matching, searching and generating dialogue response process.

[0112] In one embodiment, such as Figure 10 As shown, the user first submits their input containing the suggested terms to the large model. After obtaining the suggested terms contained in the user input, as follows... Figure 11 As shown, step 70 includes: Step 701: Determine the identifier corresponding to the associated term based on the mapping relationship.

[0113] like Figure 10 As shown, in order to address the possibility that the associated terms may have subtle differences in expression from the natural language that actually corresponds to the KV-Cache, the key content corresponding to the associated terms is used to perform a similarity search (e.g., semantic similarity search) in the mapping table to retrieve the corresponding keyword (e.g., keyword c). Based on the mapping relationship of the keyword, the corresponding network node ID (i.e., identifier) ​​is determined.

[0114] Step 702: Obtain the KV-Cache corresponding to the associated term from the network node corresponding to the identifier.

[0115] Locate the network node according to the identifier and retrieve the KV-Cache from that network node.

[0116] Step 703: Use the KV-Cache as an intermediate calculation result to complete the calculation of the dialogue response.

[0117] The use of historical KV-Cache data follows the existing technical methods described above and will not be repeated here. Since the KVCache already contains all semantic information and positional relationships of historical tokens within the model, directly using the KVCache avoids redundant forward computation. From the perspective of large-scale model computation, the KVCache can completely restore the impact of historical tokens on the currently generated data without information loss. In an optional embodiment, in scenarios requiring external retrieval, multimodal processing, system debugging, or cache management, when external knowledge retrieval based on historical dialogue is needed, historical output token text is used to retrieve relevant documents, the KV-Cache is used to maintain dialogue coherence, and the combination of both generates a more accurate response.

[0118] In one embodiment, the method further includes: when the number of times the selected associated terms corresponding to the cached KV-Cache are selected is less than a third preset value within a third preset time period, and the distributed network meets the timed cleanup conditions, the distributed network sends a cleanup instruction to the network node where the KV-Cache is located, so that the network node performs a cleanup operation on the KV-Cache.

[0119] The third preset time, the third preset value, and the timed cleanup conditions are all determined by those skilled in the art based on the specific application scenario. In one embodiment, the timed cleanup condition can be: the amount of memory independently allocated to each network node maintained by the processor of the distributed network reaches a set cleanup stage. That is, if the number of times the corresponding KV-Cache is subsequently selected by Lenovo is less than a preset value within a specified time, and the amount of memory independently allocated to each computing node maintained by the corresponding processor reaches the set cleanup stage, a cleanup operation will be sent to its respective node.

[0120] Example 2: like Figure 12 The diagram shown is an architectural schematic of a near-memory computing cascade device based on distributed parameters and cache storage according to an embodiment of the present invention. This near-memory computing cascade device based on distributed parameters and cache storage includes one or more processors 21 and a memory 22. Figure 12 Take a processor 21 as an example.

[0121] Processor 21 and memory 22 can be connected via a bus or other means. Figure 12 Taking the example of a connection between China and Israel via a bus.

[0122] Memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the near-memory computing cascade method based on distributed parameters and cache storage in Embodiment 1. Processor 21 executes the near-memory computing cascade method based on distributed parameters and cache storage by running the non-volatile software programs and instructions stored in memory 22.

[0123] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0124] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, they execute the near-memory computing cascade method based on distributed parameters and cache storage described in Embodiment 1 above. For example, they execute each step of the near-memory computing cascade method based on distributed parameters and cache storage described above.

[0125] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0126] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0127] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A near-memory computing cascade method based on distributed parameters and cache storage, characterized in that, include: By using multiple cascaded near-memory computing chips and connected local memory, a distributed approach is adopted to store the parameters of large models and cached data during the inference process. Each of the near-memory computing chips obtains the corresponding large model layering parameters and cached data from its connected local memory. Using the large model layering parameters, the cached data, and the input data generated by the previous level, it completes the layering calculation of the corresponding large model decoding stage to obtain updated cached data and the next level input. If the output of the final-level near-memory computation does not generate a termination token, it returns to the first-level near-memory computation chip to continue inference computation, so that the distributed and cascaded inference computation of the large model can be completed through the cooperation of multiple cascaded near-memory computation chips.

2. The near-memory computing cascade method based on distributed parameters and cache storage according to claim 1, characterized in that, The method further includes: The local memory stores model parameters of a preset number of layers in a distributed manner; The local memory corresponding to the local memory's near-memory computing chip broadcasts the model parameters to other near-memory computing chips.

3. The near-memory computing cascade method based on distributed parameters and cache storage according to claim 1, characterized in that, The method further includes: Upon completion of each round of large model interaction, determine whether to cache the KV-Cache of this round of large model interaction; When it is determined that KV-cache needs to be stored, the user-input terms are stored on the front-end CPU, and the KV-Cache is cached by network nodes in the distributed network. When a user interacts with a large model, suggested terms are generated based on the cached KV-Cache; these terms are then compared with existing user-input terms stored in the CPU. If highly similar or identical terms are found, it is considered a cache hit. In the event of a cache hit, the CPU instructions retrieve the corresponding KV-Cache from the memory of the corresponding near-memory computing chip node based on the highly similar or identical terms hit; and use the retrieved KV-Cache to generate a dialogue response.

4. The near-memory computing cascade method based on distributed parameters and cache storage according to claim 3, characterized in that, The method includes: When the network node has available resources, determine the key value to be stored corresponding to the KV-Cache; The network node analyzes the correlation between the key value to be stored and its own stored key values; When the correlation degree meets the second preset condition, the network node determines to cache the KV-Cache so as to cache the KV-Cache in local storage.

5. The near-memory computing cascade method based on distributed parameters and cache storage according to claim 3, characterized in that, The resource information includes the remaining storage capacity and the number of stored terms for the corresponding network node; The method includes: The network node corresponding to the received storage message is selected as a candidate node; from all candidate nodes, available nodes with remaining storage capacity greater than a first preset value are determined. The product of the number of stored terms and the weight parameter is determined as the amount used, and the comprehensive score of the available nodes is determined based on the ratio of the remaining storage amount to the amount used; wherein, the weight parameter is greater than a second preset value; Send an acknowledgment message to the network node with the highest overall score among all available nodes to select the network node to cache the KV-Cache; When a first preset time has elapsed since the storage message was sent, the network node that has not received the confirmation information deletes the KV-Cache from its local storage.

6. The near-memory computing cascade method based on distributed parameters and cache storage according to claim 3, characterized in that, The method includes: Establish a mapping relationship between the identifier of the network node and the key corresponding to the cached KV-Cache; Identify the key content involved in this interaction; use the key content to perform semantic similarity matching with all keywords to obtain multiple matching keywords; The matching keywords are identified as related terms; The step of obtaining KV-Cache from the corresponding network node based on the associated terms and generating a dialogue response using the obtained KV-Cache includes: The identifier corresponding to the associated term is determined based on the mapping relationship; Retrieve the KV-Cache corresponding to the associated term from the network node corresponding to the identifier; The KV-Cache is used as an intermediate calculation result to complete the calculation of the dialogue response.

7. The near-memory computing cascade method based on distributed parameters and cache storage according to claim 3, characterized in that, The method includes: When one or more of the current IP segmentation, interactive content popularity, and / or account type classification involved in this round of large model interaction meet the corresponding first preset conditions, the KV-Cache is cached.

Images