A key-value data management method and related device

Abstract

Embodiments of the present application provide a KV data management method, the method comprising: determining a first node from a plurality of nodes, each node comprising a KV. Each session comprises one or more inference processes performed by a model in order to obtain an answer corresponding to a user's question. Since KV reuse may occur between different inference processes in the same session, if the session information (at least containing a session identifier) of the first node indicates that the session associated with the KV in the first node is an active session, the probability of subsequent reuse of the first node is relatively high. If the first node is not associated with the active session, the probability of subsequent reuse of the first node is relatively low. Based on the first node, a node performing a first deletion operation is determined, which is equivalent to determining which nodes in the plurality of nodes have a relatively high or low probability of subsequent reuse of the KV in the node. This is conducive to retaining nodes with a relatively high probability of subsequent reuse.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method for managing key-value data and related equipment. Background Technology

[0002] Currently, the transformer model architecture has wide applications in the field of natural language processing, including but not limited to machine translation, text generation, text classification, and question answering systems. Due to the excellent performance of the transformer module, generative models in the field of natural language processing (NLP) often adopt the transformer model architecture.

[0003] In one inference process of the generative model described above, after we input the input text into the model, the model outputs an answer. For example, if the answer includes at least one token, then the model has performed at least one inference process. Each inference process includes: inputting the input token of this inference process into the model to obtain a token generated by the model; the generated token can be concatenated with the input token of this inference process as the input token for the next inference process. After repeating the inference process at least once, at least one token included in the answer of the input text can be obtained.

[0004] In the transformer model architecture, each inference process requires the use of the key (K) and value (V) of all tokens in the input token (hereinafter referred to as KV). To avoid repeatedly generating the KV of previous tokens, KV caching technology can be used. KV caching technology improves inference efficiency by caching the KV of previous tokens.

[0005] However, as the number of stored key-value pairs increases, the storage space they occupy also increases. Therefore, a key-value pair deletion scheme is urgently needed. Summary of the Invention

[0006] This application provides a method and related equipment for managing key-value data, which can retain nodes with a high probability of subsequent reuse when performing deletion operations on KV data, so as to improve the effective utilization rate of stored KV.

[0007] In one aspect, this application provides a method for managing key-value (KV) data. In this method, a first device determines at least one first node from a plurality of nodes, and then, based on the at least one first node among the plurality of nodes, determines at least one node to perform a first deletion operation. The plurality of nodes includes KV data, and each of the plurality of nodes includes KV data.

[0008] In one scenario, the session information included in each first node indicates that the session associated with the key-value pair (KV) in each first node is an active session. Exemplarily, the session information in this application at least includes a session identifier. In this scenario, "determining at least one first node from multiple nodes" can be understood as determining at least one first node retained during the execution of the first deletion operation from multiple nodes. In this scenario, the KV in each first node is used by a reasoning process that has already completed its reasoning, rather than by a reasoning process that is currently inference, and the session to which the reasoning process that has already completed its reasoning belongs has not yet completed its reasoning. Alternatively,

[0009] In another scenario, each first node is a node not associated with an active session. For example, a first node not associated with an active session could be one where the key-value pairs (KVs) in the first node are associated with a session that has completed inference, or one where the KVs in the first node are not associated with any session.

[0010] Each session in this application includes one or more reasoning processes executed through a model, each session being used to obtain an answer corresponding to a user's question. In other words, each session includes one or more reasoning processes for obtaining a complete answer to a user's question.

[0011] In this implementation, if the session information included in the first node indicates that the session associated with the key-value pair in the first node is an active session, it means that the key-value pair in the first node has been used by the currently active session. Furthermore, key-value reuse may occur between different inference processes within the same session, so the key-value pair in the first node may also be reused by subsequent inference processes of the aforementioned active session, thus the probability of the key-value pair in the first node being reused later is relatively high. If the first node is not associated with an active session, it means that the probability of the key-value pair in the first node being reused later is relatively low. After identifying at least one first node from multiple nodes, it's equivalent to identifying the nodes among the multiple nodes with a high (or low) probability of subsequent KV reuse. Since it's clear which nodes among the multiple nodes have a high or low probability of subsequent KV reuse, it's easy to find the nodes with a low probability of subsequent reuse. In other words, it's beneficial to select the nodes with a low probability of subsequent reuse to perform the first deletion operation, while retaining the nodes with a high probability of subsequent reuse. This also helps improve the effective utilization of stored KV, which not only improves the effective utilization of storage space, but also, the more stored KV is reused, the fewer KV are generated, which also helps improve the efficiency of the model's inference process, thereby reducing the latency of the model's inference process and improving the user experience of the user's interaction with the model.

[0012] In one possible implementation, if the session information included in the first node indicates that the session associated with the KV in the first node is an active session, the first device determines the node to perform the first deletion operation based on the first node among multiple nodes. This may include: the first device determining the node to perform the first deletion operation from nodes other than the first node among multiple nodes.

[0013] For example, the first device not only determines at least one first node from a plurality of nodes, but also determines at least one node in use from a plurality of nodes, and then determines at least one node to perform the first deletion operation from at least one node other than the first node and the node in use from a plurality of nodes; in other words, the node in use among the plurality of nodes is a node that cannot be deleted, and each first node is a node that needs to be retained in the first deletion operation.

[0014] Specifically, for example, the first device uses a preset strategy to determine at least one node from among multiple nodes, excluding the first node and the node currently in use, to perform the first deletion operation. For example, the preset strategy may be a least recently used (LRU) strategy, a first-in-first-out (FIFO) strategy, or other strategies.

[0015] In this implementation, if the session associated with the key-value pair in the first node is an active session, the key-value pair in the first node may be reused in the subsequent reasoning process of the aforementioned active session. In this case, the node to perform the first deletion operation is determined from among the nodes other than the first node among multiple nodes; that is, the first node is selected as the node to be retained when performing the first deletion operation. This is beneficial for retaining key-value pairs with a high probability of subsequent reuse (i.e., the key-value pairs in the first node) and prioritizing the deletion of key-value pairs that will not be used subsequently. This helps to improve the effective utilization rate of stored key-value pairs, thereby improving the effective utilization rate of storage space.

[0016] In one possible implementation, the first node includes session information of a first session, which is an active session. The first session employs a breadth-first search algorithm and includes at least one stage of reasoning. The session information of the first session indicates that the key-value pairs in the first node are associated with the current stage in at least one stage of the first session. And / or,

[0017] The first node includes the session information of the second session, which is an active session. The second session uses a depth-first search algorithm and includes an inference process with the attribute of branch expansion and an inference process with the attribute of simulated rollout. The session information of the second session indicates that the key-value pairs in the first node are associated with the inference process with the attribute of branch expansion in the second session.

[0018] In this implementation, if the first node includes session information of an active first session, and the first session uses a breadth-first search algorithm, considering the characteristics of the breadth-first search algorithm, if the key-value pairs (KV) in one of the multiple nodes are associated with a stage preceding the current stage in the first session, then the probability of that node being reused is relatively low. Optionally, the first node will only be retained during the first deletion operation if the session information of the first session indicates that the KV in the first node is associated with the current stage of the first session. That is, in this optional embodiment, if the session information of the first session indicates that the KV in the first node is associated with a stage preceding the current stage of the first session, then the first node cannot be retained during the first deletion operation. This further ensures that the retained first node has a high probability of being reused in subsequent inference processes, which is beneficial for improving the effective utilization rate of stored KV. And / or,

[0019] If the first node includes session information of an active second session, and the second session uses a depth-first search algorithm, considering the characteristics of the depth-first search algorithm, if the key-value pairs (KV) of a node among multiple nodes are associated with a reasoning process in the second session with the attribute "rollout," then the probability of that node being reused in subsequent reasoning processes is relatively low. Conversely, if the KV of a node among multiple nodes is associated with a reasoning process in the second session with the attribute "branch expansion," then the probability of that node being reused in subsequent reasoning processes is relatively high. Optionally, the first node will only be retained when the session information of the second session included in the first node indicates that the KV of the first node is associated with a reasoning process in the second session with the attribute "branch expansion." In this way, it is further ensured that the retained first node has a high probability of being reused in subsequent reasoning processes, which is beneficial to improving the effective utilization rate of stored KV.

[0020] In one possible implementation, when the first node is a node not associated with an active session, the first device determines the node to perform the first deletion operation based on the first node among a plurality of nodes, including: the first device determining the node to perform the first deletion operation from at least one first node included in the plurality of nodes.

[0021] In this implementation, if the first node is not associated with an active session, the probability of the key-value pairs in the first node being reused in the subsequent inference process of the active session is low. In this case, the node to perform the first deletion operation is determined from at least one first node including multiple nodes, that is, the node to be deleted is determined from at least one first node with a low reuse probability. Key-value pairs that will not be used later are deleted first, so as to retain nodes with a higher reuse probability. This is beneficial to improve the effective utilization of stored key-value pairs, thereby improving the effective utilization of storage space.

[0022] In one possible implementation, if the preset conditions are met based on the storage space occupied by multiple nodes after the first deletion operation, the method further includes: the first device determining the node to perform the second deletion operation based on all nodes included in the multiple nodes after the first deletion operation. For example, the first device uses a preset strategy to determine at least one node to perform the second deletion operation from at least one node other than the node currently in use among the multiple nodes after the first deletion operation.

[0023] In this implementation, if, after performing the first deletion operation, the preset conditions for performing deletion operations on multiple nodes are still met based on the storage space occupied by multiple nodes, at least one node for performing the second deletion operation is determined based on all nodes included in the multiple nodes. In other words, during the process of determining the node for performing the second deletion operation, the first node determined in step 501 is no longer retained. If there is insufficient storage space, the inference process executed through the model may not be able to continue. Therefore, although the key-value pairs in the first node have a high probability of being reused in subsequent inference processes, the first node is no longer retained when there is insufficient storage space. This helps to determine more nodes that can be deleted, thereby freeing up more storage space and ensuring the smoothness and stability of the inference process executed through the model.

[0024] In one possible implementation, the key-value (KV) data of multiple nodes is stored in a memory block. Optionally, each node in the multiple nodes is stored in a memory block. In this case, the number of tokens in at least one token in each node is less than or equal to M, where M is an integer greater than or equal to 1, and M is a preset value. In this implementation, since memory is often divided into multiple memory blocks, and each node in the multiple nodes includes KV data with less than or equal to M tokens, it is possible to store each node in the multiple nodes in a single memory block. In other words, one memory block is sufficient to store one node from multiple nodes. Moreover, by setting the value of M, it is possible for one node from multiple nodes to occupy as much of the storage space of a memory block as possible, which is beneficial to improving the effective utilization of memory. When a node in the multiple nodes is deleted, a memory block is released. When a node is added to the multiple nodes, a new memory block can be requested. That is, the granularity of node management in the multiple nodes is aligned with memory management, thus avoiding increasing the difficulty of memory management.

[0025] In one possible implementation, the key-value pairs (KV) of each of the multiple nodes are the KV of a token, and each node also includes a token. Since the KV of the input token for the current inference process is used in each inference operation through the model, to reuse the KV stored in multiple nodes, the first device obtains query information. The query information includes a first token and first session information, where the first session information includes the session identifier of the session to which the first token belongs. Optionally, the input token for the current inference process is split into groups of M tokens to obtain at least one set of tokens arranged in sequence, with the first token being one of these groups. If the first token matches a second token in a second node included in the multiple nodes, the first device adds the first session information to the second node, and the KV of the second token included in the second node is used as the KV of the first token.

[0026] In this implementation, when querying the key-value pairs (KV) of the first token from multiple nodes, if the first token matches the second token in one of the multiple nodes (i.e., the second node), the KV of the second token can be directly used as the KV of the first token, eliminating the need to generate the KV of the first token. This improves the efficiency of the model inference process. Furthermore, first session information is added to the second node. This first session information includes the session identifier of the session to which the first token belongs, indicating that the KV in the second node is associated with the session to which the first token belongs. In other words, it indicates that the KV in the second node was used by the session to which the first token belongs. When performing deletion operations on multiple nodes, the first session information can be used to determine whether the second node needs to be retained. In other words, this application not only provides how to use session information to determine which nodes need to be retained but also how to add session information to existing nodes across multiple nodes, improving the completeness of this solution.

[0027] In one possible implementation, if the session to which the first token belongs uses a breadth-first search algorithm, the first session information also indicates which stage of the session the second node is associated with. This facilitates determining whether the key-value pairs in the second node are associated with the current stage of the session when performing a deletion operation, thus more accurately determining whether the second node needs to be retained. Alternatively, if the session to which the first token belongs uses a depth-first search algorithm, the first session information also indicates which attribute's inference process in the session to which the second node is associated with. This facilitates determining whether the key-value pairs in the second node are associated with the inference process of the session where the attribute is branched during deletion, thus more accurately determining whether the second node needs to be retained. In other words, the goal is still to retain only key-value pairs with high reuse probability among multiple nodes, thereby improving the effective utilization of storage space and the probability of successful matching of key-value pairs stored in multiple nodes, thus improving the efficiency of the model inference process.

[0028] In one possible implementation, a successful match between the first token and a second token in a second node (which includes multiple nodes) indicates that the first hash value and the second hash value are the same. The first hash value is derived from the first token, and the second hash value is derived from the second token in the second node. In this implementation, using hash matching when the first token is matched with tokens in multiple nodes improves the efficiency of the matching process.

[0029] Optionally, multiple nodes employ a prefix tree data storage structure. The hash value corresponding to each token in the prefix tree is obtained by hashing all tokens from the root node to that node. The first hash value corresponding to the first token is obtained by hashing the first token and all tokens before it in the input tokens of the current inference process. If the first hash value is the same as the hash value corresponding to the second token in a certain node, it can be proven that all tokens from the root node to that node in the prefix tree completely match the first token and all tokens before it in the input tokens of the current inference process, thereby improving the accuracy of the hash matching process.

[0030] In one possible implementation, the key-value pairs (KV) of each of the multiple nodes are the KV of the token, and each of the multiple nodes also includes the token. The method further includes: a first device obtaining query information, which includes a first token and first session information, the first session information including the session identifier of the session to which the first token belongs. If no token matching the first token is found among the multiple nodes, a third node is added among the multiple nodes, the third node including the first token, the first token's KV, and the first session information.

[0031] In this implementation, when querying the key-value pair of the first token from multiple nodes, if no token matching the first token exists among the multiple nodes, a new node can be added. The new node includes the first token, the key-value pair of the first token, and the session information of the session to which the first token belongs (i.e., the aforementioned first session information). This application not only provides how to add session information to existing nodes in multiple nodes, but also how to add session information when adding a new node in multiple nodes. In short, it provides how to ensure that a single node in multiple nodes can contain session information under different circumstances, so as to use the session information to determine which nodes need to be retained. This not only improves the completeness of this solution, but also expands the application scenarios of this solution.

[0032] Secondly, this application provides a KV data management device, which includes: a determining unit, configured to determine a first node from a plurality of nodes, each of the plurality of nodes including a KV, wherein session information included in the first node indicates that the session associated with the KV in the first node is an active session, or the first node is a node not associated with an active session, wherein the session information at least includes a session identifier of the session, each session includes one or more inference processes executed by a model, and each session is used to obtain an answer corresponding to a user's question; the determining unit is further configured to determine the node that performs a first deletion operation based on the first node among the plurality of nodes.

[0033] In the second aspect of this application, the KV data management device is also used to perform the steps performed by the first device in the first aspect and various possible implementations of the first aspect. The specific implementation methods, the meanings of the terms, and the beneficial effects of the steps in the second aspect can all be found in the first aspect, and will not be repeated here.

[0034] Thirdly, this application provides an apparatus including a processor and a memory, the processor being coupled to the memory, the memory storing program instructions, and the method described in the first aspect being implemented when the program instructions stored in the memory are executed by the processor.

[0035] Fourthly, this application provides a system that stores KV data, the system includes multiple devices, and at least one device in the system is used to perform the steps performed by the first device in the first aspect and various possible implementations of the first aspect.

[0036] In one possible implementation, the KV data is stored in the local memory of the device in the system, and / or, the KV data is stored in the shared memory of the system.

[0037] Fifthly, this application provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method described in the first aspect.

[0038] Sixthly, this application provides a computer program product comprising a program that, when run on a computer, causes the computer to perform the method described in the first aspect.

[0039] Seventhly, this application provides a chip system including a processor for supporting the implementation of the functions involved in the foregoing aspects, such as transmitting or processing data and / or information involved in the foregoing methods. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the terminal device or communication device. This chip system may be composed of chips or may include chips and other discrete devices.

[0040] The second to seventh aspects of this application correspond to the first aspect or multiple possible ways of the first aspect, and have corresponding beneficial effects. Attached Figure Description

[0041] Figure 1 This is a schematic diagram illustrating the execution of two inference processes using a generative model employing a transformer model architecture, as provided in an embodiment of this application.

[0042] Figure 2 A schematic diagram of the system architecture provided in this application embodiment;

[0043] Figure 3 Another schematic diagram of the system provided in the embodiments of this application;

[0044] Figure 4 Another schematic diagram of the system provided in the embodiments of this application;

[0045] Figure 5 A flowchart illustrating the KV data management method provided in this application;

[0046] Figure 6 A schematic diagram of a prefix tree provided in an embodiment of this application;

[0047] Figure 7 A schematic diagram of the breadth-first search algorithm provided in an embodiment of this application;

[0048] Figure 8 A schematic diagram illustrating the deletion of invalid session information provided in an embodiment of this application;

[0049] Figure 9 Another schematic diagram of the KV data management method provided in the embodiments of this application;

[0050] Figure 10 Another flowchart illustrating the KV data management method provided in this application;

[0051] Figure 11 A schematic diagram illustrating the addition of a node among multiple nodes, provided in an embodiment of this application;

[0052] Figure 12 Another flowchart illustrating the KV data management method provided in this application;

[0053] Figure 13 A schematic diagram of a key-value data management device provided in an embodiment of this application;

[0054] Figure 14 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will recognize that, with the emergence of new application scenarios, the technical solutions provided by this application are also applicable to similar technical problems.

[0056] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0057] To facilitate understanding, some technical terms used in this application will be introduced below.

[0058] (1) Token

[0059] In the field of AI technology, especially in natural language processing (NLP), a token typically refers to the smallest unit or basic element in the text processing process. A token can be a word, a phrase, a punctuation mark, a subword, or a character, depending on the specific application scenario.

[0060] (2) A reasoning process performed through the model

[0061] The model in this application refers to a generative model employing a transformer model architecture or a similar architecture, characterized by relying on intermediate data generated during the token generation process. For example, a generative model employing a transformer model architecture could be a large language model (LLM). A large language model refers to a deep learning model trained on a large amount of text data, capable of generating natural language text or understanding the meaning of language text. Large language models often have a large number of parameters; for example, the number of parameters in a large language model can be greater than or equal to 100 million or even 1 billion. Alternatively, a generative model employing a transformer model architecture can also be a model with a smaller number of parameters that includes transformer neural network modules. For example, the intermediate data may include the key-value pairs of the token.

[0062] In one inference process of the above model, after we input the input text into the model, the model will output an answer, for example, the answer includes N tokens. One inference process of the model executes N inference steps, generating one token in each step, where N is an integer greater than or equal to 1. Each inference step in the N inference steps includes: inputting the input token of the current inference step into the model to obtain a token generated by the model in this inference step; concatenating the generated token with the input token of the current inference step as the input token for the next inference step, and repeating this inference process until a generated token represents a terminator. It should be noted that the value of N is not fixed; when a token generated by the model represents a terminator, it proves that the answer to the input text has been obtained.

[0063] For example, given the input text "What to eat for lunch?", a reasoning process performed by the model yields the answer "Coconut Chicken". This reasoning process can consist of four steps. In the first step, the model's input token represents "What to eat for lunch?", and a token generated during this step represents "Coconut". In the second step, the model's input token represents "What to eat for lunch? Coconut", and a token generated during this step represents "Child". In the third step, the model's input token represents "What to eat for lunch? Coconut", and a token generated during this step represents "Chicken". In the fourth step, the model's input token represents "What to eat for lunch? Coconut Chicken", and a token generated during this step represents the terminator. Thus, the answer "Coconut Chicken" is obtained corresponding to the input text "What to eat for lunch?". It should be understood that this example is only for illustrative purposes and is not intended to limit the scope of this solution.

[0064] (3) Key Value of Token

[0065] Generative models employing the transformer architecture include at least an attention-based neural network layer, which utilizes key-value pairs (KV) tokens during processing. Optionally, the model may also include a feed-forward neural network (FFN) layer, or other neural network layers, depending on the specific application scenario.

[0066] To understand this solution more intuitively, please refer to [link / reference]. Figure 1 , Figure 1 This is a schematic diagram illustrating the execution of two inference processes using a generative model employing a transformer model architecture, as provided in an embodiment of this application. Figure 1 The two inference processes shown are inference process 1 and inference process 2. In inference process 1, the first key (key, K) and first value (value, V) of the input token are obtained, and the query (query, Q) corresponding to a token generated by inference process 1 is obtained. The first key, first query, and first value are input into an attention-based neural network layer to obtain the first updated features generated by the attention-based neural network layer. The first updated features are processed by FFN to obtain a token1 generated in inference process 1.

[0067] In inference process 2, the second key and second value of the input token for inference process 2 are obtained. The input token for inference process 2 has one more token1 than the input token for inference process 1; therefore, the second key has the key of token1 more than the first key, and the second value has the value of token1 more than the first value. The second query corresponding to a token generated in inference process 2 is obtained. The second key, second query, and second value are input into an attention-based neural network layer to obtain the second updated feature generated by the attention-based neural network layer. The second updated feature is then processed by an FFN to obtain a token 2 generated in inference process 2. It should be understood that... Figure 1 The examples in this document are for illustrative purposes only and are not intended to limit the scope of this solution.

[0068] For example, the KV in the embodiments of this application can be understood as a KV tensor. Here, a tensor is a form of data storage, and the dimension of the data is called the order of the tensor. A tensor can be viewed as a generalization of vectors and matrices in a multidimensional space; that is, a vector can be viewed as a one-dimensional tensor, and a matrix can be viewed as a two-dimensional tensor.

[0069] When using a model with a transformer model architecture for inference, key-value pairs (KVs) may be reused in different inference processes within an N-stage inference process. Therefore, KV caching technology can be used to cache the generated KVs to achieve KV reuse and avoid regenerating KVs each time, thereby improving the efficiency of the model inference process.

[0070] However, as the model's inference processes increase and the number of stored key-value pairs grows, the storage space required also increases. Therefore, this application provides a key-value data management method. This method can be applied to AI scenarios involving model-based inference, for example, to manage key-value data stored during the inference process. Alternatively, it can be applied to AI scenarios involving model training and inference (also known as AI training and inference scenarios), managing key-value data stored during the inference process. In the method provided in this application, the applicant discovered that to obtain a complete answer to a user's question, it may be necessary to execute one or more inference processes through the model. In this case, not only will key-value pairs need to be reused in different inference processes within the same process, but key-value pairs may also be reused in multiple different inference processes. To avoid deleting key-value pairs that may be used in subsequent inference processes, the method provided in this application proposes the concept of a session. Each session includes one or more inference processes for obtaining a complete answer to a user's question.

[0071] Key-value (KV) data can be stored across multiple nodes, with each node containing KV data. During the first deletion operation on multiple nodes, the first device first determines at least one first node from among the multiple nodes, and then, based on this first node, determines at least one node to perform the first deletion operation. The session information included in the first node indicates that the session associated with the KV data in the first node is an active session, or that the first node is not associated with an active session. For example, the session information at least includes a session identifier, and each session includes one or more inference processes executed through a model, each session used to obtain an answer corresponding to a user's question. A session identifier can uniquely point to a session; for example, a session identifier can include characters, text, or other elements, which can be determined based on the actual application scenario.

[0072] In this embodiment, if the session information included in the first node indicates that the session associated with the key-value pair in the first node is an active session, it means that the key-value pair in the first node has been used by the currently active session. Furthermore, key-value pairs may be reused between different inference processes within the same session. Therefore, the key-value pair in the first node may be reused by the subsequent inference processes of the aforementioned active session, resulting in a high probability that the key-value pair in the first node will be reused later. If the first node is not associated with an active session, it means that the probability that the key-value pair in the first node will be reused later is low. After identifying at least one first node from multiple nodes, it's equivalent to identifying the nodes among the multiple nodes with a high (or low) probability of subsequent KV reuse. Since it's clear which nodes among the multiple nodes have a high or low probability of subsequent KV reuse, it's easy to find the nodes with a low probability of subsequent reuse. In other words, it's beneficial to select the nodes with a low probability of subsequent reuse to perform the first deletion operation, while retaining the nodes with a high probability of subsequent reuse. This also helps improve the effective utilization of stored KV, which not only improves the effective utilization of storage space, but also, the more stored KV is reused, the fewer KV are generated, which also helps improve the efficiency of the model's inference process, thereby reducing the latency of the model's inference process and improving the user experience of the user's interaction with the model.

[0073] For example, the first device can be a server, terminal device, base station, or other type of device; alternatively, it can be a virtual machine instance, container instance, etc. Further, for example, the first device can be a server in a cloud platform, which can be a public cloud, a private cloud (e.g., a data center deployed within an enterprise), a hybrid cloud (e.g., a combination of public and private clouds), or an industry cloud (e.g., jointly built and operated by enterprises within a specific industry), etc. Exemplarily, the first device can be a server providing cloud services within a cloud platform. For example, the terminal device can be a smartphone, laptop, desktop computer, augmented reality terminal, tablet computer, e-book reader, laptop computer, or IoT device, etc. The product form of the first device can be determined based on the actual application scenario, and this application embodiment does not exhaustively list them.

[0074] Before detailing the key-value (KV) data management method provided in the embodiments of this application, the system provided in the embodiments of this application will be introduced first. This system stores KV data and includes multiple devices, among which at least one first device is included. To provide a more intuitive understanding of the system provided in this application, it will be described below in conjunction with the accompanying drawings. Please refer to the accompanying drawings first. Figure 2 , Figure 2 This is a schematic diagram of the architecture of a system provided in an embodiment of this application. Figure 2 As shown, system 100 includes multiple first devices, each of which stores KV data from multiple nodes in its local memory. Each first device can manage the KV data stored in its local memory using the KV data management method provided in this application. Exemplarily, the local memory is provided by a memory storage device in the first device. For example, the memory storage device can be dynamic random access memory (DRAM), high bandwidth memory (HBM), random access memory (RAM), or other types of memory storage devices.

[0075] See also Figure 3 , Figure 3 This is a schematic diagram of another system architecture provided in an embodiment of this application. For example... Figure 3 As shown, system 100 includes multiple first devices, which share memory. Each first device can access the data in the shared memory, and the key-value (KV) data of multiple nodes is stored in the shared memory of system 100. Each first device in system 100 can manage the KV data stored in the shared memory using the KV data management method provided in this application.

[0076] Continue reading Figure 4 , Figure 4This is a schematic diagram of another system architecture provided in an embodiment of this application. For example... Figure 4 As shown, system 100 includes multiple devices that share memory. Each device in system 100 can access the data in the shared memory, and the key-value (KV) data from multiple nodes is stored in the shared memory of system 100. Among the multiple devices in system 100, there is a first device that can manage the KV data stored in the shared memory using the KV data management method provided in this application.

[0077] It should be understood that Figures 2 to 4 These are all examples of systems provided in the embodiments of this application. The specific architecture of a system containing multiple devices can be determined in combination with the actual application scenario. No exhaustive examples are made in the embodiments of this application.

[0078] Based on the above description, please refer to Figure 5 , Figure 5 This is a flowchart illustrating the KV data management method provided in this application. Figure 5 The method for managing key-value data shown includes the following steps 501 to 503.

[0079] 501. Determine a first node from a plurality of nodes, each of which includes a key-value pair (KV). The session information included in the first node indicates that the session associated with the KV in the first node is an active session, or the first node is a node not associated with an active session. The session information includes at least a session identifier of the session. Each session includes one or more inference processes performed by the model. Each session is used to obtain an answer corresponding to the user's question.

[0080] For example, each of the multiple nodes includes a key-value pair of at least one token, and the key-value pairs included in the multiple nodes are stored in a memory block, which may be a memory block in local memory and / or a memory block in shared memory.

[0081] Optionally, for ease of management, each node in a multi-node network is stored in a memory block. In this case, each node contains at least one key-value pair containing at least M tokens, where M is an integer greater than or equal to 1 and is a preset value. For example, M can be 2, 3, 4, 5, or other values. Since memory is often divided into multiple memory blocks, and each node in a multi-node network includes at least M tokens in its key-value pair, it's possible to store each node in a single memory block. In other words, one memory block is sufficient to store one node from multiple nodes. Furthermore, by setting the value of M, it's possible for one node from multiple nodes to occupy as much of the storage space of a memory block as possible, which helps improve the effective utilization of memory. When a node is deleted from a multi-node network, a memory block is released. When a node is added to a multi-node network, a new memory block can be requested. This means that the granularity of node management in a multi-node network is aligned with memory management, thus avoiding increasing the complexity of memory management.

[0082] Optionally, multiple nodes among the multiple nodes can adopt a trie tree data structure. In this application, a trie tree can be replaced by a dictionary tree or a word lookup tree, etc. A trie tree is a tree structure that includes multiple nodes. Alternatively, the data structure used for multiple nodes can also be a linked list or an array, where an element in a linked list or array can be understood as a node. The subsequent descriptions of the embodiments in this application all use a trie tree data structure as an example. When using other types of data structures, please refer to the relevant documentation for understanding; they will not be repeated in the embodiments of this application.

[0083] For example, each node of a prefix tree includes key-value pairs of M tokens. Prefix trees can reuse common prefixes from different input tokens to reduce the storage space they occupy. For a more intuitive understanding of the concept of storing multiple nodes within a multiple-node data structure using a prefix tree, please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a schematic diagram of a prefix tree provided in an embodiment of this application. Figure 6 Taking a prefix tree where each node contains a key-value pair of three or fewer tokens as an example, Figure 6 Taking a prefix tree with four input tokens (KV) as an example, where input token1 represents the text "What do you want to eat for breakfast tomorrow?", input token2 represents the text "Do you have plans for lunch tomorrow?", input token3 represents the text "What are your plans for this weekend?", and input token4 represents the text "Are you planning to watch a movie this weekend?", etc. Figure 6As shown, the prefix tree includes a root node. The common prefix of input token1 and input token2 includes three tokens representing "your today". Therefore, input token1 and input token2 reuse the key-value pairs of the aforementioned three tokens. The common prefix of input token3 and input token4 includes six tokens representing "this weekend" and "your plans". Therefore, input token3 and input token4 reuse the key-value pairs of the aforementioned six tokens. It should be understood that... Figure 6 The examples in this document are for illustrative purposes only and are not intended to limit the scope of this solution.

[0084] Each session may include at least one reasoning process. A session may include at least one reasoning process performed to obtain a complete answer to a question entered by the user. More simply, for example, after a user enters a question through the interactive interface, at least one reasoning process can be performed to obtain a complete answer to the question, and then the complete answer is output to the user through the interactive interface. The at least one reasoning process included in a session refers to all the reasoning processes performed to complete one interaction with the user.

[0085] In one scenario, a session may include at least one reasoning process comprising: at least one reasoning process performed using a breadth-first search (BFS) algorithm to obtain a complete answer to a user's question. If a session employs a breadth-first search algorithm, the session may include at least one step of the reasoning process, with each step including at least one reasoning process. For example, the breadth-first search algorithm may be the best of n (BON), beam search, or other breadth-first search algorithms, etc., and this application does not exhaustively list them all in its embodiments.

[0086] To further understand this solution, let's consider a specific example: a user's question is, "I don't like spicy food, what should I eat for lunch today?" In the process of using a breadth-first search algorithm to determine the complete answer to this question, the input text in the first stage is based on the user's question. The input text in the first stage could be "I don't like spicy food, what should I eat for lunch today?". The model executes four reasoning processes to obtain four possible answers to the input text: coconut chicken, hot pot, spicy hot pot, and boiled chicken. In the first stage, coconut chicken and hot pot were chosen as the answers.

[0087] In the second stage, input text 1 is "I don't like spicy food, how about coconut chicken for lunch today?" The model performs two reasoning processes to obtain two answers to input text 1: "Considering you don't like spicy food, you can eat coconut chicken. Coconut chicken is not spicy, so I recommend you eat coconut chicken for lunch today"; and "Considering you don't like spicy food, you can eat coconut chicken. Coconut chicken is not spicy and is quite nutritious, so I recommend you eat coconut chicken." Input text 2 is "I don't like spicy food, how about hot pot for lunch today?" The model performs two reasoning processes to obtain two answers to input text 2: "Considering you don't like spicy food, you can eat hot pot. If you use mushroom or tomato broth, it won't be spicy, so I recommend you eat hot pot for lunch, and use mushroom or tomato broth"; and "Considering you don't like spicy food, you can eat hot pot, but most hot pots are spicy, so I don't recommend you eat hot pot for lunch." In the second stage, the answer "Considering you don't like spicy food, you can eat coconut chicken. Coconut chicken is not spicy and is quite nutritious, so I recommend you eat coconut chicken" was selected as the complete answer to question 1. It should be understood that this example is only for ease of understanding of this solution. The breadth-first search algorithm used to obtain a complete answer corresponding to a user's question may involve more or fewer stages of reasoning; the specific situation can be determined based on the actual application scenario. It is important to note that if a session uses the breadth-first search algorithm, and a specific answer is selected in a certain stage, and the answer is further expanded upon in subsequent stages to improve the content, then the expanded answers in subsequent stages will include the specific answer.

[0088] To understand the breadth-first search algorithm more intuitively, please refer to [link to relevant documentation]. Figure 7 , Figure 7 This is a schematic diagram of a breadth-first search algorithm provided in an embodiment of this application. Figure 7 The breadth-first search algorithm in China uses beam search, with a beam search width of 4 as an example. Figure 7 The diagram illustrates the three stages of the breadth-first search algorithm. In stage 1, the input text is obtained based on the user's question. The model performs four inference processes to obtain four answers to the input text (i.e., ... Figure 7 Answers 1-1 to 1-4 were selected from the four answers in Phase 1 (i.e., ... Figure 7 (Answers 1-2 and 1-4 in the model). In stage 2, based on the two answers selected in stage 1, two input texts are obtained. The model then performs four inference processes to obtain two answers for each of the two input texts, for a total of four answers (i.e., ...). Figure 7 Answers 2-1 to 2-4 in the text), 2 answers were selected from the 4 answers in stage 2 (that is... Figure 7(Answers 2-1 and 2-4 in the model). In stage 3, based on the two answers selected in stage 2, two input texts are obtained respectively. The model performs four inference processes to obtain two answers for each of the two input texts, for a total of four answers (i.e., ...). Figure 7 Answers 3-1 to 3-4 in the text), one answer was selected from the four answers in stage 3 (that is... Figure 7 Answer 3-3) in the text serves as the complete answer to the user's question. It should be understood that... Figure 7 The examples in this document are for illustrative purposes only and are not intended to limit the scope of this solution.

[0089] Based on the above examples, it can be seen that if the breadth-first search algorithm is used, at least one reasoning process needs to be executed in order to obtain a complete answer to a user's question. All the reasoning processes executed in order to obtain a complete answer to a user's question are considered as the reasoning processes included in a session.

[0090] In another scenario, a session may include at least one reasoning process comprising: at least one reasoning process performed using a depth-first search (DFS) algorithm to obtain a complete answer to a user's question. If the session employs a DFS algorithm, it may include at least one reasoning process with the attribute of branch expansion, and a reasoning process with the attribute of rollout. For example, the DFS algorithm could be a Monte Carlo tree search (MCTS) or other algorithms.

[0091] For example, a reasoning process with the attribute "branch expansion" can be understood as expanding to produce a better answer to a user's question. A reasoning process with the attribute "rollout" can be understood as evaluating each expanded answer to obtain a score for each expanded answer, which allows selection of one answer from the multiple expanded answers.

[0092] For example, an active session can be understood as a session that has not yet finished reasoning. In other words, a session may include at least one reasoning process performed to obtain a complete answer to a question input by a user. If the complete answer has not yet been obtained, the session has not yet finished reasoning, i.e., it is an active session. In this application, the active state can also be referred to as the live state.

[0093] For example, associating a node (hereinafter referred to as the "target node") with a session can be understood as follows: the key-value pair of the token in the target node has been used in a certain inference process in the session. In other words, when the key-value pair of the token needs to be used in the aforementioned inference process, the token to be used is matched with the tokens in the multiple nodes, and the token to be used matches the token of the target node. Thus, the key-value pair of the token in the target node is used in the aforementioned inference process. The specific implementation process of the aforementioned steps will be described in detail in the following embodiments, and will not be described in detail here.

[0094] Specifically, if the key-value pairs (KV) of a target node among multiple nodes have been used in the reasoning process of a certain session (hereinafter referred to as the "target session" for ease of description), it can be understood that the target node is associated with the target session. The target node includes not only the token and the token's KV, but also the session information of the target session. Among them, the session information of the target session includes at least the identifier of the target session.

[0095] In one scenario, if the target session uses a breadth-first search algorithm, the session information also includes stage information, indicating which stage of the inference process within the target session used the key-value pairs (KV) in the target node. In another scenario, if the target session uses a depth-first search algorithm, the session information also includes attributes, indicating which attribute of the inference process within the target session used the KV in the target node; these attributes are either branch expansion or rollout.

[0096] It should be noted that since the inference process of different sessions in one or more sessions may use the same key-value pairs included in the same node, a node may be associated with one or more sessions, and thus a node may include one or more session information. Of course, a node among multiple nodes may not be associated with any session, and thus its information may not contain session information.

[0097] Optionally, before executing step 501, the first device may determine whether preset conditions are met based on the storage space occupied by multiple nodes. If the determination result is yes, that is, if the preset conditions are met, then steps 501 and 502 are executed, that is, the first deletion operation is performed on the nodes among the multiple nodes. Alternatively, the first device may also perform a deletion operation on the nodes among the multiple nodes every preset time interval, etc. The triggering conditions for steps 501 and 502 can be determined in combination with the actual application scenario.

[0098] For example, a preset condition is that the storage space occupied by all data in multiple nodes is greater than or equal to a first threshold. Another example is that the total storage space used to store KV data is a first value, and the storage space occupied by all data in multiple nodes is a second value. The preset condition is that the difference between the first value and the second value is less than or equal to the second threshold; in other words, the preset condition is that the remaining storage space for storing KV data is less than or equal to the second threshold. Alternatively, the preset condition can be other conditions, which are not exhaustively listed in this embodiment.

[0099] In step 501, the first device may determine one or more first nodes from a plurality of nodes. In one case, the session information included in the first node indicates that the session associated with the key-value pair in the first node is an active session. Alternatively, in another case, the first node is a node not associated with an active session. The two cases described above are described in detail below.

[0100] I. The session information included in the first node indicates that the session associated with the key-value pair in the first node is an active session.

[0101] For example, if the session information included in each first node indicates that the sessions associated with the key-value pairs in each first node are all active sessions, then "determining one or more first nodes from multiple nodes" can be understood as determining one or more first nodes retained during the execution of the first deletion operation from multiple nodes. In this case, the key-value pairs in each first node are used by a reasoning process that has already finished reasoning, rather than by a reasoning process that is currently reasoning, and the session to which the reasoning process that has already finished reasoning belongs has not yet finished reasoning.

[0102] For example, an active session includes three reasoning processes that have already completed: Reasoning Process 1, Reasoning Process 2, and Reasoning Process 3. The reasoning process currently in progress is Reasoning Process 4. In this case, the key-value pair in the first node is used by at least one of Reasoning Process 1, Reasoning Process 2, and Reasoning Process 3, but not by Reasoning Process 4. It should be understood that this example is for the purpose of understanding the concept of the first node and is not intended to limit this scheme.

[0103] Further, if the KV associated with the first node includes an active first session, then the first node will include the session information of the first session. If the first session uses a breadth-first search algorithm, then the first session includes at least one stage of the reasoning process. Optionally, when the session information of the first session indicates that the KV in the first node is associated with the current stage of at least one stage of the first session, it means that the session information of the first session included in the first node is valid session information. Alternatively, as long as the session information in one of the multiple nodes indicates that it is associated with an active session, then the session information in the aforementioned node is valid session information. In other words, if the session information in a node indicates that the KV associated with the aforementioned node is an active session using a breadth-first search algorithm, in an optional embodiment, it is considered that the aforementioned session information is only considered valid session information if the aforementioned session information indicates that it is associated with the current stage of the session. In other embodiments, session information indicating association with an active session may also be considered valid session information, without requiring association with the current state of an active session. And / or,

[0104] If the key-value pairs associated with a first node include an active second session, then the first node includes the session information of the second session. If the second session uses a depth-first search algorithm, then the second session includes an inference process with the attribute of branch expansion and an inference process with the attribute of rollout. Optionally, when the session information of the second session indicates that the key-value pairs in the first node are associated with an inference process in the second session with the attribute of branch expansion, it means that the session information of the second session included in the first node is valid session information. Alternatively, as long as the session information in one of the multiple nodes indicates that it is associated with an active session, then the session information in the aforementioned node is valid session information. In other words, if the session information in a certain node indicates that the active session associated with the key-value pairs in the aforementioned node uses a depth-first search algorithm, in an optional embodiment, the aforementioned session information is considered valid only if the aforementioned session information indicates that it is associated with an inference process in the session with the attribute of branch expansion. In other embodiments, session information indicating association with an active session may be considered valid session information, without requiring association with a reasoning process in an active session whose attribute is branch extension.

[0105] Specifically, before determining one or more first nodes to be retained during the first deletion operation from multiple nodes, the first device may further obtain active session information, wherein the active session information includes the session identifier of each session in at least one currently active session; optionally, if an active session uses a breadth-first search algorithm, the active session information also includes the stage information of the session, indicating the current stage of the session. Based on the active session information, the first device checks each of the multiple nodes to delete invalid session information from the multiple nodes. Then, the first device determines the first node to be retained during the first deletion operation from the multiple nodes after the invalid session information deletion operation has been performed; wherein, after the invalid session information deletion operation has been performed, if any of the multiple nodes still contains session information, that information will be used as the first node to be retained during the first deletion operation.

[0106] Furthermore, the first device, based on active session information, checks each of the multiple nodes to delete invalid session information from the multiple nodes. This can include: after obtaining any session information included in the target node, the first device can determine whether the active session information includes the session identifier in the aforementioned session information. If the active session information does not include the session identifier in the aforementioned session information, it proves that the session pointed to by the session identifier is no longer an active session, but a session that has already been inferred and ended. Therefore, the session information containing the session identifier is invalid session information and can be deleted. In other words, invalid session information includes session information corresponding to sessions that have already been inferred and ended.

[0107] If the active session information includes the session identifier from the previous session information, optionally, in one case, if the session corresponding to the aforementioned session information uses a breadth-first search algorithm, determine whether the stage information in the session information indicates the current stage of the session. If the stage information indicates a stage earlier than the current stage of the session, the aforementioned session information is invalid and is deleted. If the stage information indicates the current stage of the session, the aforementioned session information is valid and is retained. In another case, if the session corresponding to the aforementioned session information uses a depth-first search algorithm, determine whether the attribute in the session information is rollout. If the attribute in the session information is rollout, the aforementioned session information is invalid and is deleted. If the attribute in the session information is branch expansion, the session information is valid and is retained. Alternatively, if the active session information includes the session identifier from the previous session information, then the session information is considered valid.

[0108] The first device can perform the above operation on each of the multiple nodes that contain session information, in order to delete all invalid session information in the multiple nodes.

[0109] To understand this solution more intuitively, please refer to [link / reference]. Figure 8 , Figure 8 This is a schematic diagram illustrating the deletion of invalid session information provided in an embodiment of this application. Figure 8 Taking a data storage structure with multiple nodes using a prefix tree as an example, such as... Figure 8 As shown, the prefix tree includes the root node and nodes 1 through 8. When deleting invalid session information from multiple nodes begins, the currently active sessions are session 2 and session 3. Session 2 uses a breadth-first search algorithm, and its current stage is stage 1 (i.e.,...). Figure 8 In step 1 of the sequence, session 3 uses a depth-first search algorithm. For example... Figure 8 As shown, some nodes in the prefix tree are associated with sessions, while others are not associated with sessions.

[0110] Figure 8 (a) Sub-diagram represents a diagram showing invalid session information that has not yet been deleted, such as Figure 8 As shown in sub-diagram (a), the session information in node 1 indicates that node 1 is associated with step 0 of session 1, and the session information in node 6 indicates that node 6 is associated with step 0 of session 1. The session information in node 4 indicates that node 4 is associated with step 0 of session 2, and also with the inference process in session 3 with the attribute of branch expansion. The session information in node 5 indicates that node 5 is associated with step 0 of session 2, and also with the inference process in session 3 with the attribute of branch expansion. The session information in node 7 indicates that node 7 is associated with step 1 of session 2. The session information in node 8 indicates that node 8 is associated with the inference process in session 3 with the attribute of branch expansion.

[0111] Figure 8 (b) Sub-diagram represents the diagram after deleting invalid session information, in contrast. Figure 8 (a) Sub-diagram and Figure 8In sub-diagram (b), since session 1 is not an active session, the session information in nodes 1 and 6 is invalid and is therefore deleted. Although session 2 is an active session, since the current stage of session 2 is step 1, the session information in nodes 4 and 5 indicating association with step 0 of session 2 is invalid and is also deleted. Since the session information in node 7 indicates that node 7 is associated with step 1 of session 2, that is, the session information in node 7 indicates that node 7 is associated with the current stage of an active session, the session information in node 7 is valid and is therefore retained. Session 3 is an active session, and the session information in nodes 4, 5, and 8 indicating association with the inference process of branch extension in session 3 is valid and is therefore retained. It should be understood that... Figure 8 The examples in this document are for illustrative purposes only and are not intended to limit the scope of this solution.

[0112] Optionally, if multiple nodes employ a prefix tree data storage structure, the first device may determine the first node to be retained during the first deletion operation from among the multiple nodes after the invalid session information deletion operation. This determination may include: the first device may determine the first node to be retained during the first deletion operation from all leaf nodes of the prefix tree after the invalid session information deletion operation. Specifically, if a leaf node of the prefix tree still contains session information after the invalid session information deletion operation, that leaf node will be determined as the first node.

[0113] Second, the first node is a node that is not associated with an active session.

[0114] For example, the case where the first node is not associated with an active session can be: the key-value pairs in the first node are associated with a session that has finished inference, or the key-value pairs in the first node are not associated with any session.

[0115] In this scenario, the first device may first delete all invalid session information from multiple nodes. The specific implementation of the aforementioned steps can be found in the description of the previous scenario and will not be repeated here. Then, from the multiple nodes after the invalid session information deletion operation, at least one first node that is not associated with an active session is determined. After the invalid session information deletion operation, if a node no longer contains session information, it means that the node is not associated with an active session.

[0116] Optionally, if multiple nodes employ a prefix tree data storage structure, the first device may determine, from among the nodes after the invalid session information deletion operation, the first node not associated with the active session. This can include: the first device may determine the first node not associated with the active session from all leaf nodes of the prefix tree after the invalid session information deletion operation. Specifically, after the invalid session information deletion operation, if a leaf node of the prefix tree does not contain session information, that leaf node will be determined as the first node.

[0117] 502. Determine the node to perform the first deletion operation based on the first node among multiple nodes.

[0118] For example, in one case, the session information included in the first node indicates that the session associated with the key-value pair in the first node is an active session. Alternatively, in another case, the first node is a node not associated with an active session. The implementation of step 502 in the aforementioned two cases will be described in detail below.

[0119] I. The session information included in the first node indicates that the session associated with the key-value pair in the first node is an active session.

[0120] In this scenario, step 502 may include: the first device determining the node to perform the first deletion operation from among the nodes other than the first node among the plurality of nodes. For example, the first device not only determines at least one first node from the plurality of nodes, but also at least one node currently in use from the plurality of nodes, and then determines at least one node to perform the first deletion operation from among the nodes other than the first node and the node currently in use; in other words, the node currently in use among the plurality of nodes is a node that cannot be deleted, and each first node is a node that needs to be retained in the first deletion operation. After the first device deletes the at least one node performing the first deletion operation, the storage space occupied by the aforementioned at least one node is released, that is, the first deletion operation is completed.

[0121] Specifically, for example, the first device employs a preset strategy to determine at least one node from among multiple nodes, excluding the first node and the node currently in use, to perform the first deletion operation. For example, the preset strategy could be a least recently used (LRU) strategy, a first-in-first-out (FIFO) strategy, or other strategies, which can be determined based on the actual application scenario; this embodiment does not exhaustively list them. Specifically, the LRU strategy increases the probability of a node's information being deleted the earlier it was last used. The FIFO strategy increases the probability of a node's information being deleted the earlier it was generated.

[0122] Optionally, if multiple nodes use a prefix tree data storage structure and adopt an LRU strategy, the first device can exclude the first node and the leaf nodes currently in use from all leaf nodes of the prefix tree to determine the remaining leaf nodes. The first device obtains the most recent usage time of each of the remaining leaf nodes, and determines the leaf node to perform the first deletion operation based on the most recent usage time of each of the remaining leaf nodes.

[0123] For example, the leaf nodes among the remaining leaf nodes whose most recent usage time is earlier than the time threshold can be determined as the leaf nodes to perform the first deletion operation; or, the S leaf nodes among the remaining leaf nodes whose most recent usage time is the earliest can be determined as the leaf nodes to perform the first deletion operation, etc. The specific rules can be determined in combination with the actual application scenario.

[0124] To understand this solution more intuitively, please refer to [link / reference]. Figure 9 , Figure 9 Another schematic diagram of the KV data management method provided in the embodiments of this application. Figure 9 Taking a data storage structure using a prefix tree with multiple nodes as an example, the prefix tree includes multiple nodes, each storing a token and its key-value pairs (KV). If the KV of a node has been used by at least one session, that node also stores session information for at least one associated session. The first device can determine at least one node currently in use from the multiple nodes included in the prefix tree; each node currently in use cannot be deleted. The first device also determines at least one reserved first node from the multiple nodes included in the prefix tree, where the session information indicates that the reserved node is associated with an active session. The first device considers every node in the prefix tree other than the currently used node and the at least one reserved first node as a deletable node in the process of performing the first deletion operation. Based on a preset strategy, the first device determines at least one leaf node from the leaf nodes of all deletable nodes included in the prefix tree to perform the first deletion operation, i.e., determines at least one node to perform the first deletion operation. It should be understood that... Figure 9 The examples in this document are for illustrative purposes only and are not intended to limit the scope of this solution.

[0125] In this embodiment, if the session associated with the key-value pair in the first node is an active session, the key-value pair in the first node may be reused in the subsequent reasoning process of the aforementioned active session. In this case, the node to perform the first deletion operation is determined from among the nodes other than the first node among multiple nodes, that is, the first node is selected as the node to be retained when performing the first deletion operation. This is beneficial for retaining key-value pairs with a high probability of subsequent reuse (i.e., the key-value pairs in the first node) and prioritizing the deletion of key-value pairs that will not be used subsequently. This helps to improve the effective utilization rate of stored key-value pairs, thereby improving the effective utilization rate of storage space.

[0126] Furthermore, if the first node includes session information of an active first session, and the first session uses a breadth-first search algorithm, considering the characteristics of the breadth-first search algorithm, if the key-value pairs (KV) in one of the multiple nodes are associated with a stage preceding the current stage in the first session, then the probability of that node being reused is relatively low. Optionally, the first node will only be retained during the first deletion operation if the session information of the first session indicates that the KV in the first node is associated with the current stage of the first session. That is, in this optional embodiment, if the session information of the first session indicates that the KV in the first node is associated with a stage preceding the current stage of the first session, then the first node cannot be retained during the first deletion operation. This further ensures that the retained first node has a high probability of being reused in subsequent inference processes, which is beneficial for improving the effective utilization rate of stored KV. And / or,

[0127] If the first node includes session information of an active second session, and the second session uses a depth-first search algorithm, considering the characteristics of the depth-first search algorithm, if the key-value pairs (KV) of a node among multiple nodes are associated with a reasoning process in the second session with the attribute "rollout," then the probability of that node being reused in subsequent reasoning processes is relatively low. Conversely, if the KV of a node among multiple nodes is associated with a reasoning process in the second session with the attribute "branch expansion," then the probability of that node being reused in subsequent reasoning processes is relatively high. Optionally, the first node will only be retained when the session information of the second session included in the first node indicates that the KV of the first node is associated with a reasoning process in the second session with the attribute "branch expansion." In this way, it is further ensured that the retained first node has a high probability of being reused in subsequent reasoning processes, which is beneficial to improving the effective utilization rate of stored KV.

[0128] Second, the first node is a node that is not associated with an active session.

[0129] In this scenario, step 502 may include: the first device determining the node to perform the first deletion operation from at least one first node included in the plurality of nodes. Optionally, the first device employs a preset strategy to determine at least one node to perform the first deletion operation from at least one first node included in the plurality of nodes. For example, the preset strategy may be a least recently used (LRU) strategy, a first-in-first-out (FIFO) strategy, or other strategies, which can be determined based on the actual application scenario. The specific implementation of the aforementioned steps can be referred to the description in the previous scenario, the difference being that "from at least one node other than the first node and the node currently in use" in the above description is replaced with "from at least one first node included in the plurality of nodes," which will not be elaborated here.

[0130] In this embodiment, if the first node is not associated with an active session, the probability of the key-value pairs (KVs) in the first node being reused in the subsequent inference process of the active session is low. In this case, the node to perform the first deletion operation is determined from at least one first node including multiple nodes, that is, the node to be deleted is determined from at least one first node with a low reuse probability. KVs that will not be used later are deleted first, so as to retain nodes with a higher reuse probability. This is beneficial to improve the effective utilization of stored KVs and thus improve the effective utilization of storage space.

[0131] 503. If the preset conditions are met based on the storage space occupied by multiple nodes after the first deletion operation, determine the node to perform the second deletion operation based on all nodes included in the multiple nodes.

[0132] Step 503 is optional. Optionally, after performing the first deletion operation, the first device can determine whether a preset condition is met based on the storage space occupied by multiple nodes after the first deletion operation. If the determination result is yes, that is, if the preset condition is met, then step 503 is executed, that is, the second deletion operation is performed on the nodes among the multiple nodes. If the determination result is no, that is, if the preset condition is not met, then the second deletion operation is not performed. In other words, in the process of determining the node to be deleted in the second deletion operation, the first node determined in step 501 is not retained.

[0133] For example, the first device determines at least one node that is currently in use from among a plurality of nodes after the first deletion operation; and determines at least one node that will perform a second deletion operation from among at least one node other than the node currently in use among the plurality of nodes after the first deletion operation; in other words, the node currently in use among the plurality of nodes is a node that cannot be deleted. The first device deletes the aforementioned at least one node, thereby releasing the storage space occupied by the aforementioned at least one node, that is, the second deletion operation is completed.

[0134] Specifically, for example, the first device uses a preset strategy to determine at least one node to perform the second deletion operation from at least one node other than the node currently in use among a plurality of nodes after the first deletion operation has been performed.

[0135] The specific implementation of the above steps can be found in the description of step 501 above. The difference is that "at least one node other than the first node and the node currently in use among the multiple nodes" in the above description is replaced with "at least one node other than the node currently in use among the multiple nodes after the first deletion operation is performed". This will not be described in detail here.

[0136] In this embodiment, if, after performing the first deletion operation, the preset conditions for performing deletion operations on multiple nodes are still met based on the storage space occupied by multiple nodes, at least one node for performing the second deletion operation is determined based on all nodes included in the multiple nodes. In other words, in the process of determining the node for performing the second deletion operation, the first node determined in step 501 is no longer retained. If there is insufficient storage space, the inference process executed by the model may not be able to continue. Therefore, although the key-value pairs in the first node have a high probability of being reused in subsequent inference processes, the first node is no longer retained when there is insufficient storage space. This helps to determine more nodes that can be deleted, thereby freeing up more storage space and ensuring the smoothness and stability of the inference process executed by the model.

[0137] Optionally, embodiments of this application also provide specific implementations using key-value pairs in multiple nodes; please refer to [link to relevant documentation]. Figure 10 , Figure 10 Another flowchart illustrating the KV data management method provided in this application. Figure 10 The KV data management method shown includes the following steps 1001 to 1003.

[0138] 1001. Obtain query information, including the first token.

[0139] For example, since the key-value pairs of the input token of the current inference process are used in each inference process performed by the model, in order to reuse the key-value pairs stored in multiple nodes, the first device will also obtain query information, which includes a first token. The first token in the query information is obtained based on the input token of the current inference process.

[0140] Optionally, if each of the multiple nodes is stored in a memory block, the number of tokens in the first token is less than or equal to M. For example, the input tokens of the current inference process are split into groups of M tokens to obtain at least one group of tokens arranged in order, and the first token is one of the aforementioned groups of at least M tokens.

[0141] Optionally, the query information may also include first session information, which includes the session identifier of the session to which the first token belongs. Further, if the session to which the first token belongs uses a breadth-first search algorithm, the first session information may also include stage information, indicating the current stage of the session to which the first token belongs. Alternatively, if the session to which the first token belongs uses a depth-first search algorithm, the first session information may also include attributes of the current inference process of the session to which the first token belongs. It should be noted that if the current inference process does not belong to any session, the query information may not include session information, or it may only include the first token.

[0142] Before detailing each step in steps 1001 to 1003, let's first introduce the overall approach of steps 1001 to 1003. For example, in order to obtain the key-value pairs (KV) of the input tokens for the current inference process, the first device can sequentially query whether each set of tokens in at least one set has successfully matched a token in multiple nodes, in order from front to back. If there is a match, the matching operation is performed on the next set of tokens. If not, the KV of the unmatched tokens needs to be generated and the newly generated KV is added to multiple nodes. The specific processing procedure for each matching is explained in subsequent steps 1002 and 1003.

[0143] For example, if the input tokens for the current inference process include tokens 1 to 7, and M is 3, then tokens 1 to 7 will be divided into three groups: the first group consists of tokens 1, 2, and 3; the second group consists of tokens 4, 5, and 6; and the third group consists of token 7. The first token in the first matching process is the first group of tokens. The first device checks if the first group of tokens can be successfully matched across multiple nodes. If a match is successful in the first matching process, the second matching process continues, and so on. If no match is found in the first matching process, key-value pairs (KV) for the first, second, and third groups of tokens are generated sequentially, and the three newly generated nodes are added to the multiple nodes. If a match is successful in the first matching process but not in the second, key-value pairs for the second and third groups of tokens are generated sequentially, and the two newly generated nodes are added to the multiple nodes. If a match is successful in the first and second matching processes, but not in the third, a third set of key-value pairs for the tokens can be generated, and the newly generated node can be added to multiple nodes. It should be understood that this example is for illustrative purposes only and is not intended to limit the scope of this solution.

[0144] After obtaining the query information, the first device can match the first token with tokens from multiple nodes to determine whether a second token exists among the multiple nodes that successfully matches the first token. This process can be performed using hash matching. If the first token successfully matches a second token from a second node among the multiple nodes, the process proceeds to step 1002; if no second token exists among the multiple nodes that successfully matches the first token, the process proceeds to step 1003.

[0145] Optionally, if multiple nodes use a prefix tree data storage structure, and the number of tokens in the first token is less than or equal to M. Each node in the prefix tree includes not only the token and its key-value pairs, but also a hash value. In one implementation, the hash value corresponding to each token in a node is obtained by hashing all tokens from the root node to that node. The first device performs a hash calculation on the first token and all tokens preceding it in the input tokens of the current inference process to obtain the first hash value.

[0146] The first device determines whether there exists a hash value in the prefix tree that is the same as the first hash value. If the hash value in a node is the same as the first hash value, then the first token matches the second token in that node, meaning the first token matches the second token in the second node included in that node. If there is no hash value in the prefix tree that is the same as the first hash value, then there is no node among multiple nodes that matches the first token, meaning there is no token among multiple nodes that matches the first token.

[0147] Alternatively, in another implementation, the hash value in each node is obtained by hashing all tokens in that node. The first device hashes the first token to obtain the first hash value. The first device can determine a node (hereinafter referred to as the "first node") that successfully matches a group of tokens preceding the first token in the input tokens of the current inference process. It then checks if any child node of the first node has a hash value identical to the first hash value. If a child node of the first node has a hash value identical to the first hash value, then the first token successfully matches the second token in that node, meaning the first token successfully matches the second token in the second node included in that node. If none of the child nodes of the first node have a hash value identical to the first hash value, then none of the multiple nodes successfully match the first token, meaning none of the multiple nodes successfully match the first token.

[0148] For example, if the input tokens for the current inference process include tokens 1 to 7, and M is 3, then tokens 1 to 7 will be divided into 3 groups: the first group consists of tokens 1, 2, and 3; the second group consists of tokens 4, 5, and 6; and the third group consists of token 7. The prefix tree includes nodes 1 to 5, where the root node's children include nodes 1 and 5, node 1's children include nodes 2 and 3, and node 2's children include node 4. The first token in the first matching process is the first group of tokens, and this first group of tokens successfully matches the second token included in node 1 of the prefix tree.

[0149] In the second matching process, the first token becomes the second set of tokens. The first device obtains the first hash value of the second set of tokens. Since node 1 includes two child nodes, node 2 and node 3, we take the hash value of the three tokens in node 2 as hash value 1 and the hash value of the three tokens in node 3 as hash value 2 as an example. In the second matching process, the first device compares the first hash value with hash value 1 and hash value 2. If the first hash value is the same as hash value 1, it means that the second set of tokens has successfully matched the second token in node 2; if the first hash value is the same as hash value 2, it means that the second set of tokens has successfully matched the second token in node 3; if the first hash value is different from both hash value 1 and hash value 2, it means that there is no token in multiple nodes that successfully matches the second set of tokens. It should be understood that this example is for ease of understanding and is not intended to limit this scheme.

[0150] In this embodiment, when the first token is matched with tokens in multiple nodes, a hash matching method is used, which helps improve the efficiency of the aforementioned matching process. Optionally, the multiple nodes adopt a prefix tree data storage structure. The hash value corresponding to the token in each node of the prefix tree is obtained by hashing all tokens from the root node to that node. The first hash value corresponding to the first token is obtained by hashing the first token and all tokens before the first token in the input token of the current inference process. If the first hash value is the same as the hash value corresponding to the second token in a certain node, it can be proven that all tokens from the root node to that node in the prefix tree and the first token and all tokens before the first token in the input token of the current inference process are completely matched, thereby improving the accuracy of the hash matching process.

[0151] 1002. If the first token matches the second token in a second node that is included in multiple nodes, the key-value pair of the second token is used as the key-value pair of the first token.

[0152] For example, if the first token successfully matches a second token in a second node that includes multiple nodes, the first device obtains the key-value pair (KV) of the second token. The KV of the second token is then used as the KV of the first token, eliminating the need to regenerate the KV of the first token. Optionally, if the query information includes first session information, the first device also adds the first session information to the second node. Optionally, if multiple nodes use a prefix tree data storage structure, this can be understood as the first device adding the first session information to the node containing the KV of the second token. The first session information indicates that the second node is associated with the session to which the first token belongs; that is, the first session information indicates that the KV in the second node is used by the session to which the first token belongs.

[0153] Furthermore, if the session to which the first token belongs uses a breadth-first search algorithm, the first session information also indicates which stage of the session the second node is associated with. This facilitates determining whether the key-value pairs in the second node are associated with the current stage of the session when performing a deletion operation, thus more accurately determining whether the second node needs to be retained. Alternatively, if the session to which the first token belongs uses a depth-first search algorithm, the first session information also indicates which attribute's inference process in the session to which the second node is associated with. This facilitates determining whether the key-value pairs in the second node are associated with the inference process of the session where the attribute is branched during deletion, thus more accurately determining whether the second node needs to be retained. In other words, the goal is still to retain only key-value pairs with high reuse probability across multiple nodes, thereby improving the effective utilization of storage space and increasing the probability of successful matching of key-value pairs stored across multiple nodes, thus improving the efficiency of the model inference process.

[0154] Optionally, the input token of the current inference process includes at least two sets of tokens. The first token includes one set of tokens from the input tokens of the current inference process. Then, after the first device completes step 1002, in order to obtain the key-value pair of the next set of tokens from the aforementioned at least two sets of tokens from multiple nodes, it can re-enter step 1001.

[0155] In this embodiment, when querying the key-value pairs (KV) of a first token from multiple nodes, if the first token matches a second token in one of the multiple nodes (i.e., the second node), the KV of the second token can be directly used as the KV of the first token, eliminating the need to generate a new KV for the first token. This improves the efficiency of the model inference process. Furthermore, first session information is added to the second node. This first session information includes the session identifier of the session to which the first token belongs, indicating that the KV in the second node is associated with the session to which the first token belongs. In other words, it indicates that the KV in the second node was used by the session to which the first token belongs. When deleting multiple nodes, the first session information can be used to determine whether the second node needs to be retained. In other words, this application not only provides how to use session information to determine which nodes need to be retained but also how to add session information to existing nodes across multiple nodes, improving the completeness of this solution.

[0156] 1003. If there is no token that matches the first token among multiple nodes, add a third node among the multiple nodes. The third node includes the first token and the key-value pair of the first token.

[0157] For example, if no token matching the first token is found among multiple nodes, the first device can obtain the key-value pair (KV) of the first token and add a third node among the multiple nodes. The third node includes the first token and its KV. Optionally, if the query information includes first session information, the third node also includes the first session information.

[0158] Specifically, the first session information indicates that the third node is associated with the session to which the first token belongs; that is, the first session information indicates that the key-value pairs in the third node are used by the session to which the first token belongs. Furthermore, if the session to which the first token belongs uses a breadth-first search algorithm, the first session information also indicates which stage of the session the third node is associated with. Alternatively, if the session to which the first token belongs uses a depth-first search algorithm, the first session information also indicates which attribute's reasoning process within the session to which the third node is associated with.

[0159] Optionally, if multiple nodes employ a prefix tree data storage structure, in one scenario, if a set of tokens preceding the first token in the input tokens of the current inference process matches successfully, adding a third node to the multiple nodes by the first device may include: adding a child node (i.e., a third node) to a node where a set of tokens preceding the first token in the input tokens of the current inference process matches successfully, and storing the first token and its key-value pair. In another scenario, if the first token is the first set of tokens in the input tokens of the current inference process, adding a third node to the multiple nodes by the first device may include: adding a child node (i.e., a third node) to the root node.

[0160] Optionally, the input token for the current inference process includes at least two sets of tokens. The first token includes one set of tokens from the input tokens of the current inference process. If there is at least one more set of tokens after the first token among the aforementioned at least two sets of tokens, the first device further obtains the key-value pairs (KV) of each set of tokens after the first token, and adds at least one new node corresponding one-to-one with the at least one set of tokens among multiple nodes. Each new node includes a set of tokens after the first token and the KV of the aforementioned set of tokens. Optionally, each new node also includes first session information.

[0161] Optionally, if multiple nodes use a prefix tree data storage structure, child nodes can be added to the third node to store the new node corresponding to the next set of tokens of the first token. Optionally, if there is at least one more set of tokens after the next set of tokens of the first token, then child nodes can be added to the child nodes of the third node, and so on.

[0162] To understand this solution more intuitively, please refer to [link / reference]. Figure 11 , Figure 11 This is a schematic diagram illustrating the addition of a node among multiple nodes, as provided in an embodiment of this application. Figure 11 Taking a data storage structure with multiple nodes using a prefix tree as an example, Figure 11 (a) The sub-diagram represents the prefix tree before the addition of nodes. Figure 11 The sub-diagram (a) can be combined with the above-mentioned... Figure 6The explanation above will be provided beforehand, and will not be repeated here. The input token in the previous inference process represents "Are you planning to hike the XX Greenway this weekend?" The input tokens from the previous inference process are divided into 5 groups of 3 tokens each. The meanings of the 5 groups are "This weekend", "You plan to", "Go to XX", "Greenway hike", and "Walk?". The two groups of tokens meaning "This weekend" and "You plan to" successfully match the tokens in the prefix tree. The three groups of tokens meaning "Go to XX", "Greenway hike", and "Walk?" do not match the tokens in the prefix tree.

[0163] Figure 11 (b) represents the prefix tree after adding nodes, in contrast. Figure 11 As shown in sub-diagrams (a) and (b), a child node is added to the node corresponding to the token whose text means "You intend to" to store the key-value pairs of the token whose text means "Go to XX". A child node is also added to the node corresponding to the token whose text means "Go to XX" to store the key-value pairs of the token whose text means "Greenway hiker". Finally, a child node is added to the node corresponding to the token whose text means "Greenway hiker" to store the key-value pairs of the token whose text means "Walk?". It should be understood that... Figure 11 The examples in this document are for illustrative purposes only and are not intended to limit the scope of this solution.

[0164] In this embodiment, when querying the key-value pair of the first token from multiple nodes, if no token matching the first token exists in the multiple nodes, a new node can be added. The new node includes the first token, the key-value pair of the first token, and the session information of the session to which the first token belongs (i.e., the aforementioned first session information). This application not only provides how to add session information to existing nodes in multiple nodes, but also how to add session information when adding a new node in multiple nodes. In short, it provides how to ensure that a single node in multiple nodes can contain session information under different circumstances, so as to determine which nodes need to be retained by using the session information. This not only improves the completeness of this solution, but also expands the application scenarios of this solution.

[0165] Based on the above description, please refer to Figure 12 , Figure 12 Another flowchart illustrating the KV data management method provided in this application. Figure 12 Taking the first node as an example, which is associated with an active session. For example... Figure 12As shown, the first device obtains query information, which includes a first token and first session information. The first device matches the first token with multiple nodes included in the multiple nodes. If the first token successfully matches a second token in a second node included in the multiple nodes, the key-value pairs of the second token are used as the key-value pairs of the first token, and the first session information is added to the second node. If no token successfully matches the first token in the multiple nodes, before adding a third node, it can determine whether a preset condition is met based on the storage space occupied by the multiple nodes. If the determination result is negative, a third node is added to the multiple nodes. The third node includes the first token, the key-value pairs of the first token, and the first session information.

[0166] If the determination result is yes, it means that at least one node among multiple nodes needs to be deleted. The first device can determine at least one node that is currently in use from the multiple nodes, and each node that is currently in use cannot be deleted. The first device will also determine at least one first node from the multiple nodes, and the session information in each first node indicates that the key-value pair in the first node is associated with an active session. The first device determines at least one node to perform the first deletion operation from the nodes that are currently in use among the multiple nodes and at least one node other than the first node.

[0167] After performing the first deletion operation, the first device will determine whether the preset conditions are met based on the storage space occupied by multiple nodes after the first deletion operation. If the determination result is yes, the first device will determine at least one node from at least one node other than the node currently in use among the multiple nodes to perform the second deletion operation, and after performing the second deletion operation, add a third node among the multiple nodes. If the determination result is no, then after performing the first deletion operation, it is not necessary to perform the second deletion operation, and a third node will be added among the multiple nodes. It should be understood that... Figure 12 The examples in this document are for illustrative purposes only and are not intended to limit the scope of this solution.

[0168] exist Figures 1 to 12 Based on the corresponding embodiments, in order to better implement the above-described solutions of this application, related equipment for implementing the above solutions is also provided below. See details. Figure 13 , Figure 13This is a schematic diagram of a key-value data management device provided in an embodiment of this application. The key-value data management device 1300 includes: a determining unit 1301, configured to determine a first node from a plurality of nodes, each of the plurality of nodes including a key-value pair (KV), wherein the session information included in the first node indicates that the session associated with the KV in the first node is an active session, or the first node is a node not associated with an active session, wherein the session information at least includes a session identifier of the session, each session includes one or more inference processes executed by a model, and each session is used to obtain an answer corresponding to a user's question; the determining unit 1301 is further configured to determine the node that performs a first deletion operation based on the first node among the plurality of nodes.

[0169] Optionally, if the session information included in the first node indicates that the session associated with the KV in the first node is an active session, the determining unit 1301 is specifically used to determine the node that will perform the first deletion operation from among the nodes other than the first node among multiple nodes.

[0170] Optionally, if the first node is a node not associated with an active session, the determining unit 1301 is specifically configured to determine the node that will perform the first deletion operation from at least one first node included in a plurality of nodes.

[0171] Optionally, the first node includes session information of a first session, wherein the first session is an active session, the first session employs a breadth-first search algorithm, the first session includes at least one stage of the reasoning process, and the session information of the first session indicates that the key-value pairs in the first node are associated with the current stage in at least one stage of the first session; and / or,

[0172] The first node includes the session information of the second session, which is an active session. The second session uses a depth-first search algorithm and includes an inference process with the attribute of branch expansion and an inference process with the attribute of simulated rollout. The session information of the second session indicates that the key-value pairs in the first node are associated with the inference process with the attribute of branch expansion in the second session.

[0173] Optionally, if the preset conditions are met based on the storage space occupied by multiple nodes after the first deletion operation is performed, the determining unit 1301 is further configured to determine the node to perform the second deletion operation based on all nodes included in the multiple nodes.

[0174] Optionally, the key-value pairs (KV) in each of the multiple nodes are the KV of the token, and each node also includes a token. The key-value data management device 1300 further includes: an acquisition unit 1302, used to acquire query information, the query information including a first token and first session information, the first session information including the session identifier of the session to which the first token belongs; and a processing unit 1303, used to add the first session information to the second node if the first token successfully matches the second token in the second node included in the multiple nodes, and the KV of the second token included in the second node is used as the KV of the first token.

[0175] Optionally, if the session to which the first token belongs uses a breadth-first search algorithm, the first session information also indicates the current stage of the session to which the first token belongs; or, if the session to which the first token belongs uses a depth-first search algorithm, the first session information also includes the attribute of the current inference process of the session to which the first token belongs.

[0176] Optionally, a successful match between the first token and the second token in a second node, which includes multiple nodes, means that the first hash value and the second hash value are the same, wherein the first hash value is obtained based on the first token and the second hash value is obtained based on the second token in the second node.

[0177] Optionally, the key-value pairs in each of the multiple nodes are the key-value pairs of the token, and each node also includes the token. The key-value data management device 1300 further includes: an acquisition unit 1302, used to acquire query information, the query information including the first token and the first session information, the first session information including the session identifier of the session to which the first token belongs; and a processing unit 1303, used to add a third node among the multiple nodes if there is no token that successfully matches the first token among the multiple nodes, the third node including the first token, the key-value pairs of the first token and the first session information.

[0178] It should be noted that the information interaction and execution process between the modules / units in the key-value data management device 1100 are different from those in this application. Figures 1 to 12 The various method embodiments are based on the same concept, and the details can be found in the descriptions of the method embodiments shown above in this application, which will not be repeated here.

[0179] The following describes a device provided in an embodiment of this application. Please refer to [link / reference]. Figure 14 , Figure 14 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Optionally, device 1400 performs... Figures 1 to 12 The functions of the first device in the corresponding method embodiments.

[0180] Device 1400 includes a memory 1402 and at least one processor 1401. Optionally, device 1400 further includes at least one accelerator 1403. Optionally, processor 1401 implements the method in the above embodiments by reading program instructions stored in memory 1402; or, processor 1401 reads program instructions stored in memory 1402 and implements the steps executed by the machine learning model in the method in the above embodiments through accelerator 1403; or, processor 1401 may also implement the method in the above embodiments by reading program instructions stored internally; or, processor 1401 may also read program instructions stored internally and implement the steps executed by the machine learning model in the method in the above embodiments through accelerator 1403.

[0181] When the processor 1401 reads the program instructions stored in the memory 1402 to implement the method in the above embodiments, the memory 1402 stores the program instructions that implement the method provided in the above embodiments of this application.

[0182] Optionally, at least one processor 1401 is one or more CPUs, either a single-core CPU or a multi-core CPU. For example, memory 1402 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, or optical memory. Memory 1402 stores program instructions for the operating system. For example, at least one accelerator 1403 may include at least one of the following: GPU, NPU, TPU, ASIC, FPGA, or other types of accelerators. After the program instructions stored in memory 1402 are read by the at least one processor 1401, device 1400 executes the corresponding operations in the foregoing embodiments.

[0183] Optionally, device 1400 also includes a network interface 1404, which can be a wired interface or a wireless interface, and is used for... Figures 1 to 12 The corresponding method implementations execute the sending and receiving of data.

[0184] It should be understood that network interface 1404 has the functions of receiving and sending data. The functions of "receiving data" and "sending data" can be integrated into the same transceiver interface, or the functions of "receiving data" and "sending data" can be implemented in different interfaces, which is not limited here. In other words, network interface 1404 may include one or more interfaces for implementing the functions of "receiving data" and "sending data".

[0185] After the processor 1401 reads the program instructions from the memory 1402, other functions that the device 1400 can perform are described in the preceding method embodiments.

[0186] Optionally, the device 1400 also includes a bus 1405, through which the processor 1401 and memory 1402 are typically interconnected, or in other ways.

[0187] The device 1400 provided in this application embodiment is used to execute the method executed by the first device in the above-described method embodiments and achieve the corresponding beneficial effects. Figure 14 The specific implementation of the device 1400 shown can be referred to the descriptions in the aforementioned method embodiments, and will not be repeated here.

[0188] This application embodiment also provides a system that stores KV data. The system includes multiple devices, including at least one first device, which performs the aforementioned actions. Figures 1 to 12 The steps performed by the first device in the method described in the illustrated embodiment.

[0189] For example, KV data is stored in the local memory of the devices in the system, and / or, KV data is stored in the shared memory of the system.

[0190] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the aforementioned actions. Figures 1 to 12 The steps performed by the first device in the method described in the illustrated embodiment.

[0191] This application also provides a computer program product, which includes a program that, when run on a computer, causes the computer to perform the aforementioned actions. Figures 1 to 12 The steps performed by the first device in the method described in the illustrated embodiment.

[0192] This application embodiment also provides a circuit system, the circuit system including a processing circuit, the processing circuit being configured to perform the aforementioned... Figures 1 to 12 The steps performed by the first device in the method described in the illustrated embodiment.

[0193] The first device or key-value data management device provided in this application embodiment can specifically be a chip. The chip includes a processing unit, which may be, for example, a processor. Optionally, the chip also includes a communication unit, which may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip to perform the above-mentioned operations. Figures 1 to 12The method described in the illustrated embodiment. Optionally, the storage unit is a storage unit within the chip, such as a register, cache, etc. The storage unit can also be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM), etc.

[0194] The processor mentioned above can be a general-purpose central processing unit, microprocessor, GPU, NPU, TPU, ASIC, FPGA, or one or more integrated circuits used to control the execution of the program in the first aspect of the above method.

[0195] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0196] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CLUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0197] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0198] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. A method for managing key-value (KV) data, characterized in that, The method includes: A first node is determined from a plurality of nodes, each of which includes a key-value pair (KV). The session information included in the first node indicates that the session associated with the KV in the first node is an active session, or the first node is a node not associated with an active session. The session information includes at least a session identifier of the session. Each session includes one or more inference processes performed by the model, each session being used to obtain an answer corresponding to a user's question. Based on the first node among the plurality of nodes, the node to perform the first deletion operation is determined.

2. The method according to claim 1, characterized in that, If the session information included in the first node indicates that the session associated with the KV in the first node is an active session, the step of determining the node to perform the first deletion operation based on the first node among the plurality of nodes includes: From the nodes other than the first node among the plurality of nodes, determine the node that will perform the first deletion operation.

3. The method according to claim 1, characterized in that, In the case that the first node is not associated with an active session, determining the node to perform the first deletion operation based on the first node among the plurality of nodes includes: The node that performs the first deletion operation is determined from at least one of the first nodes included in the plurality of nodes.

4. The method according to claim 2, characterized in that, The first node includes session information of a first session, which is an active session. The first session uses a breadth-first search algorithm and includes at least one stage of reasoning process. The session information of the first session indicates that the key-value pairs in the first node are associated with the current stage in at least one stage of the first session. And / or, The first node includes session information of the second session, which is an active session. The second session uses a depth-first search algorithm and includes an inference process with the attribute of branch expansion and an inference process with the attribute of simulated rollout. The session information of the second session indicates that the key-value pairs in the first node are associated with the inference process with the attribute of branch expansion in the second session.

5. The method according to any one of claims 1 to 4, characterized in that, If, based on the storage space occupied by the plurality of nodes after performing the first deletion operation, it is determined that a preset condition is met, the method further includes: Based on all the nodes included in the plurality of nodes, the node to perform the second deletion operation is determined.

6. The method according to any one of claims 1 to 4, characterized in that, The key-value pair (KV) in each of the plurality of nodes is the KV of the token, and each of the plurality of nodes also includes a token. The method further includes: Obtain query information, which includes a first token and first session information, wherein the first session information includes the session identifier of the session to which the first token belongs; If the first token successfully matches the second token in the second node included in the plurality of nodes, the first session information is added to the second node, and the key-value pairs of the second token included in the second node are used as the key-value pairs of the first token.

7. The method according to claim 6, characterized in that, If the session to which the first token belongs uses a breadth-first search algorithm, the first session information also indicates the current stage of the session to which the first token belongs; or, If the session to which the first token belongs uses a depth-first search algorithm, the first session information also includes attributes of the current inference process of the session to which the first token belongs.

8. The method according to claim 6, characterized in that, A successful match between the first token and the second token in the second node, which includes the plurality of nodes, means that the first hash value and the second hash value are the same, wherein the first hash value is obtained based on the first token and the second hash value is obtained based on the second token in the second node.

9. The method according to any one of claims 1 to 4, characterized in that, The key-value pairs in each of the plurality of nodes are key-value pairs of the token, and each of the plurality of nodes also includes the token. The method further includes: Obtain query information, which includes a first token and first session information, wherein the first session information includes the session identifier of the session to which the first token belongs; If no token that matches the first token is found among the plurality of nodes, a third node is added among the plurality of nodes. The third node includes the first token, the key-value pair of the first token, and the first session information.

10. A management device for key-value (KV) data, characterized in that, The device includes: A determining unit is configured to determine a first node from a plurality of nodes, each of the plurality of nodes including a key-value pair (KV). The session information included in the first node indicates that the session associated with the KV in the first node is an active session, or the first node is a node not associated with an active session. The session information includes at least a session identifier of the session. Each session includes one or more inference processes performed by a model, each session being used to obtain an answer corresponding to a user's question. The determining unit is further configured to determine the node that performs the first deletion operation based on the first node among the plurality of nodes.

11. The apparatus according to claim 10, characterized in that, If the session information included in the first node indicates that the session associated with the KV in the first node is an active session, the determining unit is specifically used to determine the node that performs the first deletion operation from among the nodes other than the first node among the plurality of nodes.

12. The apparatus according to claim 10, characterized in that, In the case that the first node is not associated with an active session, the determining unit is specifically configured to determine the node that performs the first deletion operation from at least one of the first nodes included in the plurality of nodes.

13. The apparatus according to claim 11, characterized in that, The first node includes session information of a first session, which is an active session. The first session uses a breadth-first search algorithm and includes at least one stage of reasoning process. The session information of the first session indicates that the key-value pairs in the first node are associated with the current stage in at least one stage of the first session. And / or, The first node includes session information of the second session, which is an active session. The second session uses a depth-first search algorithm and includes an inference process with the attribute of branch expansion and an inference process with the attribute of simulated rollout. The session information of the second session indicates that the key-value pairs in the first node are associated with the inference process with the attribute of branch expansion in the second session.

14. The apparatus according to any one of claims 10 to 13, characterized in that, If the preset conditions are met based on the storage space occupied by the plurality of nodes after the first deletion operation is performed, the determining unit is further configured to determine the node to perform the second deletion operation based on all the nodes included in the plurality of nodes.

15. The apparatus according to any one of claims 10 to 13, characterized in that, The key-value pair (KV) of each of the plurality of nodes is the KV of the token, and each node also includes a token. The device further includes: The acquisition unit is used to acquire query information, which includes a first token and first session information, wherein the first session information includes the session identifier of the session to which the first token belongs; The processing unit is configured to add the first session information to the second node if the first token successfully matches the second token in the second node included in the plurality of nodes, and the key-value pairs of the second token included in the second node are used as the key-value pairs of the first token.

16. The apparatus according to claim 15, characterized in that, If the session to which the first token belongs uses a breadth-first search algorithm, the first session information also indicates the current stage of the session to which the first token belongs; or, If the session to which the first token belongs uses a depth-first search algorithm, the first session information also includes attributes of the current inference process of the session to which the first token belongs.

17. The apparatus according to claim 15, characterized in that, A successful match between the first token and the second token in the second node, which includes the plurality of nodes, means that the first hash value and the second hash value are the same, wherein the first hash value is obtained based on the first token and the second hash value is obtained based on the second token in the second node.

18. The apparatus according to any one of claims 10 to 13, characterized in that, The key-value pair in each of the plurality of nodes is the key-value pair of the token, and each node also includes a token. The device further includes: The acquisition unit is used to acquire query information, which includes a first token and first session information, wherein the first session information includes the session identifier of the session to which the first token belongs; The processing unit is configured to add a third node among the plurality of nodes if no token matching the first token is found among the plurality of nodes. The third node includes the first token, the key-value pairs of the first token, and the first session information.

19. A device, characterized in that, The device includes a processor and a memory, the memory storing code, the processor being configured to execute the code, and when the code is executed, the processor performing the method as described in any one of claims 1 to 9.

20. A system, characterized in that, The system stores key-value (KV) data and includes multiple devices. At least one device in the system is used to perform the KV data management method as described in any one of claims 1 to 9.

21. The system according to claim 20, characterized in that, The KV data is stored in the local memory of the devices in the system, and / or the KV data is stored in the shared memory of the system.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 9.

23. A computer program product, characterized in that, The computer program product includes a program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 9.

Images