Large model retrieval processing method and device

Abstract

The invention discloses a large model retrieval processing method and device. The method comprises the following steps: obtaining query data; determining K document blocks reaching a preset similarity with the query data, wherein K is a positive integer; dividing the K document blocks into at least one set, wherein the correlation among the document blocks contained in each set meets a set condition; key value data of the at least one set is obtained at least through a key value storage structure, the key value storage structure stores the key value data in a tree structure mode, and the connection relation between nodes in the tree structure represents the dependency relation between the different key value data. The key value data stored in the key value storage structure are cached in different types of cache modes.

Description

Technical Field

[0001] The present application relates to large model retrieval technology, and more specifically, to a large model retrieval processing method and device. Background Art

[0002] Large model technology has been widely used in various fields. It faces huge challenges in data real-time, dynamic, privacy and context length limitation in professional scenarios and industry segments. In this context, RAG (Retrieval-augmented Generation) technology was proposed.

[0003] RAG can find K documents that are most relevant to user input through various means, such as semantic search, keyword search, and search based on knowledge graphs, and then input these documents into the big model together with the user's question as the context of the user's question. The big model analyzes and understands the input and generates appropriate responses. The big model's analysis and understanding process requires the generation of KV data for each document. In order to shorten the response time, the previously generated KV data can be cached and reused. However, the current KV caching method has certain problems, such as the cache mode ignores the correlation between each document, resulting in poor accuracy of the final model output result; based on the absolute position encoding storage of each document, the storage structure is too complex and the cost is too high. Summary of the invention

[0004] In view of this, this application provides the following technical solutions:

[0005] The first aspect of the present application provides a large model retrieval processing method, the method comprising:

[0006] Get query data;

[0007] Determine K document blocks that have a preset similarity with the query data, where K is a positive integer;

[0008] Dividing the K document blocks into at least one set, wherein the correlation between the document blocks contained in each set meets a set condition;

[0009] The key-value data of at least one set is obtained at least through a key-value storage structure, the key-value storage structure stores the key-value data in a tree structure, the connection relationship between nodes in the tree structure represents the dependency relationship between different key-value data, and the key-value data stored in the key-value storage structure is cached using different types of cache modes.

[0010] In a possible implementation, dividing the K document blocks into at least one set includes:

[0011] Determining the inter-entity semantic relevance between the K document blocks;

[0012] The document blocks whose semantic relevance between the entities exceeds a first threshold are divided into a collection.

[0013] In a possible implementation, obtaining the key-value data of the at least one set at least through a key-value storage structure includes:

[0014] Determine coverage of the set by the key-value data stored in the key-value storage structure, wherein the coverage represents a proportion of document blocks belonging to the set that have key-value data in the key-value storage structure among all document blocks in the set;

[0015] The key-value data of the collection is obtained based on the coverage.

[0016] In a possible implementation, obtaining the key-value data of the set based on the coverage includes:

[0017] If the coverage represents that there is a key value cache corresponding to all document block combinations in the current collection in the key value storage structure, reuse the key value cache corresponding to all document block combinations as the key value data of the current collection;

[0018] If the coverage represents that there is a key value cache corresponding to a partial combination of document blocks in the current collection in the key value storage structure, the key value cache corresponding to the partial combination of document blocks is reused in combination with the reasoning of document blocks not covered by the key value storage structure to determine the key value data of the current collection;

[0019] If the coverage represents that there is no key value cache corresponding to any document block combination in the current collection in the key value storage structure, the key value data of the document block combination in the current collection is obtained by inference.

[0020] In a possible implementation, it also includes:

[0021] If the key-value data obtained is key-value data that does not exist in the key-value storage structure, the key-value data is saved in the key-value storage structure.

[0022] In a possible implementation, the storing the key-value data in the key-value storage structure includes at least one of the following:

[0023] If the key-value data is data obtained by reasoning for a document block alone, a target node with the root node as the parent node is added to the tree structure, and the key-value data is saved in the target node;

[0024] If the key-value data is data obtained by processing a document block in combination with a target key-value cache already existing in the tree structure, a target node is added to the end of the path corresponding to the target key-value cache in the tree structure, and the key-value data is saved in the target node.

[0025] In a possible implementation, it also includes:

[0026] If there are multiple key-value cache paths corresponding to some document combinations in the current collection in the key-value storage structure, determine the optimal path among the key-value cache paths, where the optimal path is determined based on the reuse cost of the key-value cache paths;

[0027] The reuse cost is related to the migration time of the key-value cache data in the key-value cache path to the graphics processor and the computing power required to determine the key-value data of the document block not covered by the key-value storage structure.

[0028] In a possible implementation, in the key-value storage structure, the key-value data of multiple document blocks in the same collection are stored in a full-sequence cache mode, and there is a dependency relationship between the key-value data stored in the full-sequence cache mode; the key-value data of document blocks in different collections are stored in an independent cache mode.

[0029] In a possible implementation, determining K document blocks having a preset similarity with the query data includes:

[0030] Performing vectorization processing on the query data to obtain vectorized data;

[0031] K document blocks whose semantic similarity with the vectorized data meets a preset condition are retrieved from a preset vector database.

[0032] The second aspect of the present application provides a large model retrieval processing device, the device comprising:

[0033] A query acquisition module is used to obtain query data;

[0034] A document block determination module, used to determine K document blocks having a preset similarity with the query data, where K is a positive integer;

[0035] A set partitioning module, used for partitioning the K document blocks into at least one set, wherein the correlation between the document blocks contained in each set meets a set condition;

[0036] A key value acquisition module is used to obtain the key value data of at least one set through at least a key value storage structure, wherein the key value storage structure stores the key value data in a tree structure, and the connection relationship between nodes in the tree structure represents the dependency relationship between different key value data. The key value data stored in the key value storage structure is cached using different types of cache modes. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0038] Figure 1 A flowchart of a large model retrieval processing method disclosed in an embodiment of the present application;

[0039] Figure 2 A schematic diagram of a key-value storage structure disclosed in an embodiment of the present application;

[0040] Figure 3 An example diagram of reasoning attention masks for different cache modes disclosed in an embodiment of the present application;

[0041] Figure 4 A flowchart of obtaining key value data of a set disclosed in an embodiment of the present application;

[0042] Figure 5 A schematic diagram of the implementation structure of the large model retrieval processing solution disclosed in the embodiment of the present application;

[0043] Figure 6 A schematic diagram of the structure of a large model retrieval processing device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0044] For the purpose of reference and clarity, the descriptions, abbreviations or acronyms of the technical terms used below are summarized as follows:

[0045] KV Cache: KV Cache (Key-Value Cache) is an optimization technology used by the Transformer model in the autoregressive decoding process, which aims to reduce repeated calculations and improve reasoning efficiency.

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] Figure 1 This is a flowchart of a large model retrieval processing method disclosed in an embodiment of the present application. Figure 1 As shown, the large model retrieval processing method may include:

[0048] Step 101: Obtain query data.

[0049] The query data may be a query condition entered by the user in the search bar of the large model, or a query condition entered by the user in the search bar of other applications with a search function in the electronic device deployed with the large model. The query data may be a complete question or a search keyword, and this application does not impose any fixed restrictions on this.

[0050] The large model may be a large model product including a RAG (Retrieval-Augmented Generation) model and applied to various fields.

[0051] Step 102: Determine K document blocks that have a preset similarity with the query data, where K is a positive integer.

[0052] For the query data input by the user, the large model can first vectorize it to obtain vectorized data, and retrieve the K document blocks with the highest semantic similarity to the vectorized data in the preset vector database. The document block is the basic unit for calculating key values ​​in the large model, but the calculation of the key value data of the document block is not necessarily independent. It can be understood that for document blocks that are not highly correlated with other document blocks, their key value data can be calculated separately. For document blocks that are highly correlated with other document blocks, in order to ensure the accuracy of the final reasoning results of the large model, different document blocks with high correlation need to be combined together to jointly infer the key value data of each document block, and retain the correlation between different document blocks.

[0053] Step 103: Divide the K document blocks into at least one set, and the correlation between the document blocks included in each set meets the set conditions.

[0054] After obtaining K document blocks, the K document blocks are first divided into sets according to the correlation between each document block in the K document blocks. The number of sets is not fixed and is determined based on the correlation between each document block. In one extreme case, the K document blocks are divided into K sets, each of which contains only one document block; in another extreme case, the K document blocks are divided into one set, which contains K document blocks. The number of document blocks contained in each set can be the same or different, and there is no fixed limit.

[0055] The specific implementation of dividing the K document blocks into at least one set will be described in detail in the following embodiments, which will not be introduced in detail here.

[0056] Step 104: obtaining the key-value data of at least one set through at least a key-value storage structure, wherein the key-value storage structure stores the key-value data in a tree structure, wherein the connection relationship between nodes in the tree structure represents the dependency relationship between different key-value data, and the key-value data stored in the key-value storage structure is cached using different types of cache modes.

[0057] Since the connection relationship between nodes in the tree structure represents the dependency relationship between different key-value data, the tree structure can also be called a KV cache dependency tree.

[0058] In this application, each of the sets is used as a basic unit to determine the key value data of each set. In the previous processing, the large model can cache the key value data obtained by reasoning in the key value storage structure. The key value data stored in the key value storage structure is called KV cache. In this way, when determining the key value data of each set in the subsequent processing, first determine whether the KV cache of the key value storage structure contains the key value data of the current set. If so, the corresponding KV cache can be reused directly; if not, the large model needs to determine it through inference calculation.

[0059] Figure 2 This is a schematic diagram of the key-value storage structure disclosed in the embodiment of the present application, combined with Figure 2 As shown in Figure 1, the key-value storage structure stores key-value data in the form of a tree structure; each node in the tree structure stores the key-value data of a document block; the connection relationship between nodes in the tree structure represents the dependency relationship between different key-value data. Among them, S is the root node, and the nodes connected to the root node alone, such as C5, C7, C8, C1, C3, and C2, are independent KV caches ( Figure 2 The nodes in the dotted box in the upper middle section are the key-value data of the document block corresponding to a collection. Figure 2There is a path including multiple nodes on the right side, which can also be called a key-value cache path, such as C1-C2-C4, C3-C1-C5, C3-C1-C7, etc. Each path corresponds to a collection, and each node in the path corresponds to each document block in the collection. The key-value data of each node in the key-value cache path containing at least two nodes has a dependency relationship, and a non-independent KV cache is used, that is, a full sequence cache ( Figure 2 Nodes within the dashed box in the lower middle).

[0060] That is, in the key-value storage structure, the key-value data of multiple document blocks in the same collection are stored in a full-sequence cache mode, and there is a dependency relationship between the key-value data stored in the full-sequence cache mode; the key-value data of document blocks in different collections are stored in an independent cache mode.

[0061] The key-value storage structure manages the dependencies between the KV caches of document blocks in the form of a tree structure. Each node in the tree structure represents the KV cache of a document block, and the parent and child nodes on each path represent the order of KV caches when reused. The KV caches represented by nodes on different paths do not need to be considered in order when reused.

[0062] Figure 3 This is an example diagram of the inference attention mask of different cache modes disclosed in the embodiment of this application. The left figure is the independent cache inference attention mask, the right figure is the full sequence cache inference attention mask, and the middle figure is the inference attention mask that includes both independent cache and full sequence cache, corresponding to the multi-level cache mode of this application solution.

[0063] Document blocks in the same collection have dense cross-attention, so the full-sequence cache mode is required. The cross-attention between document blocks in different collections is sparse, so the independent cache mode can be used. The key-value data using the independent cache mode occupies less storage space. The key-value data using the full-sequence cache mode retains the association relationship, or dependency relationship, between different key-value data (different document blocks), which helps to improve the accuracy of large model RAN inference results.

[0064] The large model retrieval processing method described in this embodiment divides K document blocks that meet the similarity requirements with the query data into at least one set, and determines the key-value data of each set with the set as the basic unit. In the implementation, the key-value cache in the key-value storage structure can be reused to speed up the overall reasoning speed of the large model; the key-value data stored in the key-value storage structure are cached using different types of cache modes, and the key-value data corresponding to the document blocks with high relevance are represented by the connection relationship between the nodes in the tree structure. Due to the flexibility of the node connection in the tree structure, the key-value data with different relevance can be stored in different cache modes. This dynamic caching method can balance the cache resource consumption and improve the reuse rate of key-value data in the later stage under the premise of better maintaining the document context association.

[0065] In the above embodiment, dividing the K document blocks into at least one set may include: determining the inter-entity semantic correlation between the K document blocks; and dividing the document blocks whose inter-entity semantic correlation exceeds a first threshold into one set.

[0066] The present application scheme adopts a lightweight cross-document cross-attention estimation mechanism to optimize the KV cache reuse strategy. Using this estimation mechanism, the K document blocks retrieved from the vector database can be quickly divided into M sets, and the document blocks in each set are related to each other.

[0067] Specifically, first, the large model can be used to extract all entities in each document block, and each entity corresponds to a text description; then, the entities extracted from all document blocks are merged, and for entities that appear multiple times in different document blocks, the descriptions of the entity in all document blocks are spliced ​​together to form the final description of the entity; then, the entity description is vectorized and stored in the entity vector database. In addition, a mapping table can be stored to correspond to the entities contained in each document block.

[0068] When a user query arrives, K context documents are first retrieved from the vector database, and multiple entities corresponding to each document block are indexed according to the mapping table to form a set ε = {E1, ..., E K},in Indicates that there are n i d-dimensional entity vectors. Afterwards, the correlation between documents is calculated based on the extracted entity vectors. Specifically, the correlation between all document blocks is first calculated. The similarity between the i-th document block and the j-th document block can be expressed as:

[0069]

[0070] Set the similarity threshold δ, when S ij>δ, it means that the i-th document block is related to the j-th document block and is placed in the same collection. By adjusting the similarity threshold, the collection size can be implicitly adjusted. When the threshold is 0, it means that all documents are related, that is, the traditional full sequence cache is used. When the threshold is set to greater than 2, it means that all documents are unrelated, and each document is cached independently.

[0071] The above content introduces in detail the specific implementation of dividing K document blocks into at least one set. Technical personnel in the field can refer to the above content to better understand and implement the technical solution of the present application.

[0072] Figure 4 This is a flowchart of obtaining key value data of a collection disclosed in an embodiment of the present application. Figure 4 As shown, obtaining the key value data of the at least one set at least through the key value storage structure may include:

[0073] Step 401: Determine the coverage of the set by the key-value data stored in the key-value storage structure.

[0074] The coverage represents the proportion of document blocks belonging to the set that have key-value data in the key-value storage structure among all document blocks in the set.

[0075] For a collection that only includes one document block, since there is no correlation or poor correlation between the only document block in the collection and other document blocks, the key value data of the document block needs to be determined separately; if the key value data of the document block exists in the key value storage structure, this key value data needs to be the key value data stored in the node connected to the root node and not connected to other nodes in the tree structure. If the key value storage structure contains the independently cached key value data corresponding to this document block, the coverage of the key value data stored in the key value storage structure for the collection is 100%; if the key value storage structure does not contain the independently cached key value data corresponding to this document block, the coverage of the key value data stored in the key value storage structure for the collection is 0.

[0076] For a collection including at least two document blocks, since the document blocks in the collection have a high correlation, when determining the key value data of each document block in the collection, the dependency between the key value data corresponding to each document block needs to be considered, that is, if there is a key value cache of a partial combination of document blocks in the collection in the key value storage structure, the key value data of each document block in this partial combination of document blocks has a connection relationship in the tree structure. For example, the collection includes 4 document blocks A, B, C, and D, and there is a key value cache path in the key value storage structure, including A' node, B' node, and C' node, where A' is the key value data of document block A, B' is the key value data of document block B, and C' is the key value data of document block C, and there is no key value data for document block D in the key value storage structure, then the coverage of the key value data stored in the key value storage structure for the collection is 75%.

[0077] Step 402: Obtain key-value data of the collection based on the coverage.

[0078] After determining the coverage of the set by the key-value data stored in the key-value storage structure, the key-value data of the set can be obtained based on the coverage.

[0079] Specifically, if the coverage indicates that key-value caches corresponding to all document block combinations in the current collection exist in the key-value storage structure, the key-value caches corresponding to all document block combinations are reused as key-value data of the current collection.

[0080] If the coverage represents the existence of a key-value cache corresponding to a partial combination of document blocks in the current collection in the key-value storage structure, the key-value cache corresponding to the partial combination of document blocks is reused and combined with the reasoning of the document blocks not covered by the key-value storage structure to determine the key-value data of the current collection.

[0081] Continuing with the above example of 75% coverage, since document block D has a high correlation with document blocks A, B, and C, the key-value data of document block D needs to be combined with the key-value data of document blocks A, B, and C through a large model reasoning, rather than reasoning about the key-value data of document block D alone.

[0082] If the coverage represents that there is no key value cache corresponding to any document block combination in the current collection in the key value storage structure, the key value data of the document block combination in the current collection is obtained by inference.

[0083] If, in the process of obtaining the key value data of the at least one set, the key value data obtained is key value data that does not exist in the key value storage structure, the key value data can be saved in the key value storage structure, so that when the same document block appears in subsequent large model applications, there is no need to re-infer its key value data, and the corresponding key value data cached in the key value storage structure can be directly reused.

[0084] In one implementation, if the key-value data is data obtained by reasoning for a document block alone, a node with the root node as the parent node is added to the tree structure, and the meta information of the key-value data is saved in the node, such as the document block id, the address in the video memory / internal memory / extended memory, the size, the reuse frequency, etc. Figure 2 The C5 node in .

[0085] In another implementation, if the key-value data is data obtained by processing a document block in combination with a target key-value cache already existing in a tree structure, a target node is added to the end of the path corresponding to the target key-value cache in the tree structure, and the key-value data is saved in the target node.

[0086] If a key-value data is generated by combining a document block with other KV caches through large model reasoning, first find a path in the tree structure that contains all other KV caches and the order is consistent with the order of these KV caches when they were generated, add a node for the current key-value data and hang it at the end of this path. Figure 2 In other words, if the KVCache c8 corresponding to doc8 is obtained through large model reasoning in the order of KVCacheC3, KVCache c1, and doc8, then c8 should be appended to the end of the path s->C3->c1.

[0087] In the implementation, due to the limitations of device computing power, storage space and other conditions, it is necessary to dynamically manage the generated KV cache to achieve more efficient KV cache storage at the lowest storage cost. Therefore, the present application scheme designs a KV cache multi-level dynamic management strategy, involving KV cache search strategy and KV cache node release strategy, to minimize the delay in the process of inserting new nodes in the tree structure of the key-value cache structure and releasing the cache.

[0088] Regarding the search strategy of the KV cache, if there are multiple key-value cache paths corresponding to some document combinations in the current collection in the key-value storage structure, the optimal path among the key-value cache paths is determined, and the optimal path is determined based on the reuse cost of the key-value cache path. Among them, the reuse cost is related to the migration time of the key-value cache data in the key-value cache path to the graphics processor and the computing power required to determine the key-value data of the document block not covered by the key-value storage structure.

[0089] Specifically, after obtaining multiple document collections, find out whether there is a path corresponding to the collection in the tree structure of the key-value storage structure. Let's assume that the document collection is {Doc1, Doc3, Doc8}. Since the order of the context documents in the RAG has little effect on the generation result, recursively search for paths that cover all or part of these three documents in the cache tree. It is possible to find multiple candidate paths. At this time, by quantifying the "reuse cost" of each path, the most reasonable reused KV cache can be selected.

[0090] The migration time of the key-value cache data in the key-value cache path to the graphics processor can be understood as the reuse preparation time, and the computing power required to determine the key-value data of the document block not covered by the key-value storage structure can be understood as the generation cost based on reuse. In implementation, the reuse cost of the candidate path can be quantified from two dimensions: "reuse preparation time" and "generation cost based on reuse":

[0091] Reuse preparation time: Because some KVCaches on the path are stored in the host DRAM or extended memory, it takes time to migrate them to the GPU memory. Therefore, the migration time can be obtained by multiplying the node size (data volume) on the path by the migration speed. The total reuse preparation time is obtained by summing up the migration time of all nodes.

[0092] Generation cost based on reuse: If the KV cache on the reuse path cannot cover every document block in the document collection, the large model needs to be used to infer the uncovered document blocks. The computing power required for generation can be estimated based on the relevant parameters of the large model (such as the number of attention heads, the number of network layers, the dimension of the hidden layer vector, etc.) and the number of tokens in the document block.

[0093] In the implementation, the candidate paths can be sorted separately from the above two dimensions first, and then fused and sorted to select the optimal path with the lowest cost. The fusion method can be a traditional fusion processing algorithm such as weighted summation.

[0094] Regarding the release of KV cache nodes in the tree structure, due to limited storage space and the large number of context documents in the database, it is impossible to store all caches in the KV cache tree, and a corresponding cache release strategy needs to be designed. You can first define the initial frequency Freq when a node is inserted. init = 1. Then, a frequency function is defined to evaluate the change in the access frequency of a node in the tree, expressed as:

[0095] Δ=tanh(T now -T last )

[0096] Among them, T last and T nowIndicates the time when the node frequency was last calculated and the current time. As the time interval increases, the value of Δ gradually increases and tends to 1, indicating that the access frequency of the node gradually decreases with time. Each time the node is accessed, its frequency increases by 1 again, and the frequency of the node at this time can be expressed as:

[0097] Freq new =1+Freq old -Δ

[0098] When the cache space reaches the predefined upper limit, the cache release mechanism is triggered, and the frequency values ​​of all leaf nodes in the KV cache tree at that moment are calculated (since the cache in the tree can only be accessed from top to bottom, the access frequency of the upper nodes must be greater than or equal to the lower nodes), and the leaf nodes with the smallest frequency are released cyclically until the cache space meets the space requirements for the newly added cache.

[0099] Figure 5 This is a schematic diagram of the implementation structure of the large model retrieval processing solution disclosed in the embodiment of this application. Figure 5 As shown in , for the user's input query, it is first vectorized, and the K document blocks with the highest semantic similarity are retrieved from the vector database. Afterwards, unlike the traditional RAG, a KV cache reuse planning module is first used to calculate the inter-entity semantic correlation between these context documents, and the documents with higher correlation are placed in the same collection. Based on the basic finding that the mutual attention between highly correlated documents is relatively intensive, the document blocks in the same collection are used as the basic units to reuse and calculate the KV cache. Note that when the number of document blocks in a collection is 1, it is equivalent to calculating an independent KV cache for a single document block. The dependency relationship between all KV caches is as follows: Figure 5 As shown in the lower right corner, the KV cache manager maintains and updates the KV cache and guides the reuse of the KV cache. The KV cache multi-level dynamic management mechanism places the KV cache in GPU memory, host memory, or extended memory according to the strategy, and effectively manages the reading, release, and storage of the KV cache during the reasoning process of the large model, thereby accelerating RAG reasoning.

[0100] In summary, the technical solution disclosed in the present application constructs a context document KV cache dependency tree, and realizes reasonable KV cache reuse and ensures the quality of RAG generation by hierarchically managing independent and dependent KV caches; at the same time, a lightweight cross-document cross-attention estimation mechanism is designed to optimize the KV cache strategy, significantly reduce the number of KV caches, and increase the probability of cache reuse; in addition, a RAG reasoning-sensitive KV cache hierarchical storage mechanism is designed to ensure the read and write performance of the KV cache while making use of larger capacity and more cost-effective storage media as much as possible.

[0101] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0102] The method is described in detail in the embodiments disclosed in the above-mentioned application. The method of the application can be implemented by various forms of devices. Therefore, the application also discloses a device, and a specific embodiment is given below for detailed description.

[0103] Figure 6 This is a schematic diagram of the structure of a large model retrieval processing device disclosed in an embodiment of the present application. Figure 6 As shown, the large model retrieval processing device 60 may include:

[0104] The query acquisition module 601 is used to obtain query data.

[0105] The document block determination module 602 is used to determine K document blocks that have a preset similarity with the query data, where K is a positive integer.

[0106] The set division module 603 is used to divide the K document blocks into at least one set, and the correlation between the document blocks included in each set meets the set conditions.

[0107] The key value acquisition module 604 is used to obtain the key value data of the at least one set through at least a key value storage structure, wherein the key value storage structure stores the key value data in a tree structure, and the connection relationship between the nodes in the tree structure represents the dependency relationship between different key value data. The key value data stored in the key value storage structure is cached using different types of cache modes.

[0108] The large model retrieval processing device described in this embodiment divides K document blocks that meet the similarity requirements with the query data into at least one set, and determines the key-value data of each set with the set as the basic unit. In the implementation, the key-value cache in the key-value storage structure can be reused to speed up the overall reasoning speed of the large model; the key-value data stored in the key-value storage structure are cached using different types of cache modes, and the key-value data corresponding to the document blocks with high relevance are represented by the connection relationship between the nodes in the tree structure. Due to the flexibility of the node connection in the tree structure, the key-value data with different relevance can be stored in different cache modes. This dynamic caching method can balance the cache resource consumption and improve the reuse rate of key-value data in the later stage under the premise of better maintaining the document context association.

[0109] In one implementation, the set partitioning module may be specifically used to: determine the inter-entity semantic correlation between the K document blocks; and partition the document blocks whose inter-entity semantic correlation exceeds a first threshold into a set.

[0110] In one implementation, the key-value acquisition module includes: a coverage determination module, used to determine the coverage of the key-value data stored in the key-value storage structure for the collection, the coverage representing the proportion of document blocks belonging to the collection that have key-value data in the key-value storage structure among all document blocks in the collection; a key-value determination module, used to obtain the key-value data of the collection based on the coverage.

[0111] In one implementation, the key value determination module can be specifically used for: if the coverage represents that the key value cache corresponding to all document block combinations in the current collection exists in the key value storage structure, the key value cache corresponding to all document block combinations is reused as the key value data of the current collection; if the coverage represents that the key value cache corresponding to some document block combinations in the current collection exists in the key value storage structure, the key value cache corresponding to some document block combinations is reused in combination with the inference of document blocks not covered by the key value storage structure to determine the key value data of the current collection; if the coverage represents that the key value cache corresponding to any document block combination in the current collection does not exist in the key value storage structure, the key value data of the document block combination in the current collection is obtained by inference.

[0112] In one implementation, the device may further include: a key-value saving module, configured to save the key-value data into the key-value storage structure if the obtained key-value data is key-value data that does not exist in the key-value storage structure.

[0113] In one implementation, the key-value saving module can be used for at least one of the following: if the key-value data is data obtained by reasoning separately for a document block, a target node with the root node as the parent node is added to the tree structure, and the key-value data is saved in the target node; if the key-value data is data obtained by processing a document block combined with a target key-value cache already existing in the tree structure, a target node is added to the end of the path corresponding to the target key-value cache in the tree structure, and the key-value data is saved in the target node.

[0114] In one implementation, the device may also include: an optimal path determination module, which is used to determine the optimal path among each of the key-value cache paths if there are multiple key-value cache paths corresponding to some document combinations in the current collection in the key-value storage structure, and the optimal path is determined based on the reuse cost of the key-value cache path; wherein the reuse cost is related to the migration time of the key-value cache data in the key-value cache path to the graphics processor and the computing power required to determine the key-value data of the document block not covered by the key-value storage structure.

[0115] In one implementation, in the key-value storage structure, the key-value data of multiple document blocks in the same collection are stored in a full-sequence cache mode, and there is a dependency relationship between the key-value data stored in the full-sequence cache mode; the key-value data of document blocks in different collections are stored in an independent cache mode.

[0116] In one implementation, the document block determination module can be specifically used to: vectorize the query data to obtain vectorized data; retrieve K document blocks whose semantic similarity with the vectorized data meets preset conditions from a preset vector database.

[0117] The specific implementation of the above-mentioned large model retrieval processing device and the various modules it contains can be found in the corresponding part of the method embodiment, which will not be repeated here.

[0118] Any one of the large model retrieval processing devices described in the above embodiments includes a processor and a memory. The query acquisition module, document block determination module, set partitioning module, key value acquisition module, optimal path determination module, key value storage module, etc. in the above embodiments are all stored in the memory as program modules, and the processor executes the above program modules stored in the memory to realize corresponding functions.

[0119] The processor includes a kernel, which retrieves the corresponding program module from the memory. One or more kernels can be set, and the processing of the access data can be realized by adjusting the kernel parameters.

[0120] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0121] In an exemplary embodiment, a computer-readable storage medium is also provided, which can be directly loaded into the internal memory of a computer and contains software code. After being loaded and executed by a computer, the computer program can implement the steps shown in any embodiment of the above-mentioned large model retrieval processing method.

[0122] In an exemplary embodiment, a computer program product is also provided, which can be directly loaded into the internal memory of a computer, and contains software code. After being loaded and executed by a computer, the computer program can implement the steps shown in any embodiment of the large model retrieval processing method described above.

[0123] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0124] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0125] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0126] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large model retrieval processing method, the method comprising: Get query data; Determine K document blocks that have a preset similarity with the query data, where K is a positive integer; Dividing the K document blocks into at least one set, wherein the correlation between the document blocks contained in each set meets a set condition; The key-value data of at least one set is obtained at least through a key-value storage structure, the key-value storage structure stores the key-value data in a tree structure, the connection relationship between nodes in the tree structure represents the dependency relationship between different key-value data, and the key-value data stored in the key-value storage structure is cached using different types of cache modes.

2. According to the large model retrieval processing method of claim 1, the step of dividing the K document blocks into at least one set comprises: Determining the inter-entity semantic relevance between the K document blocks; The document blocks whose semantic relevance between the entities exceeds a first threshold are divided into a collection.

3. The large model retrieval processing method according to claim 1, wherein obtaining the key value data of the at least one set through at least a key value storage structure comprises: Determine coverage of the set by the key-value data stored in the key-value storage structure, wherein the coverage represents a proportion of document blocks belonging to the set that have key-value data in the key-value storage structure among all document blocks in the set; The key-value data of the collection is obtained based on the coverage.

4. The model retrieval processing method according to claim 3, wherein obtaining the key value data of the set based on the coverage comprises: If the coverage represents that there is a key value cache corresponding to all document block combinations in the current collection in the key value storage structure, reuse the key value cache corresponding to all document block combinations as the key value data of the current collection; If the coverage represents that there is a key value cache corresponding to a partial combination of document blocks in the current collection in the key value storage structure, the key value cache corresponding to the partial combination of document blocks is reused in combination with the reasoning of document blocks not covered by the key value storage structure to determine the key value data of the current collection; If the coverage represents that there is no key value cache corresponding to any document block combination in the current collection in the key value storage structure, the key value data of the document block combination in the current collection is obtained by inference.

5. The large model retrieval processing method according to claim 3 further comprises: If the key-value data obtained is key-value data that does not exist in the key-value storage structure, the key-value data is saved in the key-value storage structure.

6. The large model retrieval processing method according to claim 5, wherein the step of saving the key-value data to the key-value storage structure comprises at least one of the following: If the key-value data is data obtained by reasoning for a document block alone, a target node with the root node as the parent node is added to the tree structure, and the key-value data is saved in the target node; If the key-value data is data obtained by processing a document block in combination with a target key-value cache already existing in the tree structure, a target node is added to the end of the path corresponding to the target key-value cache in the tree structure, and the key-value data is saved in the target node.

7. The large model retrieval processing method according to claim 4 further comprises: If there are multiple key-value cache paths corresponding to some document combinations in the current collection in the key-value storage structure, determine the optimal path among the key-value cache paths, where the optimal path is determined based on the reuse cost of the key-value cache paths; The reuse cost is related to the migration time of the key-value cache data in the key-value cache path to the graphics processor and the computing power required to determine the key-value data of the document block not covered by the key-value storage structure.

8. According to the large model retrieval processing method described in claim 1, in the key-value storage structure, the key-value data of multiple document blocks in the same collection are stored in a full-sequence cache mode, and there is a dependency relationship between the key-value data stored in the full-sequence cache mode; the key-value data of document blocks in different collections are stored in an independent cache mode.

9. The large model retrieval processing method according to claim 1, wherein the step of determining K document blocks having a preset similarity with the query data comprises: Performing vectorization processing on the query data to obtain vectorized data; K document blocks whose semantic similarity with the vectorized data meets a preset condition are retrieved from a preset vector database.

10. A large model retrieval processing device, the device comprising: A query acquisition module is used to obtain query data; A document block determination module, used to determine K document blocks having a preset similarity with the query data, where K is a positive integer; A set partitioning module, used for partitioning the K document blocks into at least one set, wherein the correlation between the document blocks contained in each set meets a set condition; A key value acquisition module is used to obtain the key value data of at least one set through at least a key value storage structure, wherein the key value storage structure stores the key value data in a tree structure, and the connection relationship between nodes in the tree structure represents the dependency relationship between different key value data. The key value data stored in the key value storage structure is cached using different types of cache modes.

Images