Cache cleaning method and device based on large model reasoning

Abstract

The invention provides a cache cleaning method and device based on large model reasoning. The method comprises the following steps: monitoring the cache capacity of a hash table cache pool; when the current cache capacity of the hash table cache pool is greater than or equal to a preset capacity threshold value, obtaining access information corresponding to one or more cache entries, and screening a first to-be-cleaned cache entry through a preset cache cleaning algorithm according to the access information; and / or obtaining importance information corresponding to one or more cache entries, and screening a second to-be-cleaned cache entry meeting an importance cleaning rule according to the importance information; and according to the first to-be-cleaned cache entry and / or the second to-be-cleaned cache entry, cleaning the cache in the hash table cache pool, and updating the cache capacity of the hash table cache pool. According to the method and the device, the cleaned cache items are identified and determined according to the access information and / or the context importance information of the cache items, so that the cache redundancy and the accumulation of invalid data are reduced, and the cache management efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a cache cleaning method and device based on large model reasoning. Background Art

[0002] The efficiency of large-scale model inference determines the model's responsiveness and user experience quality in tasks such as question-answering, interactive conversations, and content generation. However, because these models commonly use a self-attention mechanism, each new token must be associated with previous tokens. As the model parameter size and context sequence length continue to grow, the computational effort increases linearly or even quadratically, and inference latency also increases.

[0003] To alleviate inference latency, existing technologies use key-value caching as an inference optimization method. This mechanism stores the key and value vectors obtained by the model from encoding previously generated tokens, allowing this information to be reused when generating new tokens. In traditional autoregressive generation, the model recalculates the query, key, and value vectors for each new token in the input sequence and uses an attention mechanism to match the new token with previous tokens. With the introduction of key-value caching, the model no longer needs to recalculate the key and value vectors for the existing sequence when generating the next token. Instead, it simply calculates the query vector for the current token, then performs an attention match on this query vector against the cached key vectors. The output is then weighted summed with the cached key vectors based on the matching results. This reduces the repeated processing of the existing sequence, reducing the time complexity of each inference step and improving response speed.

[0004] However, most key-value cache implementations use a linearly growing memory structure to sequentially store all historical key and value data, performing attention matching during each inference computation. As cache content accumulates, graphics memory usage increases, increasing memory pressure. Furthermore, excessive cache data volume reduces read and management efficiency. Therefore, a cache cleanup method for large-model inference is urgently needed. Summary of the Invention

[0005] The present application provides a cache cleaning method and device based on large model reasoning to solve the technical problem of low efficiency of key-value cache management in the prior art.

[0006] In a first aspect, the present application provides a cache cleaning method based on large model reasoning, comprising:

[0007] Based on the established hash table cache pool, monitoring the cache capacity of the hash table cache pool; wherein the hash table cache pool is used to store cache entries corresponding to known word units during the large model inference phase;

[0008] When the current cache capacity of the hash table cache pool is greater than or equal to a preset capacity threshold, obtaining access information corresponding to one or more cache entries in the hash table cache pool, and screening and obtaining at least one first cache entry to be cleared using a preset cache clearing algorithm based on the one or more access information; and / or

[0009] When a current cache capacity of the hash table cache pool is greater than or equal to a preset capacity threshold, obtaining importance information corresponding to one or more cache entries in the hash table cache pool, and screening and obtaining at least one second cache entry to be cleaned that meets an importance cleanup rule based on the one or more importance information;

[0010] According to at least one of the first cache entries to be cleared and / or at least one of the second cache entries to be cleared, the cache in the hash table cache pool is cleared, and the cache capacity of the hash table cache pool is updated.

[0011] Optionally, in the above method, obtaining access information corresponding to one or more cache entries in the hash table cache pool, and screening and obtaining at least one first cache entry to be cleared using a preset cache clearing algorithm based on the one or more access information, includes:

[0012] Traversing all hash buckets in the hash table cache pool, and obtaining access information of each cache entry based on all cache entries in each hash bucket;

[0013] Filtering at least one first cache entry to be cleared using a preset least-used algorithm according to access information of all the cache entries;

[0014] The access information includes at least one of the number of accesses, the access frequency or the last access timestamp.

[0015] Optionally, in the method described above, the access information includes the number of accesses;

[0016] Then, based on the access information of all the cache entries, at least one of the first cache entries to be cleared is screened using a preset least-used algorithm, including:

[0017] Obtaining the number of accesses corresponding to each cache entry, and sorting all the cache entries according to the number of accesses;

[0018] One or more candidate cache entries with the lowest number of accesses are screened and obtained, and the one or more candidate cache entries are recorded as first cache entries to be cleared.

[0019] Optionally, in the above method, obtaining importance information corresponding to one or more cache entries in the hash table cache pool, and screening and obtaining at least one second cache entry to be cleaned that meets the importance cleaning rule based on the one or more importance information, includes:

[0020] Traverse all hash buckets in the hash table cache pool and obtain all cache entries in each hash bucket;

[0021] According to all the cache entries, obtaining a cache word element corresponding to each cache entry, and obtaining a context importance index corresponding to each cache word element;

[0022] According to the context importance indexes of all the cached word units, filter and obtain the word units to be cleaned whose context importance indexes are less than a preset importance threshold;

[0023] According to the correspondence between the cache word element and the cache entry, the cache entry corresponding to the to-be-cleaned word element is recorded as a second to-be-cleaned cache entry.

[0024] Optionally, in the above method, obtaining the context importance index of the cached word includes:

[0025] Based on the reasoning of the large model, according to the query vector of the word to be generated and the key vector of the cached word, the attention score of the cached word is obtained through attention calculation, and the attention score is recorded or updated as the context importance indicator of the cached word.

[0026] Optionally, in the method described above, the cache capacity includes the number of cache entries;

[0027] Based on the established hash table cache pool, monitoring the cache capacity of the hash table cache pool includes:

[0028] According to all hash buckets in the hash table cache pool, the number of cache entries stored in each of the hash buckets is obtained in sequence;

[0029] Obtaining the total number of current cache entries in the hash table cache pool based on the number of cache entries stored in all the hash buckets;

[0030] According to the total number of current cache entries in the hash table cache pool, it is determined whether the total number of current cache entries is greater than or equal to a preset cache entry threshold, so as to monitor the cache capacity of the hash table cache pool.

[0031] Optionally, the method as described above further includes:

[0032] Based on the reasoning of the large model, the query word is obtained, and the query hash value corresponding to the query word is calculated;

[0033] According to the established hash table cache pool, when a hash bucket corresponding to the query hash value is queried, one or more cache entries in the hash bucket are obtained;

[0034] According to one or more cache entries, obtaining cached word elements corresponding to the cached entries, and determining whether there is a cached word element that is consistent with the word element to be queried;

[0035] If there is a cached word element that is consistent with the word element to be queried, output the cache entry corresponding to the cached word element and update the access information of the cache entry; or

[0036] If there is no cached word element that is consistent with the word element to be queried, a new cache entry is created according to the word element to be queried, and the new cache entry is stored in the hash bucket.

[0037] Optionally, the method as described above further includes:

[0038] Based on the hash table cache pool, if no hash bucket corresponding to the hash value is found, creating and storing a cache entry corresponding to the to-be-queried word in the hash table cache pool according to the to-be-queried word;

[0039] The cache entry includes at least one of a key vector, a value vector, access information, or importance information.

[0040] Optionally, in the method described above, the access information includes the number of accesses and the last access timestamp;

[0041] Then, based on the access information of all the cache entries, at least one of the first cache entries to be cleared is screened using a preset least-used algorithm, further comprising:

[0042] Obtaining the number of accesses corresponding to each cache entry, sorting all cache entries according to the number of accesses, and screening candidate entries with the lowest number of accesses;

[0043] If the candidate cache entry with the lowest number of accesses is one, then the candidate cache entry is recorded as the first cache entry to be cleared; or

[0044] If there are multiple candidate cache entries with the lowest number of accesses obtained by screening, extracting the last access timestamps of the multiple candidate cache entries respectively;

[0045] A current timestamp is obtained, and a time difference between the last access timestamp and the current timestamp is calculated, and a candidate cache entry with a maximum time difference is selected as a first cache entry to be cleared.

[0046] In a second aspect, the present application provides a cache cleaning device, comprising:

[0047] A cache monitoring module is used to monitor the cache capacity of the hash table cache pool based on the established hash table cache pool; wherein the hash table cache pool is used to store cache entries corresponding to known word units in the large model inference stage;

[0048] a first screening module, configured to obtain access information corresponding to one or more cache entries in the hash table cache pool when the current cache capacity of the hash table cache pool is greater than or equal to a preset capacity threshold, and screen and obtain at least one first cache entry to be cleared based on the one or more access information using a preset cache clearing algorithm; and / or

[0049] a second screening module, configured to obtain, when a current cache capacity of the hash table cache pool is greater than or equal to a preset capacity threshold, importance information corresponding to one or more cache entries in the hash table cache pool, and screen and obtain at least one second cache entry to be cleaned that meets an importance cleaning rule based on the one or more importance information;

[0050] A cache cleaning module is configured to clean the cache in the hash table cache pool according to at least one of the first cache entries to be cleaned and / or at least one of the second cache entries to be cleaned, and to update the cache capacity of the hash table cache pool.

[0051] In a third aspect, the present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the above-mentioned cleaning method when executing the computer program.

[0052] In a fourth aspect, the present application also provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program implements the above-mentioned cleaning method when executed by a processor.

[0053] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the above-mentioned cleaning method when executed by a processor.

[0054] The cache cleaning method and device based on large model reasoning provided in this application, by establishing a hash table cache pool and monitoring its cache capacity in real time, can timely capture the usage status of the cache pool, ensuring that cache management decisions are accurate and real-time. When the cache capacity reaches or exceeds the preset threshold, the cache entries to be cleared are identified and determined based on the access information and / or context importance information of the cache entries. The screening method based on access information ensures that cache content that has not been used for a long time or has a low usage frequency can be cleared first, thereby effectively improving the resource utilization of the cache space; and the screening mechanism based on context importance can accurately judge and remove low-value cache data that has little impact on the current model reasoning, ensuring that high-value cache content is always in a fast access state. Through the above cache cleaning operations, the structural layout and memory allocation of cache entries can be optimized, cache redundancy and the accumulation of invalid data can be reduced, and the stability and reasoning response efficiency in a continuous high-load environment can be improved. Overall, by dynamically updating the cache capacity, the real-time and accurate usage status of the cache space is guaranteed, thereby improving the efficiency of cache management while further enhancing the reliability of the model reasoning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0056] Figure 1 A cache cleaning method based on large model reasoning provided in an embodiment of the present application;

[0057] Figure 2 A flowchart of a method for screening and obtaining at least one first cache entry to be cleared provided in an embodiment of the present application;

[0058] Figure 3 A flowchart of a method for screening and obtaining at least one second cache entry to be cleared provided in an embodiment of the present application;

[0059] Figure 4 A flowchart of a word element search method provided in an embodiment of the present application;

[0060] Figure 5 A schematic diagram of the structure of a cache cleaning device provided in an embodiment of the present application;

[0061] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0062] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0063] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0064] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and corresponding operation entrances must be provided for users to choose to authorize or refuse.

[0065] In existing technologies, as model parameter size and context length continue to increase, key-value caches used during large model inference often accumulate historical key and value vectors in a linear fashion to reduce repeated computation of historical information during inference. While this linearly increasing cache approach reduces the amount of repetitive computation during inference, it also leads to a rapid increase in video memory usage, a gradual decrease in cache management efficiency, and a lack of effective cache eviction and management strategies, which in turn limits the ability to handle high-concurrency, long-sequence, and complex-context tasks.

[0066] The inventive concept of the present application is to provide a cache cleaning method for the problem of insufficient cache capacity management in the above-mentioned prior art. In the large model inference stage, a hash table cache pool is first established and its cache capacity is monitored in real time. When the cache capacity reaches or exceeds a preset threshold, the cache entries with low frequency of access and / or low semantic contribution are identified and screened as targets to be cleaned, and then these target cache entries are cleaned, and the cache capacity of the hash table cache pool is updated in real time. Through this mechanism, cache space management can be achieved, cache entry cleaning efficiency can be improved, cache structure optimization can be dynamically maintained, thereby reducing video memory occupancy and cache redundancy, and improving resource utilization and response speed of the model inference process.

[0067] Based on the above technical solutions, this application can be widely used in multiple fields and scenarios involving large-scale model reasoning, including but not limited to question-answering, online customer service systems, intelligent assistants, multi-round dialogue interactions, automatic document summarization, content generation, intelligent search and recommendation tasks, and is particularly suitable for application scenarios that require processing long sequence inputs, large amounts of historical context accumulation, and high concurrent requests. Through the cache management and cleanup mechanism proposed in this application, large-scale model reasoning in these scenarios can effectively control cache capacity, reduce resource waste, and enhance adaptability and stability to complex and dynamic tasks.

[0068] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0069] Figure 1 The cache cleaning method based on large model reasoning provided in the embodiment of this application. Figure 1 Shown, including:

[0070] S11, based on the established hash table cache pool, monitoring the cache capacity of the hash table cache pool; wherein the hash table cache pool is used to store cache entries corresponding to known word units in the large model inference stage.

[0071] In this embodiment, to ensure efficient management and dynamic scheduling of cache resources, a hash table cache pool is first established to store cache entries corresponding to known tokens generated during the inference process. As a storage structure with efficient data organization and fast access, the hash table cache pool allocates cache entries associated with each token to corresponding hash buckets through a hash mapping approach, forming a distributed storage layout that helps improve retrieval speed and access efficiency. For example, cache entries typically contain a key vector, a value vector, and other state information associated with the token. After the hash table cache pool is established and in use, its cache capacity is continuously and dynamically monitored. For example, cache capacity typically refers to the number of cache entries actually stored in the cache pool or the amount of storage resources occupied. By monitoring cache capacity in real time, changes in cache usage can be detected in a timely manner to determine the current cache pool load. The monitoring process can be performed by counting the number of cache entries in each hash bucket, recording the overall cache pool memory consumption, or making judgments based on preset capacity indicators. By establishing a hash table cache pool and monitoring its cache capacity in real time, this embodiment manages cache resource usage during the large model inference phase, providing a foundation for subsequent cache optimization control. This embodiment can ensure that reasoning can maintain the compactness of the cache structure and the stability of the overall performance under high-load tasks such as long sequence processing and multi-round dialogues.

[0072] S12, when the current cache capacity of the hash table cache pool is greater than or equal to a preset capacity threshold, obtain access information corresponding to one or more cache entries in the hash table cache pool, and based on the one or more access information, filter and obtain at least one first cache entry to be cleaned through a preset cache cleaning algorithm.

[0073] In this embodiment, when it is detected that the current cache capacity of the hash table cache pool reaches or exceeds a preset capacity threshold, cache cleanup is initiated to ensure that cache resources are released in a timely manner. The preset capacity threshold can be pre-set based on load conditions, video memory resource size, or inference tasks, and serves as the criteria for triggering cache maintenance operations. After the cleanup mechanism is triggered, access information corresponding to one or more cache entries is first obtained from the hash table cache pool. Access information generally reflects the usage activity of each cache entry and aims to record the access behavior characteristics of the cache entries during the inference process. Access information can be collected by updating the cache entry status in real time, with statistics synchronously generated with each cache hit or write operation, ensuring that accurate decisions are made based on the latest data during cleanup. Subsequently, based on the obtained access information, cache entries in the cache pool are screened according to a preset cache cleanup algorithm. The cache cleanup algorithm is used to identify cache data that should be prioritized for elimination under the current system operating conditions. The screening process can comprehensively consider various access information indicators to efficiently determine the cleanup priority. Through the screening operation, at least one first cache entry to be cleaned is obtained from the cache pool, i.e., cache entries that are determined to have low usage value and are suitable for release. This embodiment effectively implements dynamic cache resource recovery and management through a cleanup strategy triggered based on cache capacity and a real-time access information collection and screening mechanism, ensuring that the cache pool capacity can be continuously in a good state, thereby maintaining an efficient and stable operating environment during the inference process and improving the overall inference response speed and resource utilization.

[0074] S13, when the current cache capacity of the hash table cache pool is greater than or equal to the preset capacity threshold, obtain importance information corresponding to one or more cache entries in the hash table cache pool, and screen and obtain at least one second cache entry to be cleaned that meets the importance cleaning rule based on the one or more importance information.

[0075] In this embodiment, when it is detected that the cache capacity reaches or exceeds a preset capacity threshold, in addition to performing cache filtering based on access information, importance information corresponding to one or more cache entries can be obtained to assist or optimize cache cleanup. Importance information reflects the contribution of a cache entry to the current reasoning context and serves as a basis for determining whether a cache entry should be retained. Specifically, importance information associated with each cache entry in the hash table cache pool is first extracted. Importance information can be generated based on contextual attention allocation during large-scale model reasoning, internal model scoring mechanisms, or other comprehensive indicators. It typically indicates the importance of the current word in generating new words or maintaining contextual coherence. By recording and updating importance information, the semantic importance of cache entries at different stages of reasoning can be reflected in real time. After collecting importance information, cache entries are filtered using preset importance cleaning rules. The importance cleaning rules use the importance information of cache entries to identify cache entries with low contribution to the current reasoning process and are therefore prioritized for elimination. This filtering operation ensures that at least one cache entry that meets the cleaning criteria is selected as the target for subsequent cache cleaning actions. This embodiment introduces a filtering mechanism based on importance information, enabling cache cleanup based not only on access frequency but also on contextual semantic importance, enabling more accurate identification of redundant cache entries. This importance-driven cache management strategy further enhances cache structure optimization and improves semantic coherence and efficiency for large-scale model reasoning when processing long sequences, multi-turn conversations, and complex input tasks.

[0076] S14: Clear the cache in the hash table cache pool according to the at least one first cache entry to be cleared and / or the at least one second cache entry to be cleared, and update the cache capacity of the hash table cache pool.

[0077] In this embodiment, after the first cache entry to be cleared and / or the second cache entry to be cleared in the hash table cache pool are screened, the cache clearing and capacity update phase begins. First, based on the identified at least one first cache entry to be cleared and / or at least one second cache entry to be cleared, the corresponding cache clearing operation is performed. The cache clearing operation typically involves locating the target cache entry from the hash table cache pool and removing the corresponding data structure. During the clearing process, it is necessary to ensure that the chain structure or other organizational relationships within the hash bucket are properly adjusted to maintain the integrity and access efficiency of the hash table data structure. After the cache entries are cleared, the cache capacity of the hash table cache pool is updated in real time. The update operation includes recounting the number of valid cache entries in the current hash table or directly decrementing the cache capacity record based on the number of cleared entries. By timely and accurate synchronization of cache capacity information, dynamic awareness of cache resource usage can be maintained, providing a reliable basis for cache management in subsequent reasoning processes, triggering the next clearing strategy, or making cache expansion decisions. In summary, this embodiment effectively reorganizes storage resources within the cache pool by clearing cache entries to be cleared and dynamically updating cache capacity. This process not only reduces the usage of invalid data and releases computing resources, but also further improves the overall management efficiency and operational stability of the hash table cache pool during the large model inference phase.

[0078] In one embodiment, Figure 2 The flowchart of the method for screening and obtaining at least one first cache entry to be cleared provided in the embodiment of the present application is a specific description of an implementation method of the above step S12. Figure 2 Shown, including:

[0079] S21, traverse all hash buckets in the hash table cache pool, and obtain access information of each cache entry based on all cache entries in each hash bucket;

[0080] S22, screening at least one first cache entry to be cleared using a preset least-used algorithm based on access information of all cache entries; wherein the access information includes at least one of the number of accesses, the frequency of access, or the timestamp of the last access.

[0081] In this embodiment, to filter cache entries within a hash table cache pool, a full traversal operation is first performed, sequentially accessing all hash buckets stored in the hash table cache pool. A hash bucket is the basic organizational unit in a hash table. Each hash bucket typically consists of a linked list or array structure and is used to store cache entries with the same hash value or hash conflicts. Each cache entry is associated with a specific token and contains a key vector and value vector used for inference calculations, as well as metadata recording access history. When traversing each hash bucket, access information corresponding to each cache entry stored in the bucket is extracted and retrieved. Access information is an indicator of the frequency and timeliness of cache entry usage and typically includes, but is not limited to, the number of accesses (i.e., the cumulative number of cache entry accesses), access frequency (the average number of accesses per unit time), and the last access timestamp (the time when the cache entry was last used). This information can be used to quantify the activity level of each cache entry. After collecting access information for all cache entries, a preset least-used algorithm is applied based on this access information to perform filtering. The Least Frequently Used (LFU) algorithm is a cache elimination decision based on access frequency, which aims to give priority to removing cache entries with the lowest access frequency. Each cache entry is sorted according to the number of accesses or access frequency, and priority is given to cache entries that have been used the least times or with the lowest frequency in the historical reasoning process. In summary, this embodiment can identify at least one first cache entry to be cleared by traversing all hash buckets in the hash table cache pool and applying the least used algorithm based on access information. It effectively reduces the long-term occupation of cache resources by irrelevant data, and ensures that cache space is continuously reserved for commonly used context information. This embodiment improves the accuracy and dynamic adjustment capabilities of cache management through fine-grained data collection and screening.

[0082] In one embodiment, the access information includes the number of accesses;

[0083] Then, based on the access information of all cache entries, at least one first cache entry to be cleared is screened using a preset least-used algorithm, including:

[0084] S2101, obtaining the access count corresponding to each cache entry, and sorting all cache entries according to the access count;

[0085] S2102: Filter and obtain one or more candidate cache entries with the lowest number of accesses, and record the one or more candidate cache entries as first cache entries to be cleared.

[0086] In this embodiment, when selecting the first cache entry to be cleared, the corresponding access count is first obtained for each cache entry. The access count refers to the cumulative number of times the cache entry has been accessed and used since its creation, and is used to quantify the activity of the cache entry. Whenever each cache entry is accessed (e.g., for attention calculation or matching), its access count field is synchronously updated to ensure that it reflects the latest usage. After extracting the access counts for all cache entries, all cache entries are sorted based on these access counts. The sorting process is typically performed in ascending order, with cache entries with lower access counts placed first and cache entries with higher access counts placed last. To ensure efficient sorting, quick sort, heap sort, or a lightweight sorting algorithm optimized for cache management can be used. After the sorting is complete, one or more candidate cache entries with the lowest access counts are selected based on the sorting results. Cache entries with the lowest access counts typically represent the least frequently used and least active data during historical reasoning. Prioritizing the elimination of these cache entries maximizes the freeing up of invalid cache space. Finally, one or more candidate cache entries with the lowest access counts are recorded as the first cache entry to be cleaned and marked for subsequent cache cleanup. This embodiment, through access count-based screening and recording, identifies low-activity cache entries, laying the foundation for dynamic optimization and resource utilization of the cache pool.

[0087] In another specific embodiment, the access information includes the number of accesses and the last access timestamp;

[0088] Then, based on the access information of all cache entries, at least one first cache entry to be cleared is screened using a preset least-used algorithm, further comprising:

[0089] S2111, obtaining the access count corresponding to each cache entry, sorting all cache entries according to the access count, and selecting candidate entries with the lowest access count;

[0090] S2112: If the candidate cache entry with the lowest number of accesses is one, record the candidate cache entry as the first cache entry to be cleared.

[0091] S2113, if there are multiple candidate cache entries with the lowest access counts, extract the last access timestamps of the multiple candidate cache entries respectively;

[0092] S2114 , obtaining the current timestamp, calculating the time difference between the last access timestamp and the current timestamp, and selecting the candidate cache entry with the largest time difference as the first cache entry to be cleared.

[0093] In this embodiment, the access count for each cache entry in the hash table cache pool is first extracted. The access count, a fundamental indicator of cache entry usage frequency, records the cumulative number of hits and references since the cache entry was written. After extracting the access counts for all cache entries, the entries are sorted in ascending order based on the access count, with entries with fewer access counts placed first and entries with more access counts placed last. Based on the sorting results, candidate cache entries with the lowest access counts are screened. These entries are the least frequently used during the inference process and are prioritized for elimination. If only one candidate cache entry with the lowest access count is found after screening, it is directly recorded as the first cache entry to be cleared and becomes the target for subsequent clearing operations. If the screening results reveal multiple candidate cache entries with the same lowest access count, these candidate entries are further prioritized. Specifically, the last access timestamp of each candidate cache entry is extracted. The last access timestamp is the time stamp recorded when the cache entry was last used, reflecting the cache entry's most recent active state. At the same time, the current timestamp is obtained in real time, representing the system time when the current cleanup operation is in progress. The time difference between the last access timestamp and the current timestamp of each candidate cache entry is calculated to analyze the time span that the cache entry has not been accessed. Among all time differences, the candidate cache entry with the largest time difference is selected as the final first cache entry to be cleaned. The cache entry with the largest time difference indicates that it has not been used for a long time during the reasoning process, that is, it has low activity and is suitable for priority elimination. Through this embodiment, the last access time is introduced as an auxiliary judgment basis in the candidate entries with the same number of accesses, thereby realizing fine-grained optimization of cache cleanup decisions, ensuring that cache space is always reserved for high-value and high-activity data, thereby improving the overall resource utilization and reasoning response efficiency of the hash table cache pool.

[0094] In one embodiment, Figure 3 The flowchart of the method for filtering and obtaining at least one second cache entry to be cleared provided in the embodiment of the present application is a specific description of an implementation method of the above step S13. Figure 3 Shown, including:

[0095] S31, traverse all hash buckets in the hash table cache pool and obtain all cache entries in each hash bucket;

[0096] S32, based on all cache entries, obtaining a cached word element corresponding to each cached entry, and obtaining a context importance index corresponding to each cached word element;

[0097] S33, based on the context importance indexes of all cached word units, filter and obtain word units to be cleaned whose context importance indexes are less than a preset importance threshold;

[0098] S34 , based on the correspondence between the cache word and the cache entry, record the cache entry corresponding to the word to be cleaned as the second cache entry to be cleaned.

[0099] In this embodiment, a hash table cache pool is first traversed. The hash table cache pool comprises multiple hash buckets, each of which stores a set of cache entries mapped to hash values. Cache entries are typically organized within a hash bucket in a chained structure or array format to support efficient conflict resolution and fast retrieval. During the traversal process, each hash bucket is accessed sequentially and all existing cache entries within it are retrieved, ensuring that all stored data items in the cache pool are covered. After collecting all cache entries, the associated cached word element information is further extracted from each cache entry. A cached word element refers to a historical word element that has been generated and encoded as a key-value vector pair during the inference process, and each cached word element corresponds to a set of cached entries. Simultaneously, for each cached word element, the contextual importance index of its current record is extracted. This contextual importance index is dynamically calculated using an attention mechanism and reflects the importance or contribution of the cached word element to the generation of the to-be-generated word element in the current inference task. After extracting the contextual importance indexes for all cached word elements, the contextual importance indexes of all cached word elements are filtered based on a pre-set importance threshold. During the screening process, all cached words whose contextual importance index is less than a preset threshold are identified. These words are considered to have a low semantic contribution in the current reasoning stage and are prioritized for elimination. Finally, based on the one-to-one correspondence between cached words and cache entries, all cache entries associated with the words to be cleared are marked as the second cache entries to be cleared, serving as the target of subsequent clearing operations. Through this embodiment, not only is cache management based on access frequency, but a dynamic contextual importance analysis mechanism is also introduced, further improving the accuracy of cache clearing and overall resource utilization efficiency.

[0100] In a specific embodiment, obtaining the context importance index of the cached word includes:

[0101] S321, based on the reasoning of the large model, according to the query vector of the word to be generated and the key vector of the cached word, through attention calculation, obtain the attention score of the cached word, and record or update the attention score as the context importance indicator of the cached word.

[0102] In this embodiment, when new input is received and the next token is about to be generated, the corresponding query vector is first calculated based on the input state of the current token to be generated. The query vector is a low-dimensional feature representation that captures the current generation requirements and contextual dependencies, reflecting the position and focus of the token to be generated in the semantic space. Subsequently, the query vector is matched with the key vectors corresponding to each cached token stored in the hash table cache pool. The key vector is a fixed representation generated for each historical token during the encoding phase and is primarily used to match the query vector in the attention mechanism. The query vector and key vector are matched using a dot product operation and normalized by a scaling factor to form the original attention score matrix. This process essentially measures the degree of relevance or attention intensity of the current token to be generated with respect to each historical cached token. After the matching calculation is completed, the score results are normalized to obtain the attention score of each cached token. A higher attention score indicates that the cached token contributes more to the generation of the token to be generated in the current reasoning process and has a stronger semantic relevance. The attention score corresponding to each cached word is used as its contextual importance indicator and recorded or updated in the corresponding cache entry in the hash table cache pool, ensuring that the importance of each cached word reflects the latest inference context state in real time. Through this embodiment, the importance of each cached word is dynamically quantified during the inference process, enabling cache entry importance analysis based on actual inference needs. This provides a basis for subsequent importance-based information screening and cache optimization, effectively improving the response speed and resource utilization efficiency of large-model inference in long sequence processing, multi-round dialogue generation, and complex task reasoning.

[0103] In one embodiment, the cache capacity includes the number of cache entries.

[0104] Based on the established hash table cache pool, the cache capacity of the hash table cache pool is monitored, including:

[0105] S111, according to all hash buckets in the hash table cache pool, sequentially obtain the number of cache entries stored in each hash bucket;

[0106] S112, obtaining the total number of current cache entries in the hash table cache pool based on the number of cache entries stored in all hash buckets;

[0107] S113 , judging whether the total number of current cache entries in the hash table cache pool is greater than or equal to a preset cache entry threshold based on the total number of current cache entries in the hash table cache pool, so as to monitor the cache capacity of the hash table cache pool.

[0108] In this embodiment, each hash bucket is accessed sequentially, and the number of cache entries currently stored in that hash bucket is counted during the access. The count can be performed by directly reading the hash bucket's entry counter, or by traversing the cache entry nodes in the hash bucket one by one and accumulating the data, ensuring that the actual amount of data stored in each hash bucket is obtained. After completing the traversal and counting of all hash buckets, the number of cache entries stored in each hash bucket is accumulated to obtain the current total number of cache entries in the hash table cache pool. The total number of cache entries directly reflects the current storage load of the hash table cache pool and is used to measure cache capacity. Based on the current total number of cache entries, it is further compared with a preset cache entry threshold. The preset cache entry threshold is typically set based on video memory capacity, expected task load, or performance requirements. When it is detected that the current total number of cache entries is greater than or equal to the preset cache entry threshold, it is confirmed that the hash table cache pool has reached or is approaching the resource usage limit. At this time, the subsequent cache cleanup process is triggered to release some low-value cache entries and restore the available capacity of the cache pool. This embodiment monitors the storage status of the hash table cache pool through cache entry statistics and real-time capacity monitoring, ensuring that cache resources can always be efficiently managed under different task loads, thereby improving the overall inference response performance and sustainability of resource usage.

[0109] Figure 4 The flowchart of the word element query method provided in the embodiment of the present application is as follows. Figure 4 Shown, including:

[0110] S41, based on the reasoning of the large model, obtain the query word and calculate the query hash value corresponding to the query word;

[0111] S42, according to the established hash table cache pool, when a hash bucket corresponding to the query hash value is found, obtaining one or more cache entries in the hash bucket;

[0112] S43, obtaining a cached word corresponding to one or more cached entries, and determining whether there is a cached word that is consistent with the query word;

[0113] S44, if there is a cached word-element that is consistent with the query word-element, output the cache entry corresponding to the cached word-element and update the access information of the cache entry;

[0114] S45: If there is no cached word element that is consistent with the word element to be queried, a new cache entry is created according to the word element to be queried, and the new cache entry is stored in the hash bucket.

[0115] In this embodiment, a query term is obtained based on the current inference task. A query term is a term corresponding to the key vector (or value vector) required for input in the current inference phase. After obtaining the query term, a hash calculation is performed on the query term based on a predefined hash function to generate a corresponding query hash value. After obtaining the query hash value, the hash bucket corresponding to the query hash value is located based on the structure of the hash table cache pool. One or more cache entries stored in this hash bucket are further obtained. Each cache entry contains a cache term and its associated key vector, value vector, access information, and importance index. Based on the obtained one or more cache entries, the corresponding cache term is extracted and matched with the current query term. The matching process is typically performed by directly comparing the original content of the term or its encoded representation to accurately identify whether a corresponding record already exists in the cache. If a cache term matching the query term exists in the hash bucket, the cache entry corresponding to the cache term is directly output for subsequent inference use. Simultaneously, the access information of the cache entry is synchronously updated, for example, by incrementing the access count or updating the last access timestamp to reflect the entry's new usage status, ensuring continued accuracy of access information. If there is no cached word element that is consistent with the word element to be queried in the hash bucket, it indicates that the information related to the word element has not been stored in the cache pool. At this time, a new cache entry is generated based on the current word element to be queried, which usually includes calculating and storing the key vector and value vector of the word element, and initializing the access information and importance index. The generated new cache entry is then inserted into the corresponding hash bucket. It can be added at the end of the linked list or inserted according to preset rules to maintain the orderliness of the internal structure of the hash bucket and the retrieval efficiency. Through this embodiment, fast retrieval or dynamic write management of the query word element is achieved, which improves the access speed and cache maintenance capabilities of the hash table cache pool in the large model reasoning process, and ensures that the reasoning process can still maintain efficient and stable execution performance when facing long sequence inputs, multi-round dialogues or high-concurrency tasks.

[0116] Furthermore, based on the above embodiment, the present invention further includes:

[0117] Based on the hash table cache pool, if a hash bucket corresponding to the hash value is not found, a cache entry corresponding to the word to be queried is created and stored in the hash table cache pool according to the word to be queried; wherein the cache entry includes at least one of a key vector, a value vector, access information or importance information.

[0118] In this embodiment, if the query hash value generated based on the query token fails to find a corresponding hash bucket in the hash table cache pool, meaning that no storage unit associated with the hash value exists in the current hash table structure, a new hash bucket and cache entry creation process will be triggered to ensure that the data of the query token can be correctly stored and subsequently utilized. Specifically, a cache entry is first constructed based on the current query token. A cache entry typically contains the following information: a key vector, a value vector, access information, and importance information. The key vector is a low-dimensional representation extracted by the model encoder and is used for attention matching with the query vector during subsequent inference. The value vector is a vector representation of contextual memory or semantic features in the model, supporting result reconstruction based on attention weights. The access information records the cache entry's historical retrieval or usage, such as the number of accesses and the last access timestamp, for reference during subsequent cache cleanup. The importance information reflects the cache entry's semantic contribution to the current contextual reasoning and is typically dynamically updated via an attention score. After the cache entry is constructed, a new hash bucket corresponding to the hash value is created in the hash table cache pool. The hash bucket can be implemented using a linked list, array or other adaptive structure, depending on the conflict handling method of the hash table. Those skilled in the art can make flexible settings according to their needs, and no specific limitations are made here. The newly created cache entry is stored in the newly created hash bucket to ensure the integrity of the internal structure of the hash table and to support subsequent efficient access and management based on hash values. Through this embodiment, it is achieved that when the hash table cache pool structure is dynamically expanded, the situation of hash bucket missing is automatically identified and processed, so that all words to be queried can establish corresponding cache entries in time, improve the integrity and hit efficiency of cached data in the large model reasoning process, and effectively support performance stability in high-load application scenarios such as long text reasoning, multi-round interaction and complex instruction execution.

[0119] Figure 5 This is a schematic diagram of the structure of the cache cleaning device provided in the embodiment of the present application. Figure 5 As shown, the cleaning device 5 includes:

[0120] A cache monitoring module 51 is configured to monitor the cache capacity of the hash table cache pool based on the established hash table cache pool; wherein the hash table cache pool is configured to store cache entries corresponding to known word units during the large model inference phase;

[0121] a first screening module 52 configured to obtain access information corresponding to one or more cache entries in the hash table cache pool when the current cache capacity of the hash table cache pool is greater than or equal to a preset capacity threshold, and to screen and obtain at least one first cache entry to be cleared based on the one or more access information using a preset cache clearing algorithm; and / or

[0122] a second screening module 53 configured to obtain importance information corresponding to one or more cache entries in the hash table cache pool when the current cache capacity of the hash table cache pool is greater than or equal to a preset capacity threshold, and to screen and obtain at least one second cache entry to be cleared that meets the importance clearing rule based on the one or more importance information;

[0123] The cache cleaning module 54 is configured to clean the cache in the hash table cache pool according to at least one first cache entry to be cleaned and / or at least one second cache entry to be cleaned, and update the cache capacity of the hash table cache pool.

[0124] The cleaning device provided in this embodiment can execute the cleaning method of the above embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0125] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 6 As shown, the electronic device 6 includes: at least one processor 61 and a memory 62. The electronic device 6 also includes a communication component 63. The processor 61, the memory 62 and the communication component 63 are connected via a bus 64.

[0126] During the specific implementation process, at least one processor 61 executes the computer-executable instructions stored in the memory 62, so that the at least one processor 61 executes the method executed by the electronic device side as described above.

[0127] The specific implementation process of the processor 61 can be found in the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0128] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0129] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage.

[0130] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0131] The above-mentioned functions implemented by the electronic device and the main control device have introduced the solutions provided by the embodiments of the present invention. It can be understood that in order to implement the above-mentioned functions, the electronic device or the main control device includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of the various examples described in the embodiments disclosed in the embodiments of the present invention, the embodiments of the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiments of the present invention.

[0132] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above-mentioned cleaning method is implemented.

[0133] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The computer-readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0134] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in an electronic device or a main control device.

[0135] The present application also provides a computer program product, which includes: a computer program, which is stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0136] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0137] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all optional embodiments, and the actions and modules involved are not necessarily required by this application.

[0138] It should be further noted that, although the various steps in the flowchart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps may be performed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but may be performed at different times. The execution order of these sub-steps or stages is not necessarily to be performed in sequence, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0139] It should be understood that the above-described device embodiments are merely illustrative, and the device of the present application may also be implemented in other ways. For example, the division of units / modules in the above-described embodiments is merely a logical functional division, and actual implementations may employ other division methods. For example, multiple units, modules, or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0140] In addition, unless otherwise specified, the functional units / modules in the various embodiments of the present application may be integrated into a single unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The aforementioned integrated units / modules may be implemented in the form of hardware or software program modules.

[0141] If an integrated unit / module is implemented in hardware, the hardware may be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor may be any appropriate hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC. Unless otherwise specified, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0142] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, magnetic disk, or optical disk, etc., various media that can store program code.

[0143] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a particular embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined in any way. To keep the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0145] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A cache cleaning method based on large model reasoning, characterized in that: include: Based on the established hash table cache pool, monitoring the cache capacity of the hash table cache pool; wherein the hash table cache pool is used to store cache entries corresponding to known word units during the large model inference phase; When the current cache capacity of the hash table cache pool is greater than or equal to a preset capacity threshold, obtaining access information corresponding to one or more cache entries in the hash table cache pool, and screening and obtaining at least one first cache entry to be cleared using a preset cache clearing algorithm based on the one or more access information; and / or When a current cache capacity of the hash table cache pool is greater than or equal to a preset capacity threshold, obtaining importance information corresponding to one or more cache entries in the hash table cache pool, and screening and obtaining at least one second cache entry to be cleaned that meets an importance cleanup rule based on the one or more importance information; According to at least one of the first cache entries to be cleared and / or at least one of the second cache entries to be cleared, the cache in the hash table cache pool is cleared, and the cache capacity of the hash table cache pool is updated.

2. The cleaning method according to claim 1, wherein: The obtaining of access information corresponding to one or more cache entries in the hash table cache pool, and screening and obtaining at least one first cache entry to be cleared using a preset cache clearing algorithm based on the one or more access information, includes: Traversing all hash buckets in the hash table cache pool, and obtaining access information of each cache entry based on all cache entries in each hash bucket; Filtering at least one first cache entry to be cleared using a preset least-used algorithm according to access information of all the cache entries; The access information includes at least one of the number of accesses, the access frequency or the last access timestamp.

3. The cleaning method according to claim 2, characterized in that: The access information includes the number of visits; Then, based on the access information of all the cache entries, at least one of the first cache entries to be cleared is screened using a preset least-used algorithm, including: Obtaining the number of accesses corresponding to each cache entry, and sorting all the cache entries according to the number of accesses; One or more candidate cache entries with the lowest number of accesses are screened and obtained, and the one or more candidate cache entries are recorded as first cache entries to be cleared.

4. The cleaning method according to claim 1, wherein: The obtaining of importance information corresponding to one or more cache entries in the hash table cache pool, and screening and obtaining at least one second cache entry to be cleaned that meets an importance cleaning rule based on the one or more importance information, includes: Traverse all hash buckets in the hash table cache pool and obtain all cache entries in each hash bucket; According to all the cache entries, obtaining a cache word element corresponding to each cache entry, and obtaining a context importance index corresponding to each cache word element; According to the context importance indexes of all the cached word units, filter and obtain the word units to be cleaned whose context importance indexes are less than a preset importance threshold; According to the correspondence between the cache word element and the cache entry, the cache entry corresponding to the to-be-cleaned word element is recorded as a second to-be-cleaned cache entry.

5. The cleaning method according to claim 4, characterized in that: The acquisition of the context importance index of the cached word element includes: Based on the reasoning of the large model, according to the query vector of the word to be generated and the key vector of the cached word, the attention score of the cached word is obtained through attention calculation, and the attention score is recorded or updated as the context importance indicator of the cached word.

6. The cleaning method according to claim 1, wherein: The cache capacity includes the number of cache entries; Based on the established hash table cache pool, monitoring the cache capacity of the hash table cache pool includes: According to all hash buckets in the hash table cache pool, the number of cache entries stored in each of the hash buckets is obtained in sequence; Obtaining the total number of current cache entries in the hash table cache pool based on the number of cache entries stored in all the hash buckets; According to the total number of current cache entries in the hash table cache pool, it is determined whether the total number of current cache entries is greater than or equal to a preset cache entry threshold, so as to monitor the cache capacity of the hash table cache pool.

7. The cleaning method according to any one of claims 1 to 6, characterized in that: The cleaning method further comprises: Based on the reasoning of the large model, the query word is obtained, and the query hash value corresponding to the query word is calculated; According to the established hash table cache pool, when a hash bucket corresponding to the query hash value is queried, one or more cache entries in the hash bucket are obtained; According to one or more cache entries, obtaining cached word elements corresponding to the cached entries, and determining whether there is a cached word element that is consistent with the word element to be queried; If there is a cached word element that is consistent with the word element to be queried, output the cache entry corresponding to the cached word element and update the access information of the cache entry; or If there is no cached word element that is consistent with the word element to be queried, a new cache entry is created according to the word element to be queried, and the new cache entry is stored in the hash bucket.

8. The cleaning method according to claim 7, characterized in that: Also includes: Based on the hash table cache pool, if no hash bucket corresponding to the hash value is found, creating and storing a cache entry corresponding to the to-be-queried word in the hash table cache pool according to the to-be-queried word; The cache entry includes at least one of a key vector, a value vector, access information, or importance information.

9. The cleaning method according to claim 2, wherein: The access information includes the number of accesses and the last access timestamp; Then, based on the access information of all the cache entries, at least one of the first cache entries to be cleared is screened using a preset least-used algorithm, further comprising: Obtaining the number of accesses corresponding to each cache entry, sorting all cache entries according to the number of accesses, and screening candidate entries with the lowest number of accesses; If the candidate cache entry with the lowest number of accesses is one, then the candidate cache entry is recorded as the first cache entry to be cleared; or If there are multiple candidate cache entries with the lowest number of accesses obtained by screening, extracting the last access timestamps of the multiple candidate cache entries respectively; A current timestamp is obtained, and a time difference between the last access timestamp and the current timestamp is calculated, and a candidate cache entry with a maximum time difference is selected as a first cache entry to be cleared.

10. A cache cleaning device, characterized in that: include: A cache monitoring module is used to monitor the cache capacity of the hash table cache pool based on the established hash table cache pool; wherein the hash table cache pool is used to store cache entries corresponding to known word units in the large model inference stage; a first screening module, configured to obtain access information corresponding to one or more cache entries in the hash table cache pool when the current cache capacity of the hash table cache pool is greater than or equal to a preset capacity threshold, and screen and obtain at least one first cache entry to be cleared based on the one or more access information using a preset cache clearing algorithm; and / or a second screening module, configured to obtain, when a current cache capacity of the hash table cache pool is greater than or equal to a preset capacity threshold, importance information corresponding to one or more cache entries in the hash table cache pool, and screen and obtain at least one second cache entry to be cleaned that meets an importance cleaning rule based on the one or more importance information; A cache cleaning module is configured to clean the cache in the hash table cache pool according to at least one of the first cache entries to be cleaned and / or at least one of the second cache entries to be cleaned, and to update the cache capacity of the hash table cache pool.

Images