Configurable caching method and system for artificial intelligence reasoning process

Abstract

The invention provides a configurable caching method and system for an artificial intelligence reasoning process. According to the method, a cache interface layer is introduced to be integrated with an artificial intelligence framework, flexible cache strategies including a cache key generation method, a storage mode, cache capacity, an elimination strategy and the like are defined by utilizing a user configuration module, and a cache key generator is used for generating cache keys according to the strategies. The cache storage manager is responsible for managing storage, retrieval and updating of cache data and supports various storage rear ends. In the reasoning process, the cache querying and updating module is responsible for querying and updating the cache. The system can also comprise a performance monitor and a data security module which are respectively used for monitoring cache performance indexes and ensuring data security. The corresponding system comprises a cache interface layer, a user configuration module, a cache key generator, a cache storage manager and a cache query and update module. According to the method, the artificial intelligence reasoning efficiency can be improved, the resource utilization is optimized, a flexible and configurable cache solution is provided, and the method is easy to integrate with a mainstream artificial intelligence framework and adapts to different task requirements.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a cache optimization method and system for artificial intelligence reasoning processes. Background Art

[0002] The rapid development and widespread application of artificial intelligence technology have put forward higher and higher requirements on the efficiency and performance of reasoning computing. In practical applications, artificial intelligence reasoning often needs to process a large amount of input data and repeatedly perform similar or identical computing tasks, resulting in a waste of computing resources and prolonged response time. Therefore, how to optimize the artificial intelligence reasoning process and improve computing efficiency and resource utilization has become an urgent problem to be solved in the industry.

[0003] At present, mainstream AI frameworks such as TensorFlow and PyTorch have provided basic reasoning acceleration mechanisms, such as optimization of computational graphs and fusion of operators. However, these mechanisms mainly optimize the model structure and algorithm level, and lack effective solutions for the problem of repeated calculation of input data. Although some common cache solutions, such as Redis, can be used to store and reuse calculation results, they are independent of the AI ​​reasoning framework and require additional queries and data transmission, making it difficult to fully utilize the advantages of caching.

[0004] In order to further accelerate AI reasoning, some researchers have proposed combining the cache mechanism with the reasoning framework. This type of method intercepts the data flow of the reasoning framework and automatically caches repeated calculation results, which reduces redundant calculations to a certain extent. However, most existing solutions use fixed cache strategies, such as LRU, which are difficult to adapt to the data characteristics and access patterns of different tasks. In addition, these solutions are highly coupled with the reasoning framework, and have poor portability and scalability, which is not conducive to widespread application.

[0005] As AI application scenarios become increasingly diverse, higher requirements are placed on inference caching. On the one hand, different application tasks have different requirements for caching, and need to support flexible and configurable caching strategies; on the other hand, the caching mechanism should be seamlessly integrated with the inference framework to minimize additional overhead and latency. In addition, as data scale and computational complexity increase, the cache system also needs to have features such as high scalability, high availability, security and reliability. Existing inference caching solutions are still insufficient in these aspects, and a new solution is urgently needed.

[0006] The purpose of the present invention is to provide an efficient, flexible and easy-to-integrate artificial intelligence reasoning caching method and system in response to the deficiencies of the prior art. The method introduces a configurable caching strategy to support users to customize caching behaviors according to specific tasks, thereby improving the applicability of the cache; at the same time, the method adopts an architecture that is tightly integrated with the reasoning framework, minimizing the overhead of cache queries and improving reasoning performance; in addition, the present invention adopts a high-throughput, high-concurrency cache query and update mechanism, and supports cache monitoring, analysis and dynamic optimization to cope with large-scale and complex artificial intelligence application scenarios. The caching method and system of the present invention represent an innovative reasoning acceleration idea and are expected to become an important part of future artificial intelligence frameworks.

[0007] After outlining the efficiency challenges faced by artificial intelligence reasoning, this background technology section focuses on analyzing the limitations of existing reasoning cache solutions, including fixed cache strategies, high coupling with the reasoning framework, poor scalability, and other issues. On this basis, the purpose and features of the present invention are introduced, highlighting the advantages of configurable cache strategies, seamless integration with the reasoning framework, high performance and high availability, etc., paving the way for the subsequent description of the invention content. This section avoids the description of specific technical details and deletes some unnecessary background content to make the background technology section more focused and refined. Summary of the invention

[0008] The purpose of the present invention is to provide a configurable caching method and system for artificial intelligence reasoning process to solve the problems of insufficient flexibility, high coupling with reasoning framework, limited performance and scalability of existing reasoning caching solutions.

[0009] Another object of the present invention is to provide a cache mechanism that is deeply integrated with the reasoning framework, minimize the overhead of cache query and update, and significantly improve reasoning performance.

[0010] To achieve the above-mentioned purpose, the present invention provides a configurable cache method for an artificial intelligence reasoning process, characterized in that it includes the following steps: first, a cache interface layer is provided for integration with an artificial intelligence reasoning framework, and input data features extracted by the reasoning framework are received in the reasoning request processing flow; then, a cache strategy is defined through a user configuration module, and the cache strategy includes a cache key generation method, a storage method, a cache capacity and an elimination strategy, an update frequency, and custom rules for specific tasks; then, a cache key generator is used to generate a cache key according to data features extracted by the reasoning framework or defined by the user; then, a cache storage manager is used to manage the storage, retrieval, and update of cache data; finally, in the artificial intelligence reasoning process, the cache is queried through a cache query and update module, and if a hit is found, the cache result is directly returned without repeating the reasoning calculation; if a miss is found, the reasoning calculation is performed and the result is written to the cache. Wherein, the cache query and update process is part of the reasoning request processing flow, and no additional cache query service is required.

[0011] In some embodiments, the cache interface layer is integrated with the artificial intelligence reasoning framework through a pluggable mechanism, supporting a variety of mainstream reasoning frameworks, such as TensorFlow, PyTorch, Caffe, etc. This plug-in design allows the cache method of the present invention to be easily adapted and extended to different reasoning environments.

[0012] In other embodiments, the user configuration module supports defining cache strategies through a domain specific language (DSL). The DSL supports programming language features such as conditional judgment, loop statements, function definitions, etc., so that users can flexibly describe complex cache strategies, such as dynamically adjusting cache behaviors according to data attributes, access frequency, time windows, etc.

[0013] In some other embodiments, the cache key generator supports user-defined data feature extraction methods. Users can select or customize appropriate feature representations, such as word vectors for text, convolutional features for images, etc., according to the characteristics of specific tasks, to improve the recall and precision of the cache.

[0014] In some further embodiments, the cache query and update module adopts a multi-threaded parallel processing mechanism, supports batch query and update, and significantly improves cache throughput and concurrency performance.

[0015] The present invention also provides a configurable cache system for an artificial intelligence reasoning process, comprising: a cache interface layer for integrating with a reasoning framework and extracting input data features; a user configuration module for defining cache strategies; a cache key generator for generating cache keys based on data features; a cache storage manager for managing cached data; and a cache query and update module for querying and updating the cache during the reasoning process and returning cached results first.

[0016] In some embodiments, the system also includes a cache monitoring and analysis module for collecting cache status data, analyzing performance bottlenecks, and automatically tuning cache strategies; and a cache data security module for encrypting, controlling permissions, and auditing access to cache data to ensure data security and privacy.

[0017] The configurable cache method and system provided by the present invention have the following beneficial effects: (1) By introducing a flexible cache configuration mechanism, users can customize cache strategies according to specific tasks, improving the applicability and effectiveness of cache. (2) Adopting an architecture that is deeply integrated with the reasoning framework, cache query is embedded in the reasoning process, minimizing additional overhead and significantly improving reasoning performance; (3) The high-throughput and high-concurrency cache processing mechanism can efficiently support large-scale complex reasoning tasks; (4) Providing cache monitoring, analysis, optimization, and security assurance functions, further improving the availability and reliability of the cache system.

[0018] In summary, the caching method and system of the present invention are suitable for various artificial intelligence application scenarios, especially those with high requirements for reasoning efficiency and performance, and have broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 It is an overall architecture diagram of the configurable cache system of the present invention.

[0021] Figure 2 It is a timing diagram of the integration of the cache interface layer and the reasoning framework.

[0022] Figure 3 This is a DSL parsing diagram of the user configuration module.

[0023] Figure 4 This is the workflow diagram of the cache key generator.

[0024] Figure 5 This is a diagram of the multi-level storage architecture of the cache storage manager.

[0025] Figure 6 It is a flowchart of cache query and update.

[0026] Figure 7 This is the functional architecture diagram of the cache monitoring and statistics module.

[0027] Figure 8 It is a schematic diagram of cache data security protection measures. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] In the description of the present invention, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0030] Example 1: This example provides an artificial intelligence reasoning optimization method based on configurable cache. This method significantly improves the efficiency and performance of reasoning by seamlessly integrating a fine-grained cache mechanism into the reasoning framework. Unlike existing reasoning cache solutions, this method adopts a progressive cache query strategy that is isomorphic to the reasoning process, avoiding the additional overhead of external cache services and achieving minimized reasoning latency.

[0031] First, this method introduces a highly flexible cache interface layer that can be seamlessly integrated with mainstream artificial intelligence reasoning frameworks in a pluggable manner. By registering callback functions or overloading key methods, the cache interface layer can intercept key events and data in the reasoning request processing flow and extract input feature vectors. This loosely coupled architecture allows the cache system to be developed and evolved independently of the specific reasoning framework, providing great flexibility and scalability.

[0032] Next, the input feature vector extracted by the inference framework is passed to the cache key generator to generate a cache key that uniquely identifies the input. Different from the traditional cache key generation method based on word frequency or hashCode, this method innovatively proposes a key generation mechanism based on semantic hashing. This mechanism makes full use of the high-order semantic information generated by the inference framework in the feature extraction stage, through a semantic modeling function The original feature vector Mapped to a low-dimensional semantic vector : Then, the cache key generator is based on the semantic vector , using the Locality Sensitive Hashing (LSH) algorithm to generate a compact binary hash code As cache key: in, is a random hyperplane matrix, is the bias vector. Through LSH, the high-dimensional semantic space is mapped to a binary hash space, and semantically similar inputs are mapped to the same or adjacent hash buckets to achieve fast semantic similarity matching. At the same time, the parameters of LSH can be optimized using unsupervised learning algorithms to minimize the loss of original semantics while ensuring uniform hash distribution.

[0033] Based on the semantic hash key, the cache query and update module dynamically inserts cache query operations at key nodes in the execution process of the reasoning graph. Once it is found that the semantic feature to be calculated matches a historical cache item (complete match or approximate match), the module will directly return the cached result and skip the subsequent calculation nodes, thereby achieving the purpose of reusing calculations and accelerating reasoning. When the cache misses, the reasoning graph is executed according to the original calculation logic, and the new calculation results are backfilled into the cache system to achieve dynamic update of the cache. By embedding cache query into the execution process of the reasoning graph, this method realizes concurrent operation of cache and reasoning, avoiding the serial overhead caused by cache.

[0034] It is worth mentioning that in order to maximize the utilization of the cache, this method adopts a progressive cache granularity control mechanism. Traditional reasoning cache methods usually only cache the final reasoning results, ignoring the reusability of intermediate features, resulting in a large amount of repeated calculations. This method moves the cache granularity down to the semantic feature level, sets cache checkpoints at each stage of reasoning calculation (such as feature extraction, semantic modeling, interactive reasoning, etc.), reuses existing semantic features as much as possible, and thus greatly reduces the amount of repeated calculations.

[0035] In addition, this method also gives users fine-grained control over cache behavior by introducing a declarative cache configuration module. Users can use standardized configuration files such as YAML and JSON or structured programming APIs to dynamically specify the core parameters of the cache system, such as cache capacity, data storage media, elimination strategy, update frequency, etc., as well as custom cache rules for specific tasks. This flexible configuration mechanism allows the cache system to adapt to different application scenarios and performance requirements, greatly improving the applicability and scalability of the cache.

[0036] In terms of cache storage management, this method adopts a hierarchical hybrid storage architecture, which comprehensively utilizes the advantages of multiple storage media such as memory, SSD, and HDD, and performs adaptive data placement based on the access characteristics and popularity of the data. For example, for small-scale data with high real-time requirements and frequent access, memory storage is preferred to obtain the lowest read and write latency; while cold data can be sunk to SSD or HDD to save memory space. The cache storage manager will automatically schedule data migration between different media based on the cache elimination strategy configured by the user, and merge and batch the cache read and write operations to minimize the cache maintenance overhead.

[0037] Finally, this method also integrates a complete cache monitoring and analysis system to help users understand and optimize the performance bottlenecks of the inference process. By deploying lightweight probes on the critical path of cache processing, the system collects multi-dimensional performance indicators of the cache in real time, including cache hit rate, query latency distribution, space utilization of different media, etc., and summarizes them into intuitive visual reports. Users can deeply analyze cache behavior through an interactive web interface, quickly locate performance anomalies, and dynamically adjust cache strategies based on the intelligent optimization suggestions provided by the system. In addition, the cache monitoring system also supports user-defined alarm rules, and promptly notifies relevant personnel through SMS, email and other channels when cache performance is seriously degraded, thereby ensuring the high availability of the inference service.

[0038] In summary, this embodiment describes in detail a progressive reasoning cache optimization method based on semantic hashing. This method cleverly uses the high-order semantic information generated by the reasoning framework to build a set of fine-grained, multi-stage, configurable cache mechanisms, which minimizes cache overhead while maximizing the reuse of calculation results, thereby significantly improving the overall performance of reasoning. This method provides a new idea for artificial intelligence reasoning optimization and is expected to become a standard component of future reasoning frameworks.

[0039] Embodiment 2: Based on Embodiment 1, this embodiment further optimizes the generation method of cache keys and the organization form of cache data to improve cache query efficiency and storage utilization.

[0040] Specifically, this method introduces a cache key generation mechanism based on semantic hashing and a cache indexing technology based on Bloom Filter, which significantly reduces cache space occupancy and query time.

[0041] In terms of cache key generation, Example 1 adopts a method based on local sensitive hashing (LSH) to convert high-dimensional semantic vectors into binary hash codes, thereby achieving efficient similarity query. However, the LSH method still has the following shortcomings: first, binary encoding will lead to information loss, affecting the accuracy of semantic similarity calculation; second, the number of bits of the hash code usually needs to be longer (such as 64 bits or 128 bits), occupying a large storage space; finally, the binary hash code is not conducive to indexing and it is difficult to support complex query conditions and range queries.

[0042] In order to overcome the above limitations, this embodiment proposes a cache key representation method based on semantic hashing. The core idea of ​​this method is to quantize the LSH hash code into a short string as part of the cache key, and then combine it with the original numerical features to form a complete cache key. In this way, the information required for semantic similarity judgment is retained and the key length is greatly compressed.

[0043] Specifically, this method first performs segmented quantization on the LSH hash code. The $b$-bit binary hash code is divided into The length is Each sub-segment is mapped to a predefined character table to generate a length of String The design of the character table takes into account the compactness and readability of the encoding. Common schemes include Base16, Base32, Base64, etc.

[0044] Then, from the original high-dimensional semantic vector Select The most representative numerical features , normalize them, and quantize the results to In the interval, a The feature vector of : Finally, the quantized LSH substring and the eigenvector Spliced ​​together to form a complete cache key : in, Indicates a string concatenation operation. This method of generating semantic hash keys keeps the key length within a certain range while preserving semantic similarity. The order of magnitude ( The length of the feature vector is much lower than the original high-dimensional vector length. In addition, the numerical feature vector can also support efficient range query and composite condition filtering, providing more flexibility for intelligent cache retrieval.

[0045] In terms of cache query optimization, this embodiment also introduces a cache indexing technology based on Bloom Filter. Bloom Filter is an efficient indexing structure based on probability, which can quickly determine whether an element belongs to a certain set. Its core idea is to use a length of The bit array To represent a size of Collection and use Different hash functions Maps each element in the collection to of At different positions, set the corresponding bit to 1. When querying an element Is it part of the collection? When calculate hash function, and then check The corresponding Are all the ones digits 1? If any one digit is 0, then it is certain that Not ; On the contrary, if all bits are 1, then There is a high probability that However, there is also a certain probability of false positives.

[0046] This embodiment uses the characteristics of Bloom Filter to build an efficient cache index for the cache system. Specifically, the cache key $K$ is used as the input of Bloom Filter, and The hash function calculates a Bit array of bits When querying a cache key When it exists , First Perform the same hash calculation , Get the corresponding position Arrays , and then with Perform a bitwise AND operation: If the result If any bit in is 0, then we can determine If it is not in the cache, an empty result is returned directly; otherwise, and Exactly the same, then There is a high probability of hitting the cache, and further comparison of the complete cache key is required. By introducing the Bloom Filter index, most non-existent cache queries can be filtered out in a very short time, and only a small number of queries need to match the complete key, which significantly improves the average performance of cache queries.

[0047] It should be noted that, since Bloom Filter has a certain false positive probability, before the cache result is returned, the complete cache key needs to be compared to ensure the accuracy of the result. In order to further optimize the performance of this step, this embodiment introduces the key value separation and key value locality principle in the storage structure of the cache. That is, the complete cache key and cache value are stored separately, and the keys are clustered and sorted according to the locality of the key (such as access frequency, last access time, etc.), so that the associated keys are as close as possible in the storage space. In this way, when performing complete key matching, the CPU cache and pre-fetch mechanism can be used to reduce the delay of memory access as much as possible. At the same time, in the storage of cache values, this method also uses a variety of optimization technologies such as column storage, data compression, and redundant coding, which ensures high reliability and fast access to data while improving storage density.

[0048] In addition, for the life cycle management of cached data, this embodiment provides a set of fine-grained cache expiration and elimination mechanisms. Users can carry expiration time information in the cache key, and flexibly set global and local cache elimination strategies, such as FIFO, LRU, LFU, etc. through the cache configuration module. The cache system will regularly scan and compare the expiration time of the cache key, and automatically delete expired cache items, while releasing some low-priority cache space according to the elimination strategy. It is worth mentioning that the elimination process does not block normal cache read and write operations, but is executed asynchronously with the main process, and ensures the consistency and availability of cached data through mechanisms such as write-time copy.

[0049] Finally, in terms of cache persistence and recovery, this embodiment adopts an incremental Checkpoint and log-based recovery mechanism. The system will regularly perform full and incremental Checkpoints on the cache data, and persist the cache snapshots to the disk or distributed file system. At the same time, the cache write operations will be recorded in real time in the form of logs, and will be merged and compressed regularly to ensure the continuity and replayability of the log files. When a failure occurs or the system restarts, the cache system can restore the cache state from the most recent Checkpoint and replay subsequent write logs to minimize data loss. Combining Checkpoint and log recovery, this method significantly reduces the time and space costs of persistence while ensuring the reliability of cached data.

[0050] In summary, based on Example 1, this embodiment further optimizes the generation method of cache keys and the performance of cache queries, and provides key technologies such as efficient storage, expiration elimination, and persistent recovery of cache data. These design ideas and optimization mechanisms together constitute a highly intelligent reasoning cache system that can give full play to the effectiveness of cache resources and comprehensively improve the execution efficiency of reasoning tasks. The cache optimization technology described in this embodiment is not only applicable to stand-alone environments, but can also be seamlessly extended to distributed clusters, and has broad application prospects in the elastic deployment and on-demand expansion of artificial intelligence applications.

[0051] Example 3: Based on the above example, this example further provides a cache performance monitoring and analysis method for artificial intelligence reasoning process. This method realizes real-time monitoring and fine-grained analysis of the cache system operation status by introducing a special monitoring module and analysis tool, and provides data support and decision-making basis for dynamic optimization of cache strategy.

[0052] Traditional cache systems usually lack effective monitoring methods, making it difficult to gain insight into the internal behavior and bottlenecks of the cache. The monitoring module of the present invention uses a probe-based non-intrusive monitoring technology, which inserts lightweight monitoring probes into key components such as the cache interface layer, cache key generator, and cache storage manager to achieve performance indicator collection and aggregation of each link of the cache.

[0053] Specifically, the monitoring module defines a set of standardized cache performance indicators, covering multiple dimensions such as cache hit rate, cache access latency, cache space utilization, etc. Each monitoring probe is responsible for collecting the raw data required for specific indicators, such as cache key query results, read and write operation timestamps, storage space usage, etc., and sends it to the monitoring module for aggregation and analysis.

[0054] In order to minimize the impact of monitoring overhead on cache performance, the monitoring module adopts a sampling-based data collection strategy. Assume that the total number of cache events is , the user-configured sampling rate is , the monitoring module performs independent sampling on each cache event, and the sampling probability for: Through the sampling rate $\rho$, users can flexibly control the granularity and overhead of monitoring data. When , it represents the collection of all cache events; when When it is less than 1, The probability of collecting cache events is . The size of the cache event subset obtained by sampling is The mathematical expectation of is: Through the sampling strategy, the monitoring module obtains statistically representative cache event samples with low overhead, significantly reducing the monitoring cost while ensuring the accuracy of the analysis.

[0055] In terms of data analysis, the present invention provides a set of flexible multi-dimensional analysis tools. Based on online analytical processing (OLAP) technology, the tool supports users to slice, dice, drill, and perform other operations on cache performance data from multiple dimensions such as time, space, and objects, so as to deeply analyze cache behavior patterns and performance bottlenecks.

[0056] Taking cache hit rate analysis as an example, suppose the total number of cache queries within a time window is , the number of hits is , then the cache hit rate of this time window is for: By tracking the trend of cache hit rate changes in different time windows, users can promptly discover cache timeliness issues. Similarly, by calculating the cache hit rates of different objects, the locality of cached data can be compared and optimized.

[0057] In cache access latency analysis, suppose the cache query response time series of an object is , then the average cache access latency of the object for: By comparing the average cache access latency of different objects, you can locate and optimize cache items with slow query speeds.

[0058] The cache analysis tool of the present invention also introduces a machine learning model to mine and learn historical cache performance data and automatically generate a cache behavior profile. Taking cluster analysis as an example, let the cache performance indicator vector set be ,in , the goal of cluster analysis is to transform these The vector is divided into Cluster , so that the similarity of vectors within a cluster is maximized and the similarity of vectors between clusters is minimized. This can be achieved by minimizing the following objective function: in, Representation vector Is it a cluster? , Cluster The above optimization problem can be efficiently solved by clustering algorithms such as K-Means.

[0059] Through cluster analysis, cache behaviors are automatically divided into several typical profiles, which describe the access patterns in different business scenarios. These profiles can be used as a benchmark for anomaly detection, and can be compared with real-time performance data to timely discover and locate cache anomalies. On the other hand, they can also provide a reference for automatic optimization of cache strategies, and select the optimal cache configuration parameters by matching current data features with historical profiles.

[0060] Considering that artificial intelligence applications usually have dynamically changing data distribution and load characteristics, the monitoring and analysis modules of the present invention are tightly integrated with the configuration module to form a closed-loop adaptive optimization framework. The monitoring module continuously collects cached operating data, the analysis module mines the patterns and rules therein, and the configuration module dynamically adjusts the cache strategy based on the analysis results and sends it to each component in real time. Through this feedback optimization mechanism, the cache system can continuously adapt to changes in data and load and automatically evolve the best cache strategy.

[0061] In summary, based on Example 1 and Example 2, this embodiment further proposes a complete cache performance monitoring and analysis method. This method uses non-invasive probe technology to achieve fine-grained data collection, introduces multi-dimensional OLAP tools to support flexible performance analysis, and implements cache behavior modeling and anomaly detection through machine learning. The monitoring and analysis module of the present invention is seamlessly connected with the configuration module to build a closed-loop, adaptive cache optimization framework, which greatly improves the dynamic adaptability of the cache in the artificial intelligence reasoning process and provides efficient and stable cache service guarantees for complex and changeable artificial intelligence application environments.

[0062] Example 4: Based on Example 1, Example 2 and Example 3, this example further provides a distributed cache system for artificial intelligence reasoning process.

[0063] By introducing a distributed cache architecture, the system achieves horizontal expansion and load balancing of cache data, further improving the cache throughput and fault tolerance, and meeting the high concurrency and high availability requirements of large-scale artificial intelligence applications.

[0064] Traditional single-machine cache systems are difficult to cope with the massive concurrent requests during artificial intelligence reasoning, and can easily become the performance bottleneck of the entire system. The present invention adopts a distributed cache architecture, which stores cache data in multiple nodes and achieves load balancing of requests through data partitioning algorithms such as consistent hashing.

[0065] Specifically, the distributed cache system consists of multiple cache nodes, each of which runs an independent cache instance and is responsible for storing and managing a portion of the cache data. When a client initiates a cache request, its hash value is first calculated based on the cache key, and then the request is routed to the cache node responsible for the key through a consistent hashing algorithm.

[0066] Assume that the distributed cache system consists of nodes, and the hash value set of the nodes is . Given a cache key , whose hash value is , the key will be assigned to the hash value closest to In other words, the cache key Will be routed to the node ,in: in, Represents a hash value and The clockwise distance on the hash ring. Through the consistent hashing algorithm, cache keys can be evenly distributed to each node to achieve load balancing. At the same time, when the number of nodes changes (such as adding or removing nodes), only a small part of the cache keys needs to be redistributed, avoiding large-scale data migration.

[0067] In order to further improve the availability and disaster tolerance of distributed cache, the present invention introduces a data redundancy mechanism. Specifically, for each cache key, in addition to assigning it to the master node, it is also copied to Backup nodes. These backup nodes are selected in the clockwise direction of the consistent hash ring to ensure the dispersion of backup. When the primary node fails, the request can be automatically switched to the nearest backup node to ensure the continuity of the cache service.

[0068] make is a sequence of node hash values ​​sorted clockwise, then the cache key The master node index It can be calculated according to the formula in Example 3. On this basis, the cache key No. Backup node index It can be calculated as follows: With the above backup strategy, each cache key has replicas are evenly distributed on the hash ring. This data redundancy solution provides high availability while avoiding hot spots caused by backup concentration.

[0069] In order to ensure the data consistency of the distributed cache, the present invention adopts an optimistic concurrency control strategy based on version vectors. Specifically, each cache key is associated with a version vector, which records the updated version number of the key on each node. When the client reads the cache key, the distributed cache system returns the data value and its version vector; when the client updates the cache key, it needs to provide the expected version vector. After receiving a write request, the distributed cache system compares the version vector in the request with the version vector in the system to detect whether there is a write conflict.

[0070] Suppose the current version vector of cache key $k$ is ,in For Node When the client submits a version vector of When a write request is received, the distributed cache system performs the following operations: (1) Comparison and : like , then accept the write request and After adding 1 to each element, update the cache; like , the write request is rejected and a "version conflict" error is returned; (2) Update the version vector Synchronize to all backup nodes.

[0071] The version vector mechanism effectively prevents data inconsistency caused by concurrent writes. At the same time, this mechanism has no effect on read requests, ensuring low latency for cache reads.

[0072] In summary, this embodiment proposes a highly available, high-performance distributed cache system, which is optimized for the massive concurrent requests of the artificial intelligence reasoning process. The system uses consistent hashing to achieve balanced distribution of data and load, provides high availability and fault tolerance through data redundancy, and introduces an optimistic concurrency control strategy based on version vectors, which effectively ensures the final consistency of cached data. The distributed cache system of the present invention is organically combined with the aforementioned configurable cache method and cache monitoring and analysis method to form a complete set of artificial intelligence reasoning acceleration solutions, providing solid cache support for large-scale, high-concurrency artificial intelligence applications.

[0073] Example 5: This example provides an artificial intelligence reasoning acceleration system based on configurable cache. The system comprehensively applies the various technologies in Examples 1 to 4, and is specially optimized for complex artificial intelligence application scenarios, thereby achieving all-round acceleration of the artificial intelligence reasoning process.

[0074] In actual AI applications, inference requests usually have complex data dependencies and execution order constraints. An inference request may involve multiple subtasks, each of which corresponds to a different AI model. There are dependencies between subtasks, and they need to be executed in a specific topological order. In addition, different inference requests also vary greatly in terms of data characteristics, computing intensity, and real-time requirements. These complexities pose huge challenges to the cache system, and it is difficult to meet the needs of all scenarios with a single cache strategy.

[0075] In order to meet the above challenges, this embodiment introduces a task scheduling model based on a directed acyclic graph (DAG). Specifically, each inference request is abstracted as a DAG, where nodes represent subtasks and edges represent dependencies between subtasks. Each subtask node is associated with an artificial intelligence model and its input and output specifications, and the corresponding model is called during execution to complete the inference calculation. The DAG model describes the computational logic and execution constraints of the inference request in a declarative way, giving the system greater scheduling optimization space.

[0076] On this basis, this embodiment designs a two-stage task scheduling strategy. In the first stage, the system performs static analysis on each DAG to determine the critical path and parallelism. The critical path refers to the path with the longest execution time in the DAG, which determines the shortest completion time of the entire DAG. The parallelism reflects the number of tasks that can be executed concurrently in the DAG. By calculating the critical path length and parallelism of the DAG, the system can estimate its demand intensity for computing resources and cache.

[0077] In the second stage, the system dynamically adjusts the task scheduling priority and cache resource allocation according to the characteristics of the DAG and the current load. For DAGs with long critical paths and low parallelism, the system treats them as "high priority" requests and allocates dedicated computing resources and cache space to them, minimizing the waiting time for tasks in queue; while for DAGs with short critical paths and high parallelism, the system treats them as "low priority" requests, encouraging concurrent execution of tasks and cache sharing, improving overall throughput and resource utilization. In addition, the system will dynamically adjust the cache elimination strategy and allocation ratio based on runtime statistics such as cache hit rate and recycling efficiency to strike a balance between stability and response time.

[0078] When actually executing inference requests, the system uses multi-level cache and data pipeline technology to maximize the parallelism of cache and computing. Specifically, the system sets up a dedicated first-level cache for each subtask to store its input and output data; at the same time, the system also maintains a global second-level cache for sharing common data between different DAGs. When a subtask completes the calculation, its output result is first written to the first-level cache, and then asynchronously transmitted to downstream nodes and the second-level cache. This data pipeline method eliminates the need for subtasks to wait for data to be written back, and can quickly release computing resources, thereby improving the utilization of the CPU and GPU. At the same time, the multi-level cache structure also reduces the delay in data transmission and speeds up data sharing across tasks.

[0079] In terms of fault tolerance and fault recovery, this embodiment comprehensively applies multiple mechanisms to ensure the high availability of the reasoning service. On the one hand, the system uses checkpoint technology to regularly save the intermediate state of the reasoning task. When a node fails, the task execution can be restored from the most recent checkpoint to reduce the overhead of recalculation; on the other hand, the system also introduces data redundancy and backup mechanisms to store multiple copies of key model parameters and intermediate results to prevent data loss or damage; in addition, the system also provides task retry and migration functions. When a node is abnormal, the task can be automatically rescheduled to other nodes to ensure the continuity of the reasoning process. Through these measures, this system achieves end-to-end high reliability and greatly reduces the risk of business interruption caused by failures.

[0080] This embodiment also designs a set of friendly task monitoring and performance analysis tools to help users gain insight into the internal state and potential bottlenecks of the reasoning process. The tool provides a rich visual dashboard that displays key indicators such as the DAG topology, execution progress, cache hit rate, resource utilization, etc. of each reasoning request in real time. Users can easily view and analyze task execution through the Web interface and quickly locate abnormal or inefficient links. In addition, the tool also has built-in some common performance problem detection rules, which can automatically discover and diagnose potential problems such as deadlocks, resource leaks, cache pollution, etc., and provide optimization suggestions, greatly reducing the threshold and difficulty of performance tuning.

[0081] In summary, this embodiment proposes a highly flexible, scalable, and easy-to-use artificial intelligence reasoning acceleration system, which fully draws on and integrates the aforementioned key technologies. The system adopts a DAG-based task scheduling model, which can accurately describe the complex structure and execution constraints of reasoning requests; at the same time, the system comprehensively applies cache optimization, task scheduling, fault-tolerant recovery, pipeline acceleration and other mechanisms, which significantly improves the reasoning performance and system reliability; in addition, the system also provides powerful monitoring and analysis tools, giving users observability and controllability of the reasoning process. The acceleration system constructed in this embodiment is suitable for artificial intelligence applications of various scales and types, especially for dynamic and complex production environments, and can continuously and stably deliver high-quality, low-latency reasoning services. With the continuous development of artificial intelligence technology and the deepening of its application, this system is expected to become a key infrastructure for accelerating the intelligent upgrading of the industry and promoting the large-scale application of artificial intelligence.

[0082] The present invention provides a configurable cache method and system for artificial intelligence reasoning process, which significantly improves reasoning performance and reduces computing resource consumption by introducing flexible cache configuration strategy and efficient cache storage management. The various technical solutions of the present invention can be used alone or combined and integrated according to actual needs, and are suitable for artificial intelligence application scenarios of various types and scales.

[0083] It should be noted that the embodiments of the present invention are not limited to the above-mentioned specific implementation methods. Those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. For example, the cache key generation algorithm may be customized and optimized according to the characteristics of the specific task; cache storage may use other types of storage media and storage systems; cache elimination strategies and update mechanisms may be dynamically adjusted according to access patterns and load characteristics; performance monitoring and analysis tools may integrate more statistical indicators and analysis algorithms, etc. These modifications and variations shall be regarded as the protection scope of the present invention.

Claims

1. A configurable caching method for an artificial intelligence reasoning process, characterized in that: The following steps are involved: (1) Provide a cache interface layer for integration with the AI ​​reasoning framework and receiving input data features extracted by the reasoning framework during the reasoning request processing process; (2) defining a cache strategy through a user configuration module, wherein the cache strategy includes a cache key generation method, a storage method, a cache capacity and elimination strategy, an update frequency, and custom rules for specific tasks; (3) Generate cache keys using a cache key generator based on data features extracted by the inference framework or customized by the user; (4) Use the cache storage manager to manage the storage, retrieval, and update of cache data; (5) During the artificial intelligence reasoning process, the cache is queried through the cache query and update module. If a hit is found, the cache result is directly returned without repeating the reasoning calculation; if a hit is found, the reasoning calculation is performed and the result is written to the cache; the cache query and update process is part of the reasoning request processing flow and does not require an additional third-party cache query service.

2. The method according to claim 1, characterized in that The cache interface layer is integrated with the artificial intelligence reasoning framework through a pluggable mechanism and supports a variety of mainstream artificial intelligence reasoning frameworks.

3. The method according to claim 1, characterized in that The user configuration module supports defining cache strategies through a domain specific language (DSL), and the DSL supports programming language features such as conditional judgment, loop statements, and function definitions.

4. The method according to claim 1, characterized in that: The cache key generator supports user-defined data feature extraction methods, and the data features include but are not limited to text features, image features, voice features, etc.

5. The method according to claim 1, characterized in that The cache query and update module adopts a multi-threaded parallel processing mechanism, supports batch query and update, and improves cache throughput.

6. A configurable cache system for artificial intelligence reasoning processes, characterized in that: include: The cache interface layer is used to integrate with the artificial intelligence reasoning framework and extract input data features during the reasoning request processing process; User configuration module, used to define cache strategies; Cache key generator, used to generate cache keys based on data characteristics; Cache storage manager, used to manage the storage, retrieval and update of cache data; The cache query and update module is used to query and update the cache during the inference process, and give priority to returning cached results.

7. The system according to claim 6, characterized in that The system also includes a cache monitoring and analysis module for collecting cache status data, analyzing cache performance bottlenecks, and automatically tuning cache strategies.

8. The system according to claim 6, characterized in that The system also includes a cache data security module for encrypting, controlling permissions and auditing access to cache data to ensure data security and privacy.

Images