Multi-stage caching method for accelerating AI data processing

Abstract

The invention discloses a multi-stage caching method for accelerating AI (artificial intelligence) data processing, which belongs to the technical field of artificial intelligence data processing, and specifically comprises the following steps: creating a data processing pipeline, inserting a caching operation and marking a position, setting a caching client, a caching server and a shared space, and interacting the caching client with the data processing pipeline. The cache server stores a cache data set, the cache client and the cache server perform data transmission by using a gRPC framework, a data processing assembly line is converted into a graph structure and verified, the graph structure is converted, an execution graph is generated and an operator is optimized, and the data processing assembly line performs data processing according to the execution graph. And the cache client is called to communicate with the cache server, and write-in and read operation of the cache data set is executed, so that the AI data processing speed can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence data processing technology, and in particular to a multi-stage caching method for accelerating AI data processing. Background Art

[0002] In the model training of artificial intelligence and deep learning, AI data processing is an important prerequisite, so how to improve the speed of AI data processing has become a practical need.

[0003] Existing caching technologies such as Quiver and ICACHE use a caching strategy that focuses on the data loading layer, calls the GPU to cache the original data, retains important data, and evicts unimportant data, thereby improving data processing efficiency.

[0004] However, the prior art only considers caching raw data, but does not consider caching preprocessed data. Data preprocessing often takes a long time in the data loading stage. Therefore, the acceleration effect of the prior art on data processing is relatively limited. Moreover, most of the prior art is limited to use on GPUs and lacks compatibility across multiple hardware platforms. Therefore, the present invention provides a multi-stage caching method for accelerating AI data processing to solve the problems existing in the prior art. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-stage caching method for accelerating AI data processing to solve the following technical problems: The existing caching technology in the data processing process does not consider caching the pre-processed data, resulting in limited acceleration effect in data processing. In addition, the existing technology is often limited to use on GPUs and lacks compatibility across hardware platforms.

[0006] The purpose of the present invention can be achieved through the following technical solutions: A multi-stage caching method for accelerating AI data processing includes the following steps: Build a data processing pipeline through the top-level interface, insert cache operations in the data processing pipeline in advance and mark the position; A cache client, a cache server and a shared space are set up, and the data processing pipeline communicates through the cache client and the cache server. The cache server is used to store a cache data set, and the cache data set includes the loaded original data and the pre-processed data. The cache client and the cache server use the gRPC framework and the shared space to exchange information; Convert the constructed data processing pipeline into a graph structure and perform node verification on the graph structure. After verification, divide the cached dataset into mappable dataset and non-mappable dataset according to whether the cached dataset supports full mapping to memory. For different datasets, use corresponding methods to convert the graph structure, generate an execution graph and optimize repeated operators. Start the data processing pipeline and perform data processing according to the generated execution graph. When the data pipeline performs cache writing and cache reading of the cache data set, the cache client is called to send request information to the cache server. The cache server receives the request information and performs corresponding operations. After completing the operation, it returns the corresponding information to the cache client.

[0007] As a further solution of the present invention: when the gRPC framework is used between the cache client and the cache server, variables rq and reply are set for each request sent by the cache client to the cache server, wherein rq is used to record information that needs to be sent to the cache server, and reply is used to obtain response information sent from the cache server.

[0008] As a further solution of the present invention: the specific process of verifying the graph structure is: Call the cache verification function, starting from the starting node of the graph structure, traverse all nodes of the entire graph structure according to the data processing path, and filter out nodes for batch processing, connection, filtering, skipping, retrieval and compression operations respectively. When there is no pre-inserted cache operation in the subsequent data processing path of any node, the verification is judged to be passed, otherwise it is judged to be a verification error, and an error report is output after the traversal is completed to remind the user to modify the insertion position of the cache operation.

[0009] As a further solution of the present invention: after the judgment and verification are passed, the specific process of converting the graph structure is: When the cache data set is a mappable data set, a cache merge node is created in the graph structure and inserted into the map node as the parent node of the map node. The sampler is obtained from the data loading node, and a cache search node is created based on the sampler. The sampler is replaced with the cache search node to search for and return the cached data in the cache merge node. The Build() function of all nodes in the graph structure is called to build the corresponding operator of each node in turn, generate an execution graph, and mark the operator corresponding to the cache merge node as the cache merge operator, and the operator corresponding to the cache search node as the cache search operator. When the cached data set is an unmappable data set, obtain the sampler corresponding to the data loading node belonging to the leaf node in the graph structure, create a cache node based on the sampler, and use the cache node to replace the sampler. Insert the cache node into the map node as the parent node of the map node, call the Build() function of all nodes in the graph structure, build the operator corresponding to each node runtime in turn, generate an execution graph, and mark the operator corresponding to the cache node as a cache operator.

[0010] As a further solution of the present invention: the specific process of performing repeated operator optimization on the graph structure is: Determine the type of cache data set. When the cache data set is a mappable data set, obtain the position of the repeat operator in the operation queue. If the repeat operator is located before the cache search operator and the cache merge operator, set the repeat count of the cache search operator according to the repeat count set in the repeat operator. Otherwise, set the repeat count of the cache search operator according to the repeat count set in the cache merge operator. When the cached data set is an unmappable data set, the data that is repeatedly processed by the repeated operator is marked as a cache object, and the graph structure is traversed from top to bottom. During the traversal, the cache objects are added to the cache node stack one by one. When the repeated optimization mechanism traverses the graph structure from bottom to top, the corresponding cache objects are taken out from the cache node stack one by one, and the cache object repeat count is set to 1.

[0011] As a further solution of the present invention: the specific operation process of the cache merge operator and the cache search operator is: When the data loading operator performs data loading operations, it provides the sampler to the cache search operator to obtain the ID of the data row. After the cache search operator obtains the ID, it starts N child threads in the main thread, where N is a positive integer. After collecting a preset number of IDs, the main thread assigns the ID to the child threads through the internal queue. The child threads send cache query requests to the cache server in batches. If the ID of the cache query request hits the cache server, the data stream corresponding to the ID is sent to the cache merge operator. Otherwise, the cache search operator sends the missed ID to the data loading operator. The data loading operator re-reads the parsed data stream and sends the data stream with placeholders to the cache merge operator. The cache merge operator continuously waits for and receives data streams, merges the data streams corresponding to the hit IDs and stores them in the cache server. For data streams containing placeholders, the placeholders are replaced with actual data. After the replacement is completed, the complete data stream is stored in the cache server.

[0012] As a further solution of the present invention: the specific operation process of the cache operator is: The cache operator starts M sub-threads in the main thread, where M is a positive integer. Each sub-thread remains in a blocked state before successfully storing the data in the cache server and receiving the return information. When the data storage tasks of all sub-threads are completed and the return information is obtained, the cache operator switches to the acquisition state and starts reading the data stored in the cache server through the cache operator.

[0013] As a further solution of the present invention: the specific process of the data processing pipeline performing cache writing and cache reading operations on the cache data set is: When writing to the cache, the data processing pipeline calls the cache client to submit an asynchronous buffer request to the gRPC queue. After receiving the request, the gRPC queue returns a shared memory address. Then the cache client starts to continuously write the cache data set to the shared memory. When the shared memory storage is full, the buffer client submits a cache write request to the gRPC queue. The gRPC thread submits the cache write request to the request queue of the cache server. The working thread of the cache server writes the cache data set from the shared memory to the memory of the cache server in turn according to the request queue. When performing cache reading, the data processing pipeline calls the cache client to submit a data row request to the gRPC queue. After receiving the request, the gRPC queue locates the data row in the memory of the cache server and allocates shared memory for the data row. At the same time, it submits a data extraction request to the request queue of the cache server. The working thread of the cache server writes the cache data set from the memory of the cache server to the shared memory. When the cache data set is successfully read into the shared memory, the gRPC queue returns the shared memory address corresponding to the data to the cache client. The cache client copies the cache data set from the shared memory to the target location in the data processing pipeline, and submits a request to release the shared memory to the gRPC queue. The gRPC thread completes the memory release.

[0014] As a further solution of the present invention: during the cache writing and cache reading operations of the cache data set by the data processing pipeline, memory management is also optimized. The specific optimization process is as follows: When optimizing memory management, based on the NUMA framework, according to the ratio of the memory space of the cache server and the total memory space of the system, a NUMA framework-aware cache memory pool is configured. Each cache memory pool consists of memory blocks. The number of memory blocks is set to the number of CPUs, and memory space is allocated to the memory blocks through specific NUMA nodes.

[0015] Beneficial effects of the present invention: The present invention proposes a caching method that can run through the entire data processing flow, and can flexibly cache at various stages of data processing, not only caching original data, but also caching preprocessed data. And by caching in the preprocessing stage, the present invention can simultaneously reduce I / O operations and data preprocessing operations, improve the efficiency of the data processing stage, and effectively prevent data processing from becoming a bottleneck in deep neural network training. The present invention designs multiple automatic optimization modules for users to use the cached data processing pipeline, including automatic verification and transformation of the data pipeline graph structure, using shared memory to optimize data transmission, simple cache sharing, etc. At the same time, the present invention also provides direct API support for multiple popular data sets in multiple scenarios, has good hardware compatibility, and users do not need to design the interface by themselves and can directly use the designed interface, which improves the ease of use and user experience of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention will be further described below in conjunction with the accompanying drawings.

[0017] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 It is a schematic diagram of the process of cache writing and cache reading during data processing; Figure 3 It is a schematic diagram of the process of converting the graph structure under the mappable data set; Figure 4 It is a flowchart of generating graph structure conversion under unmappable data sets. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] See also Figure 1-Figure 4 As shown, the present invention is a multi-stage caching method for accelerating AI data processing, comprising the following steps: The user first enables the cache server in the terminal, then creates a session and obtains the corresponding ID, and then uses the session ID to create a cache instance ds.DatasetCache(), which needs to be passed in when using the cache later. The user then customizes the image preprocessing operation Transform, which will act on the dataset.

[0020] In the context of cache acceleration technology design, the APIs of data loading operators and map operators have been updated. If the user chooses to cache in the data loading phase, the cache instance is passed to the "cache" flag of Cifar10Dataset(). If the user chooses to cache in the data preprocessing phase, the cache instance is passed to the "cache" flag of map(). At this point, the user has completed the construction of the data processing pipeline and the insertion of the cache operator.

[0021] Set up a cache client, cache server and shared space. The user cannot communicate directly with the cache server through the cache client. You only need to insert a cache operator at the specified position in the data processing pipeline. The cache operator will automatically call the cache client interface. The cache server is used to store cache data sets, which include loaded original data and preprocessed data. The cache client and cache server use the gRPC framework and shared space to exchange information. Each request is sent in protobuf format, which is a data format for efficiently serializing structured data.

[0022] When data tensors need to be transferred, they must first be serialized. We call different functions to serialize the tensor and its corresponding header information. The header information includes the data size, an ID that uniquely identifies the tensor row, and the size of the header itself. After serializing the tensor, we write it to shared memory to optimize the transfer process. Convert the constructed data processing pipeline into a graph structure, perform node validation on the graph structure, call the CacheValidationPass function, start from the starting node of the graph structure, traverse all nodes of the entire graph structure according to the data processing path, and filter out nodes for batch, concat, filter, skip, take, and zip operations. If there is no pre-inserted cache operation in the subsequent data processing path of any node, the validation is considered to be successful. Otherwise, the validation is considered to be an error, and an error report is output after the traversal to remind the user to modify the insertion position of the cache operation. Verification is particularly important in map operators that use random data enhancement, when cache operators are nested (for example, cache is used in both data loading and preprocessing stages), or when a repeat operator is added after a cache operator in a mappable data set.

[0023] After verification, the cached data set is divided into mappable data sets and non-mappable data sets according to whether they support full mapping into memory. For different data sets, corresponding methods are used to convert the graph structure.

[0024] When the cache data set is a mappable data set, a cache merge node (CacheMergeNode) is created in the graph structure and inserted into the map node as the parent node of the map node. The sampler is obtained from the data loading node, and a cache lookup node (CacheLookupNode) is created based on the sampler. The cache lookup node is used to replace the sampler to find and return the cached data in the cache merge node. The Build() function of all nodes in the graph structure is called to build the corresponding operator of each node runtime in turn, generate an execution graph, and mark the operator corresponding to the cache node as the cache merge operator (CacheLookupOperator), and the operator corresponding to the cache lookup node as the cache lookup operator (CheLookupOpeacrator). When the cached data set is an unmappable data set, obtain the sampler corresponding to the data loading node belonging to the leaf node in the graph structure, create a cache node (CacheNode) according to the sampler, and use the cache node to replace the sampler. Insert the cache node into the map node as the parent node of the map node, call the Build() function of all nodes in the graph structure, build the operator corresponding to each node runtime in turn, generate an execution graph, and mark the operator corresponding to the cache node as a cache operator (CacheOperator).

[0025] It should be noted that CacheLookupOperator, CheLookupOpeacrator, and CacheOperator are all operators designed by us for caching operations. We only enable one or two of them according to different data sets to perform caching tasks in the two stages of the data processing pipeline.

[0026] The present invention also considers the optimization of repeated operators. We construct a RepeatPass to optimize repeated operators in the graph structure containing cache operators. Since mappable and non-mappable data sets have different processes in cache processing, two different methods are used to optimize the repeated operators in their respective data processing pipelines. The specific optimization process is as follows: For mappable datasets, if the Repeat operator is inserted before the Cache operator, when RepeatPass accesses the Cache Lookup operator, it does not set the repeat count immediately; instead, it saves its shared pointer to the Cache Lookup member variable. Later, when the Repeat operator is accessed, the repeat count of the Cache Lookup operator is set according to the repeat count specified in the Repeat operator to ensure consistency. Conversely, if the Repeat operator is inserted after the Cache operator, RepeatPass also does not set the repeat count when accessing the Cache Lookup operator; it sets the same repeat count only after accessing the Cache Merge operator. For unmappable datasets, we focus on the case where a repeating operator is inserted after a cached operator. Specifically, we consider the cached object that is repeatedly operated by the repeating operator as one cached operator. When RepeatPass traverses the graph structure from top to bottom, each cached operator is added to the cache node stack. Subsequently, when RepeatPass traverses the graph structure from bottom to top, if the cached operator is accessed again, the previously cached operator is taken out of the stack and its repeat count is set to 1, so that the cached operator is only executed once to fully store the data in the cache server.

[0027] For the cache merge operator (CheLookupOpeacrator) and cache lookup operator (CacheLookupOperator), their specific operation process is as follows: When the dataset loading operator is changed to perform indexing and encoding of data, the main thread of the cache lookup operator starts to retrieve data row IDs from its sampler. After obtaining the required data rows from the sampler, the cache lookup operator is responsible for requesting the corresponding data rows from the cache server and submitting the cache-missed data IDs to the leaf node's data loading operator for re-reading and parsing. It then sends the data stream containing placeholders to the cache merge operator. The cache merge operator keeps waiting for the cache-missed data stream to be filled, and replaces the placeholders with actual data, and finally passes the complete data to the upper-level operator; The present invention further optimizes the above steps. After the main thread retrieves the data row ID from the sampler, the cache lookup operator does not directly distribute each data row ID to the worker thread, which then sends a request to the cache server. This is because, by default, the list of data row IDs distributed to each worker thread contains only one element, which leads to poor performance because each worker thread only sends a query request for one data row ID. Therefore, the main thread of the cache lookup operator starts several additional worker threads, called Prefetchers. After collecting a specified number of data row IDs, the main thread sends a list containing these IDs to each Prefetcher through an internal queue, allowing the Prefetcher to send a query request to the cache server; The cache merge operator starts three groups of worker threads at runtime: cache hit flow thread, cache miss flow thread, and cleanup thread. The cache hit flow worker thread repeatedly calls the cache lookup operator to retrieve data. For the two tasks of caching raw data and caching pre-processed data, the cache miss flow thread will repeatedly call the leaf node's data loading operator or map operator to process the cache miss ID data. Then, they send data write requests asynchronously to store the data on the cache server. The cleanup thread is responsible for waiting for the data write request to complete and checking the result returned by the cache server.

[0028] For the cache operator (CacheOperator), its specific operation mode is as follows: First the build phase is executed, followed by the fetch phase. During the build phase, the cache operator's main thread starts multiple worker threads, which are blocked while waiting for each worker thread to write all data rows to the cache server. Once all data is cached on the server, the pipeline switches to the fetch phase, where the cache operator requests the corresponding data rows from the cache server; Since all data rows have been cached on the server, the query hit rate of the cache server is 100%. It should be noted that in distributed training, multiple data processing pipelines may share the same cache service, but only one selected pipeline will enter the build phase, build the cache service and cache all data to the cache server, while other pipelines will only block and continue to query the current stage of the cache server until the build phase is completed.

[0029] When the data processing pipeline is started, data processing and cache write and cache read operations are performed according to the execution graph. The specific steps of these two operations are: The method of the present invention is a single-node cache technology, which means that the cache client and the cache server are located on the same physical machine. If the amount of data transmission is too large, the performance of using gRPC alone may not reach the ideal level. Therefore, we have optimized the performance of this situation through shared memory. The cache server pre-allocated 4GB of shared space. Shared memory allows multiple processes to share the same physical memory area without additional memory copying, making data visible and operable directly in the shared memory area, thereby reducing latency and memory overhead during data transmission.

[0030] When data needs to be written to the cache, the cache client submits an asynchronous buffer request to the gRPC queue. After the gRPC thread returns the shared memory address, the cache client continues to write data to the asynchronous buffer. Once the buffer is full, the client submits a cache write request to the gRPC queue. Subsequently, the gRPC thread submits the cache write request to the request queue of the cache server. This cache write request will wait in the request queue for the server's worker thread to write the data from the shared memory to the cache memory. It should be noted that during the process of asynchronous buffer writing to the cache, the cache client will not block and wait for the cache server to complete the write and return a response, but will start using the next available asynchronous buffer. This minimizes the time the cache client waits for the cache server to respond. Finally, the cached result is returned to the client.

[0031] When the cached data needs to be read, the cache client submits a data row request to the gRPC queue. The gRPC thread locates the data row in memory, then allocates shared memory for the data row and submits a data extraction request to the cache server's request queue. The server's worker thread writes the data from the cache memory to the shared memory. Once the data is read into the shared memory, the gRPC worker thread returns the shared memory location of the data row to the cache client. After the cache client copies the data row from the shared memory to the target location, it submits a request to release the shared memory to the gRPC queue, and the gRPC thread completes the release process.

[0032] The present invention also considers optimizing memory management for cache write and cache read operation processes, and the specific optimization process is as follows: When optimizing memory management, based on the NUMA framework, we configure a NUMA framework-aware cache memory pool according to the ratio of the cache server's memory space to the system's total memory space. Each cache memory pool consists of memory blocks (Arenas), and the number of memory blocks is set to the number of CPUs. Memory space is allocated to memory blocks through specific NUMA nodes. In addition, we bind each thread in the gRPC thread group and the worker thread group that handles the cache service to a specific NUMA node. This approach has the following advantages: (1) When the gRPC thread processes a request, it continuously allocates memory to create CacheServerRequest objects as labels for gRPC requests. By binding to a certain node, the gRPC thread can access local NUMA memory, avoiding memory calls to other CPUs, thereby reducing access latency.

[0033] (2) Similarly, when a cache server worker thread processes a cache request, we first determine the NUMA node corresponding to the worker thread and select a memory block from that node for memory allocation, rather than random allocation. This ensures that the worker thread always caches data in the memory of the local NUMA node, thereby reducing access latency.

[0034] The present invention also takes into account multi-task training during AI training. In the present invention, a single cache server can provide services for multiple cache clients, and can uniquely identify the cache data required by the user through the session ID of the cache client and the CRC code of the cache graph structure in the data processing pipeline. Therefore, in order to share the same cache service, the pipelines do not have to be completely consistent, as long as the same session ID is specified, so the present invention is also very suitable for distributed training.

[0035] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0036] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0037] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A multi-stage caching method for accelerating AI data processing, characterized in that: The following steps are involved: Build a data processing pipeline through the top-level interface, insert cache operations in the data processing pipeline in advance and mark the position; A cache client, a cache server and a shared space are set up, and the data processing pipeline communicates through the cache client and the cache server. The cache server is used to store cache data sets, and the cache data sets include the loaded original data sets and the data sets after preprocessing. The cache client and the cache server use the gRPC framework and the shared space to exchange information; Convert the constructed data processing pipeline into a graph structure and perform node verification on the graph structure. After verification, divide the cached dataset into mappable dataset and non-mappable dataset according to whether the cached dataset supports full mapping to memory. For different datasets, use corresponding methods to convert the graph structure, generate an execution graph and optimize repeated operators. Start the data processing pipeline and perform data processing according to the generated execution graph. When the data pipeline performs cache writing and cache reading of the cache data set, the cache client is called to send request information to the cache server. The cache server receives the request information and performs corresponding operations. After completing the operation, it returns the corresponding information to the cache client.

2. A multi-stage caching method for accelerating AI data processing according to claim 1, characterized in that: When the gRPC framework is used between the cache client and the cache server, variables rq and reply are set for each request sent by the cache client to the cache server, wherein rq is used to record information that needs to be sent to the cache server, and reply is used to obtain response information sent from the cache server.

3. The multi-stage caching method for accelerating AI data processing according to claim 1, characterized in that: The specific process of verifying the graph structure is as follows: Call the cache verification function, starting from the starting node of the graph structure, traverse all nodes of the entire graph structure according to the data processing path, and filter out nodes for batch processing, connection, filtering, skipping, retrieval and compression operations respectively. When there is no pre-inserted cache operation in the subsequent data processing path of any node, the verification is judged to be passed, otherwise it is judged to be a verification error, and an error report is output after the traversal is completed to remind the user to modify the insertion position of the cache operation.

4. A multi-stage caching method for accelerating AI data processing according to claim 3, characterized in that: After the judgment and verification are passed, the specific process of converting the graph structure is as follows: When the cache data set is a mappable data set, a cache merge node is created in the graph structure and inserted into the map node as the parent node of the map node. The sampler is obtained from the data loading node, and a cache search node is created based on the sampler. The sampler is replaced with the cache search node to search for and return the cached data in the cache merge node. The Build() function of all nodes in the graph structure is called to build the corresponding operator of each node in turn, generate an execution graph, and mark the operator corresponding to the cache merge node as the cache merge operator, and the operator corresponding to the cache search node as the cache search operator. When the cached data set is an unmappable data set, obtain the sampler corresponding to the data loading node belonging to the leaf node in the graph structure, create a cache node based on the sampler, and use the cache node to replace the sampler. Insert the cache node into the map node as the parent node of the map node, call the Build() function of all nodes in the graph structure, build the operator corresponding to each node runtime in turn, generate an execution graph, and mark the operator corresponding to the cache node as a cache operator.

5. A multi-stage caching method for accelerating AI data processing according to claim 4, characterized in that: The specific process of repeated operator optimization on the graph structure is as follows: Determine the type of cache data set. When the cache data set is a mappable data set, obtain the position of the repeat operator in the operation queue. If the repeat operator is located before the cache search operator and the cache merge operator, set the repeat count of the cache search operator according to the repeat count set in the repeat operator. Otherwise, set the repeat count of the cache search operator according to the repeat count set in the cache merge operator. When the cached data set is an unmappable data set, the data that is repeatedly processed by the repeated operator is marked as a cache object, and the graph structure is traversed from top to bottom. During the traversal, the cache objects are added to the cache node stack one by one. When the repeated optimization mechanism traverses the graph structure from bottom to top, the corresponding cache objects are taken out from the cache node stack one by one, and the cache object repeat count is set to 1.

6. A multi-stage caching method for accelerating AI data processing according to claim 4, characterized in that: The specific operation process of the cache merge operator and cache search operator is as follows: When the data loading operator performs data loading operations, it provides the sampler to the cache search operator to obtain the ID of the data row. After the cache search operator obtains the ID, it starts N child threads in the main thread, where N is a positive integer. After collecting a preset number of IDs, the main thread assigns the ID to the child threads through the internal queue. The child threads send cache query requests to the cache server in batches. If the ID of the cache query request hits the cache server, the data stream corresponding to the ID is sent to the cache merge operator. Otherwise, the cache search operator sends the missed ID to the data loading operator. The data loading operator re-reads the parsed data stream and sends the data stream with placeholders to the cache merge operator. The cache merge operator continuously waits for and receives data streams, merges the data streams corresponding to the hit IDs and stores them in the cache server. For data streams containing placeholders, the placeholders are replaced with actual data. After the replacement is completed, the complete data stream is stored in the cache server.

7. The multi-stage caching method for accelerating AI data processing according to claim 4, characterized in that: The specific operation process of the cache operator is as follows: The cache operator starts M sub-threads in the main thread, where M is a positive integer. Each sub-thread remains in a blocked state before successfully storing the data in the cache server and receiving the return information. When the data storage tasks of all sub-threads are completed and the return information is obtained, the cache operator switches to the acquisition state and starts reading the data stored in the cache server through the cache operator.

8. The multi-stage caching method for accelerating AI data processing according to claim 2, characterized in that: The specific process of the data processing pipeline performing cache write and cache read operations on the cache data set is as follows: When writing to the cache, the data processing pipeline calls the cache client to submit an asynchronous buffer request to the gRPC queue. After receiving the request, the gRPC queue returns a shared memory address. Then the cache client starts to continuously write the cache data set to the shared memory. When the shared memory storage is full, the buffer client submits a cache write request to the gRPC queue. The gRPC thread submits the cache write request to the request queue of the cache server. The working thread of the cache server writes the cache data set from the shared memory to the memory of the cache server in turn according to the request queue. When performing cache reading, the data processing pipeline calls the cache client to submit a data row request to the gRPC queue. After receiving the request, the gRPC queue locates the data row in the memory of the cache server and allocates shared memory for the data row. At the same time, it submits a data extraction request to the request queue of the cache server. The working thread of the cache server writes the cache data set from the memory of the cache server to the shared memory. When the cache data set is successfully read into the shared memory, the gRPC queue returns the shared memory address corresponding to the data to the cache client. The cache client copies the cache data set from the shared memory to the target location in the data processing pipeline, and submits a request to release the shared memory to the gRPC queue. The gRPC thread completes the memory release.

9. A multi-stage caching method for accelerating AI data processing according to claim 8, characterized in that: The data processing pipeline also includes optimizing memory management during the cache write and cache read operations of the cache data set. The specific optimization process is as follows: When optimizing memory management, based on the NUMA framework, according to the ratio of the memory space of the cache server and the total memory space of the system, a NUMA framework-aware cache memory pool is configured. Each cache memory pool consists of memory blocks. The number of memory blocks is set to the number of CPUs, and memory space is allocated to the memory blocks through specific NUMA nodes.

Images