A cache processing method for a large language model and related equipment

Abstract

The application relates to a large language model, and discloses a cache processing method for a large language model and related equipment, which is applied to a CPython process enabled with a global interpreter lock (GIL). When a Python business layer holds the GIL, a callback is registered for a key-value (KV) cache loading or unloading request, and a C++ I / O task description item without a Python object is generated. A C++ native thread processes an SSD I / O completion result when the GIL is not held and a Python object is not accessed, and a callback description item containing a callback identifier and a task state is pushed into a C++ callback queue. An independent callback thread forms a callback batch when the number of queue reaches a threshold value or the waiting time of the earliest enqueued description item reaches a threshold value, then acquires the GIL to execute Python callbacks in the batch, and uniformly releases the GIL after the execution is completed. The method reduces the GIL competition and GIL acquisition times of an I / O completion path and an inference thread.

Description

Technical Field

[0001] This application relates to the fields of large language model reasoning and hierarchical cache management technology, and in particular to a cache processing method and related equipment for large language models. Background Technology

[0002] Large language models based on the Transformer architecture save the key and value tensors corresponding to historical tokens during autoregressive inference, forming a KV cache to avoid redundant computation in subsequent generation stages. As the context length and the number of concurrent requests increase, the storage space occupied by the KV cache grows rapidly. To reduce the pressure on the graphics processor's memory, the inference system can offload KV cache entries that are not currently involved in computation to host memory or a solid-state drive (SSD), and reload them when needed for reuse.

[0003] In an implementation using a CPython interpreter with the Global Interpreter Lock (GIL) enabled, where the cache management component and the inference scheduling component reside in the same process, blocking file read / write system calls can release the GIL during execution. However, Python object parsing and parameter conversion during I / O task creation, I / O completion event postback, Python callback execution, and cache metadata updates still require access to Python objects and holding the GIL. When multiple I / O tasks complete consecutively within a short period, the repeated acquisition and release of the GIL by multiple completion paths can compete with the inference scheduling thread.

[0004] Therefore, it is necessary to clearly separate code sections that must access Python objects from code sections that can execute using only native data, and to avoid multiple I / O worker threads entering the Python interpreter separately upon completion. This application primarily handles lock contention before I / O requests enter the native execution layer and when I / O completes and returns to the Python business layer, without relying on the premise that the underlying storage device's blocking read / write operations always hold the GIL. Summary of the Invention

[0005] This application provides a caching method and related device for large language models, which closes the lifecycle of Python objects and memory buffers in CPython co-process deployments with GIL enabled, reduces GIL contention during I / O completion phases, and maintains consistency between KV cache state updates and inference scheduling.

[0006] The first aspect of this application provides a caching method for large language models. Applied to a process with the Global Interpreter Lock (GIL) enabled, the process includes a business layer and a transport layer. The method includes:

[0007] When the business layer detects that it needs to load the key-value pair (KV) cache from the solid-state drive (SSD) or unload the KV cache to the SSD, while holding the GIL, it registers the callback function corresponding to this I / O task and obtains the corresponding callback identifier, and generates an I / O task description item that does not contain Python objects;

[0008] Submit the I / O task description to the transport layer, and release the GIL after submission;

[0009] The transport layer's worker thread executes the SSD's I / O requests based on the I / O task description without holding the GIL;

[0010] After the SSD's I / O request is completed, the transport layer constructs a callback description that does not contain a Python object and pushes the callback description into the callback queue; wherein, the callback description includes at least the callback identifier and the task status of this I / O task;

[0011] The callback queue is monitored by a callback execution thread independent of the worker thread, and one or more callback description items are removed from the callback queue to form the current callback batch when the batch processing trigger condition is met.

[0012] Obtain the GIL, execute the corresponding callback functions in batches according to the callback description items in the current callback batch, and release the GIL after execution.

[0013] Optionally, the business layer is a Python business layer, and the transport layer is a C++ I / O execution module;

[0014] The Python business layer communicates with the C++ I / O execution module via the pybind11 interface;

[0015] The I / O task description item includes at least the task identifier, task type, key-value identifier, SSD storage path, data length, memory buffer address or memory buffer handle, and the callback identifier;

[0016] The memory buffer remains valid until it is confirmed that the I / O backend will no longer access the memory buffer.

[0017] Optionally, the I / O task description item further includes task type and task priority, wherein the task type includes load KV cache task and unload KV cache task, and the I / O task description item includes basic priority and scheduling priority; the method further includes:

[0018] When generating the I / O task description item, the basic priority is determined according to the task type, and the basic priority is used as the initial scheduling priority; wherein, the basic priority of the loading KV cache task is higher than the basic priority of the unloading KV cache task.

[0019] Optionally, before submitting the I / O task description item to the transport layer, the method further includes:

[0020] Based on the basic priority, the scheduling priority is determined according to at least one of the urgency of the corresponding I / O request and the reuse frequency of the corresponding KV cache;

[0021] The urgency level is determined at least based on the waiting time or remaining time budget of the I / O request; the reuse frequency is determined based on the number of times the KV cache corresponding to the KV identifier is accessed within a preset statistical window.

[0022] Optionally, the method further includes:

[0023] A priority task queue is maintained within the transport layer; wherein, the worker thread retrieves the highest priority I / O task description from the priority task queue and executes it, and the priority task queue is the only task queue maintained within the transport layer, and there are no additional sub-queues for worker threads to retrieve tasks.

[0024] Optionally, the batch processing triggering conditions include:

[0025] The number of pending callback descriptions in the callback queue reaches a preset threshold N, where N is a positive integer greater than 1; or,

[0026] The callback queue is not empty, and the waiting time calculated from the enqueue time of the earliest pending callback description item entering the callback queue reaches a preset time threshold T; where T is a positive integer in microseconds.

[0027] Optionally, the method further includes;

[0028] When the number of pending callback description items reaches the preset number threshold N, N callback description items are removed from the callback queue at once to form the current callback batch;

[0029] When the waiting time reaches the preset time threshold T, the currently pending callback description items, no more than N, are removed from the callback queue at once to form the current callback batch.

[0030] Optionally, the method further includes:

[0031] When executing the callback functions in the current callback batch, they are executed sequentially according to the order in which each callback description item in the current callback batch entered the callback queue;

[0032] If any callback function encounters an exception, the exception is recorded and the remaining callback functions in the current callback batch are executed.

[0033] Optionally, the SSD I / O operation is submitted via an asynchronous I / O interface, or executed via a synchronous I / O interface;

[0034] In any I / O mode, the thread performing the SSD I / O operation does not hold the GIL and does not access the objects in the business layer.

[0035] A second aspect of this application provides a caching processing apparatus for large language models, applied to a process with the Global Interpreter Lock (GIL) enabled, wherein the process includes a business layer and a transport layer, and the apparatus includes:

[0036] The registration unit is used to register a callback function corresponding to the current I / O task and obtain the corresponding callback identifier when the business layer detects that it is necessary to load the key-value pair KV cache from the solid-state drive SSD or unload the KV cache to the SSD, while holding the GIL; and to generate an I / O task description item that does not contain Python objects.

[0037] A release unit is used to submit the I / O task description item to the transport layer and release the GIL after submission;

[0038] An execution unit is configured to execute the I / O request of the SSD according to the I / O task description item through the worker thread of the transport layer without holding the GIL;

[0039] The construction unit is configured to, after the I / O request of the SSD is completed, construct a callback description item without containing a Python object by the transport layer, and push the callback description item into the callback queue; wherein, the callback description item includes at least the callback identifier and the task status of the current I / O task;

[0040] The monitoring unit is used to monitor the callback queue through a callback execution thread independent of the worker thread, and remove one or more callback description items from the callback queue to form the current callback batch when the batch processing trigger condition is met;

[0041] The acquisition unit is used to acquire the GIL, execute the corresponding callback functions in batches according to the callback description items in the current callback batch, and release the GIL after execution.

[0042] Optionally, the business layer is a Python business layer, and the transport layer is a C++ I / O execution module;

[0043] The Python business layer communicates with the C++ I / O execution module via the pybind11 interface;

[0044] The I / O task description item includes at least the task identifier, task type, key-value identifier, SSD storage path, data length, memory buffer address or memory buffer handle, and the callback identifier;

[0045] The memory buffer remains valid until it is confirmed that the I / O backend will no longer access the memory buffer.

[0046] Optionally, the I / O task description item further includes task type and task priority, wherein the task type includes loading KV cache task and unloading KV cache task, and the I / O task description item includes basic priority and scheduling priority; the device further includes: a determination unit;

[0047] The determining unit is configured to determine the basic priority based on the task type when generating the I / O task description item, and use the basic priority as the initial scheduling priority; wherein the basic priority of the loading KV cache task is higher than the basic priority of the unloading KV cache task.

[0048] Optionally, the device further includes:

[0049] The determining unit is further configured to determine the scheduling priority based on the basic priority, according to at least one of the urgency of the corresponding I / O request and the reuse frequency of the corresponding KV cache;

[0050] The urgency level is determined at least based on the waiting time or remaining time budget of the I / O request; the reuse frequency is determined based on the number of times the KV cache corresponding to the KV identifier is accessed within a preset statistical window.

[0051] Optionally, the device further includes: a maintenance unit;

[0052] The maintenance unit is used to maintain a priority task queue inside the transport layer; wherein, the worker thread retrieves the highest priority I / O task description from the priority task queue and executes it, and the priority task queue is the only task queue maintained inside the transport layer, and there are no additional sub-queues for worker threads to retrieve tasks.

[0053] Optionally, the batch processing triggering conditions include:

[0054] The number of pending callback descriptions in the callback queue reaches a preset threshold N, where N is a positive integer greater than 1; or,

[0055] The callback queue is not empty, and the waiting time calculated from the enqueue time of the earliest pending callback description item entering the callback queue reaches a preset time threshold T; where T is a positive integer in microseconds.

[0056] Optionally, the device further includes: a removal unit;

[0057] The removal unit is used to remove N callback description items from the callback queue at one time to form the current callback batch when the number of pending callback description items reaches the preset number threshold N.

[0058] The removal unit is further configured to remove, when the waiting time reaches the preset time threshold T, a total of no more than N callback description items currently pending processing from the callback queue to form the current callback batch.

[0059] Optionally, the device further includes: a recording unit;

[0060] The execution unit is further configured to execute the callback functions in the current callback batch sequentially according to the order in which each callback description item in the current callback batch enters the callback queue when executing the callback functions in the current callback batch.

[0061] The recording unit is used to record the exception when any callback function execution fails and to continue executing the remaining callback functions in the current callback batch.

[0062] Optionally, the apparatus further includes: the SSD I / O operation is submitted via an asynchronous I / O interface, or executed via a synchronous I / O interface;

[0063] The execution unit is further configured such that, in any I / O mode, the thread performing the SSD I / O operation does not hold the GIL and does not access the objects in the business layer.

[0064] A third aspect of this application provides an electronic device comprising at least one processor and a memory connected to the processor, the memory storing a computer program that, when executed by the processor, implements the method described in the first aspect of this application.

[0065] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect of this application.

[0066] The caching method for large language models provided in this application restricts I / O task creation and callback registration to the business layer holding the GIL, while sinking the actual SSD I / O scheduling and execution to the GIL-free transport layer. Furthermore, callback processing is decoupled and executed in batches on independent threads, thus achieving a systematic minimization of the GIL binding scope. The technical advantages are: I / O worker threads do not need to hold the GIL during SSD read / write operations, allowing for true parallelism of multiple I / O operations and fully releasing the multi-queue concurrency capabilities of NVMe SSDs; simultaneously, callback execution and I / O execution are completely decoupled at the physical thread level, eliminating mutual blocking between I / O threads and the inference main thread at the GIL level, enabling the inference main thread to schedule GPU computation in a timely manner. In addition, the batch callback mechanism merges multiple independent GIL acquisitions into a single batch acquisition, significantly reducing the frequency of GIL entry and exit and interpreter scheduling overhead. In summary, the above improvements significantly enhance SSD bandwidth utilization and inference system throughput in hierarchical KV caching scenarios, effectively reduce first-token latency, and have outstanding practical value in the production deployment of large language models with long contexts and high concurrency. Attached Figure Description

[0067] The accompanying drawings are used to illustrate embodiments of this application. The same reference numerals in the drawings denote the same or corresponding objects. The drawings are schematic representations and do not require that all objects be drawn to actual scale.

[0068] Figure 1 This is a schematic diagram of the same-process Python / C++ caching architecture in an embodiment of this application;

[0069] Figure 2 This is a schematic diagram of a caching method for a large language model according to an embodiment of this application;

[0070] Figure 3 This is a schematic diagram illustrating the operation dependencies and state transitions of the same KV identifier in an embodiment of this application;

[0071] Figure 4 This is a schematic diagram of a single-priority task queue scheduling according to an embodiment of this application;

[0072] Figure 5 This is a schematic diagram of the callback batch processing flow in an embodiment of this application;

[0073] Figure 6 This is a schematic diagram of the cache processing device structure according to an embodiment of this application;

[0074] Figure 7 This is a schematic diagram of the electronic device structure according to an embodiment of this application. Detailed Implementation

[0075] The embodiments of this application will be further described below with reference to the accompanying drawings. The described embodiments are used to explain the technical solutions of this application and are not intended to limit this application to a single software framework, storage device model, or parameter combination.

[0076] The caching method for large language models proposed in this application is mainly applied to processes with the Global Interpreter Lock (GIL) enabled. These processes have a business layer and a transport layer. The business layer is a Python business layer, and the transport layer is a C++ I / O execution module. The Python business layer and the C++ I / O execution module communicate via the pybind11 interface.

[0077] Therefore, in one specific embodiment, the caching method of this application is applied to a CPython process with the Global Interpreter Lock (GIL) enabled. The CPython process includes a Python business layer, a C++ I / O execution module that communicates with the Python business layer via the pybind11 interface, and a C++ callback execution module. The C++ I / O execution module includes a task receiving interface and a task queue. The C++ callback execution module includes a C++ callback queue and a callback execution thread independent of the native thread of the C++ I / O execution module. The "GIL-enabled CPython process" referred to in this application means a CPython build that requires a thread to hold the GIL to access Python objects or call Python / C APIs, and where the Python business layer and the inference scheduling thread reside in the same interpreter process. This application does not presuppose that different processes share the GIL, nor does it consider optional free-threaded builds as the runtime environment defined in the claims.

[0078] The term "Python object" as used in this application includes Python callback function objects, Python containers, Python exception objects, and other objects whose reference counts need to be managed by the Python interpreter. The term "raw data" or "not containing Python objects" as used in this application refers to data that can be copied, moved, and destroyed by C++ code without calling Python / C APIs and without increasing or decreasing the reference count of Python objects.

[0079] See Figure 1Within the same CPython process, there are an inference scheduling module 101, a key-value management and callback registration module 102, a task receiving and dependency checking module 103, an I / O execution module 104, an SSD 105, and a callback execution module 106. The I / O execution module 104 contains only one priority task queue for worker threads to consume. The callback execution module 106 includes a C++ callback queue, a batch control component, and a callback execution thread independent of the I / O worker thread and the completion thread.

[0080] The inference scheduling module 101 generates KV cache loading or unloading requests and hands them over to the KV management and callback registration module 102. While holding the GIL, the KV management and callback registration module 102 reads the request parameters, determines the KV identifier and buffer to be operated on, and registers the Python callback function in its internal callback registry, obtaining the callback_id.

[0081] The internal callback registry holds valid references to Python callback functions. `callback_id` is a native identifier that can be independently stored and compared in C++ code. The worker and completion threads in I / O execution module 104 only pass `callback_id`; they do not hold, copy, destroy, or interpret Python callback function objects.

[0082] The KV management and callback registration module 102 generates C++ I / O task description items and submits them to the task receiving and dependency checking module 103. The C++ I / O task description item includes at least the task identifier (task_id), task type, KV identifier, SSD storage path, file offset, data length, buffer address or buffer_handle, basic priority, scheduling priority, version number, operation sequence number, and callback_id.

[0083] Key-value (KV) identifiers can be determined by the model identifier and model version, namespace, layer identifier, token block digest, block number, and the type of K or V data. C++ I / O task descriptions may also include data type, tensor shape, stride, checksum, deadline, and buffer memory type.

[0084] For tasks whose storage space is provided by Python tensors or buffers, the key-value management and callback registration module 102 establishes a mapping between the buffer owner and the buffer_handle while holding the GIL. The buffer and its Python owner remain valid until a completion event of the original request is received or a clear cancellation result is obtained, confirming that the I / O backend will no longer access the buffer_handle. The native thread in the I / O execution module 104 only accesses the memory pointed to by the buffer_handle and valid for the I / O backend, without interpreting the corresponding Python object.

[0085] Python threads that call the pybind11 interface acquire the Global Interpreter Lock (GIL) upon entering the binding function. After the binding function completes Python parameter checks, callback registration, and native field conversions, it can release the GIL within its local scope and write the task description to the task receiving and dependency checking module 103. Upon exiting this local scope, it reacquires the GIL and returns to the Python business layer. Any creation, reference count changes, or exception conversions of Python objects occur within the scope holding the GIL.

[0086] The task receiving and dependency checking module 103 writes tasks with satisfied dependencies into a unique priority task queue within the I / O execution module 104, and registers tasks with unsatisfied dependencies into a pending task table. Worker threads in the I / O execution module 104 only retrieve tasks from this unique priority task queue, performing synchronous read / write operations on the SSD 105 or submitting asynchronous read / write requests. The worker thread or completion thread that processes the results generates a callback description; the callback description includes task_id, callback_id, key-value identifier, version number, operation sequence number, actual number of bytes completed, task status, and error code, but does not contain a Python object.

[0087] The worker thread or completion thread that has finished processing the results pushes the callback descriptor into the C++ callback queue in the callback execution module 106. The callback execution thread determines whether to trigger batch processing based on the queue size and the earliest enqueue time; if triggered, it first forms a local callback batch of no more than N callback descriptors without holding the GIL, and then releases the queue lock and acquires the GIL.

[0088] After acquiring the GIL, the callback execution thread retrieves the Python callback function from the callback registry of the KV management and callback registration module 102 based on the callback_id. It then executes the KV status update and completion notification, and deletes the callback registration entry and releases the corresponding Python reference and buffer owner upon completion or task termination. If a task is not enqueued due to a full queue, submission timeout, or system shutdown, the submission thread deletes the callback registration entry and releases the buffer owner before returning to the Python business layer. After the status update is completed, the callback execution module 106 notifies the KV management and callback registration module 102 or the inference scheduling module 101 to continue processing the corresponding request.

[0089] Figure 1 The arrows from the KV management and callback registration module 102 to the task receiving and dependency checking module 103 indicate the task control flow containing only native fields. Module 103 performs dependency checks on the task before handing it over to the I / O execution module 104. The bidirectional arrows from the I / O execution module 104 to the SSD 105 indicate KV data reading and writing, and the arrows from the I / O execution module 104 to the callback execution module 106 indicate the completion of the descriptor transfer. The callback execution thread in the callback execution module 106 acquires the GIL only when it enters the Python business layer, releases the GIL after completing the batch callback, and returns a status update and notification to module 102.

[0090] See Figure 2 The caching method in this application embodiment includes steps 201 to 206.

[0091] 201. When the business layer detects that it needs to load the key-value pair (KV) cache from the solid-state drive (SSD) or unload the KV cache to the SSD, while holding the GIL, register the callback function corresponding to this I / O task and obtain the corresponding callback identifier, and generate an I / O task description item that does not contain Python objects.

[0092] In some embodiments, a Python callback function corresponding to the current I / O task is registered and a reference to the Python callback function is maintained until the callback is completed, the task is terminated, or the submission fails, the corresponding callback identifier is obtained, and a C++ I / O task description item without Python objects is generated.

[0093] In some embodiments, during the inference process of the large language model, when the Python business layer detects that it needs to load the key-value pair (KV) cache from the solid-state drive (SSD) or unload the KV cache to the SSD, while holding the Global Interpreter Lock (GIL), it registers a Python callback function corresponding to this I / O task and maintains a reference to the Python callback function until the callback is completed, the task is terminated, or the submission fails. It obtains the corresponding callback identifier and generates a C++ I / O task description item that does not contain a Python object. The C++ I / O task description item includes at least the task identifier, task type, KV identifier, SSD storage path, data length, memory buffer address or memory buffer handle, and callback identifier. The memory buffer remains valid until it is confirmed that the I / O backend will no longer access the memory buffer.

[0094] 202. Submit the I / O task description to the transport layer and release the GIL after submission.

[0095] In some embodiments, C++ I / O task descriptions are submitted to the task receiving interface of the C++ I / O execution module via the pybind11 interface. The task receiving interface then writes C++ I / O task descriptions that meet the enqueueing conditions into the task queue. The pybind11 interface returns to the Python business layer before the completion of the current I / O task. Specifically, the Python thread that calls the pybind11 interface only releases the GIL during the execution of C++ code sections that do not access Python objects, and reacquires the GIL before returning to the Python business layer.

[0096] 203. The transport layer worker thread executes the SSD's I / O requests based on the I / O task description without holding the GIL.

[0097] In some embodiments, the worker thread of the C++ I / O execution module retrieves C++ I / O task descriptions from the task queue and performs synchronous SSD I / O operations or submits asynchronous SSD I / O requests based on the C++ I / O task descriptions without holding the GIL or accessing Python objects or calling Python / C APIs.

[0098] In some embodiments, SSD I / O operations employ any of the following I / O modes:

[0099] Asynchronous SSD I / O requests can be submitted via the io_uring interface or the POSIX AIO interface, and the corresponding asynchronous completion result can be obtained by the C++ I / O execution module; or,

[0100] Synchronous SSD I / O operations are performed by calling the pread or pwrite interface through one or more worker threads of the C++ I / O execution module, making the synchronous SSD I / O operations asynchronous relative to the Python business layer;

[0101] In any I / O mode, worker threads in the C++ I / O execution module that perform synchronous SSD I / O operations, or native threads that submit asynchronous SSD I / O requests and process the results, do not hold the GIL and do not access Python objects or call Python / C APIs.

[0102] 204. After the SSD I / O request is completed, the transport layer constructs a callback descriptor that does not contain a Python object and pushes the callback descriptor into the callback queue.

[0103] In some embodiments, after an SSD I / O operation is completed, a callback description item without Python objects is constructed by a worker thread or completion thread in the C++ I / O execution module that processes the completion result, and the callback description item is pushed into the C++ callback queue; wherein, the callback description item includes at least a callback identifier and the task status of this I / O task.

[0104] 205. Monitor the callback queue through a callback execution thread independent of the worker thread, and remove one or more callback descriptions from the callback queue to form the current callback batch when the batch processing trigger condition is met.

[0105] In some embodiments, the C++ callback queue is monitored by the callback execution thread, and when the batch processing trigger condition is met, one or more callback descriptors are removed from the head of the C++ callback queue at once to form the current callback batch without holding the GIL.

[0106] 206. Obtain the GIL, execute the corresponding callback functions in batches according to the callback description items in the current callback batch, and release the GIL after execution.

[0107] In some embodiments, after the current callback batch is formed, the GIL is obtained through the callback execution thread, the corresponding Python callback function is obtained according to the callback identifier in the current callback batch, the Python callback functions in the current callback batch are executed sequentially, and the GIL is released uniformly after the callback execution and callback registration cleanup of the current callback batch are completed.

[0108] This embodiment discloses a caching method that designs I / O task descriptions as pure C++ data structures without Python objects, completely sinking SSD I / O operations to the transport layer for execution. Worker threads do not hold the GIL or access Python objects during the entire I / O process. This allows multiple SSD I / O operations to be executed in true parallel, significantly improving SSD bandwidth utilization. Furthermore, through a layered design of the business layer and transport layer, I / O-intensive tasks are isolated from the CPU-intensive tasks of the inference main thread at the GIL level. The inference main thread can freely acquire the GIL and perform batch building, token scheduling, and GPU kernel startup operations while waiting for I / O completion, without being frequently preempted by I / O threads. Experiments show that in SSD reuse scenarios, inference throughput can be improved by more than 1.2 times, first-token latency (TTFT) is reduced by 35%–45%, and GPU idle waiting time is significantly reduced. Simultaneously, a callback execution thread independent of the I / O worker thread is set up. Callback descriptions after I / O completion are uniformly pushed into a callback queue and centrally processed by this independent thread. The I / O worker threads do not enter the Python callback path at all, eliminating competition for the Global Interpreter Lock (GIL) among multiple I / O threads; the callback execution thread further reduces the frequency of GIL acquisition through a batch processing mechanism. Furthermore, through optimizations such as eliminating cross-queue task migration, reducing the number of lock operations, and batch processing callbacks, the software stack overhead in the I / O path is significantly reduced.

[0109] Specifically, steps 201 to 206 above can be described in a simplified manner.

[0110] Step 201: The Python business layer holds the GIL, registers Python callback functions, keeps the relevant buffers valid, and generates C++ I / O task description items that do not contain Python objects.

[0111] Step 202: Submit the C++ I / O task description to the task receiving interface via the pybind11 interface. The calling thread only releases the GIL in the C++ code section that does not access Python objects, and reacquires the GIL before returning to the Python business layer.

[0112] Step 203: The native thread of the C++ I / O execution module performs synchronous SSD I / O operations without holding the GIL and without accessing Python objects, or submits and processes asynchronous SSD I / O.

[0113] Step 204: After the SSD I / O operation is completed, the worker thread or completion thread that processes the result constructs a callback descriptor that does not contain a Python object and pushes it into the C++ callback queue.

[0114] Step 205: When the quantity threshold N or the time threshold T is met, remove one or more callback descriptors from the C++ callback queue to form the current callback batch. The GIL is not held when forming the batch, and the queue lock is released before waiting for the GIL.

[0115] Step 206: The callback execution thread acquires the GIL, executes the Python callback function in the current batch according to the callback identifier, and releases the GIL after completing the status update, notification and resource cleanup.

[0116] In step 201, the task type includes at least a load KV cache task and an unload KV cache task. In an optional implementation, the task type also includes a delete task that removes a specified KV version from the SSD. The delete task does not require a buffer field, but it includes at least a KV identifier, version number, storage path, and callback_id sufficient to uniquely identify the target data.

[0117] For loading or unloading tasks, the buffer address or buffer_handle in the task description is a necessary field; objects that only provide the task type or callback identifier but lack the location data and fields required to complete data transmission do not constitute an executable I / O task description in this embodiment.

[0118] The base priority of a loading task can be higher than that of an unloading task. In an optional task deletion implementation, the base priority of a deleted task is lower than that of an unloading task. The base priority represents the impact of the task type on the inference critical path under normal circumstances.

[0119] Scheduling priority is determined by combining at least one of urgency and reuse frequency with a base priority. Urgency can be determined based on the request's waiting time, remaining time budget, or service level; reuse frequency can be determined based on the number of times the same key-value pair or context prefix is ​​accessed within a preset statistical window. Urgency is determined at least by the I / O request's waiting time or remaining time budget; the longer the waiting time or the smaller the remaining time budget, the higher the urgency. Reuse frequency is determined by the number of times the key-value cache corresponding to the key-value pair is accessed within the preset statistical window; the more times it is accessed, the higher the reuse frequency.

[0120] The scheduling priority is written to the task description before the task is submitted to the task receiving and dependency checking module 103. If aging processing is required for queued tasks, the scheduling key can be updated using an indexed heap structure under the protection of the synchronization mechanism, or the heap can be recalculated and rebuilt periodically; queue elements must not be modified in place without synchronization. For tasks with the same scheduling priority, a definite dequeue order can be maintained according to the monotonically increasing submission sequence number.

[0121] See Figure 3 For the same KV identifier and version, the status management table records the set of valid storage locations, reference count, current operation status, and the next expected operation sequence number. Figure 3 The displayed statuses include No valid version 301, Loading 302, Available 303, In storage 304, Already stored 305, and Deleting 306.

[0122] After the initial calculation or regeneration of the KV cache, the status changes from 301 (no valid version) to 303 (available). Loading can only begin from 305 (already stored) and proceed to 302 (loading in progress) if a corresponding valid version exists in the SSD. Once the data length verification passes and is completed (if a checksum is configured), the status changes to 303 (available). If loading fails, is canceled, or the result version does not match, the status reverts to 305 (already stored), and the target KV cache is not marked as available.

[0123] After initiating an uninstallation from the available 303 status, the status changes to stored 304. Data can first be written to a temporary location corresponding to the target version. Once the length verification passes and the verification is completed if a checksum is configured, the version is registered as stored 305. If the write fails or is canceled, it will fall back to available 303, while retaining the original valid version.

[0124] The deletion task only proceeds from stored (305) to deletion (306) when the reference count is zero and there are no pending loading or unloading tasks. Upon successful deletion, it proceeds to no valid version (301); upon failure or cancellation, it reverts to stored (305). Before execution, the key-value (KV) identifier, version number, and operation sequence number are verified again. Failure, cancellation, and timeout are considered task results; no further long-term KV error states are set.

[0125] Loading, unloading, and optional deletion tasks of the same key-value identifier and version take effect sequentially according to their operation sequence numbers. Tasks whose preceding dependencies have not yet been satisfied are registered in the dependency record of the state management table using the task identifier `task_id`. Simultaneously, a complete C++ I / O task description is stored in the pending task table indexed by `task_id`, thus maintaining the lifecycle of `callback_id` and `buffer_handle`. After the preceding task is completed, the state management table updates the next expected operation sequence number, retrieves the task from the pending task table based on `task_id`, re-checks its dependencies, and writes it to the unique priority task queue when the dependency is satisfied. Neither the dependency record nor the pending task table is a task sub-queue; different key-value identifiers can be rearranged according to scheduling priority.

[0126] For tasks using operation sequence numbers, the task receiving and dependency checking module 103 first reserves the capacity of C_task, and then atomically allocates and publishes the operation sequence number. Tasks not received by the task receiving interface due to full capacity, submission timeout, or closure are not allocated operation sequence numbers; if sequence numbers are pre-generated in the implementation, they are atomically marked as skipped when submission is rejected. When a preceding task terminates with a success, failure, timeout, or cancellation status, the status management table advances to the next expected operation sequence number and re-checks subsequent pending tasks. If the failure, timeout, or cancellation of the preceding task makes it impossible to satisfy the state prerequisites of the subsequent task, the system generates a dependency failure or dependency cancellation result for the subsequent task, removes it from the pending task table, releases the C_task capacity, releases callback_id and buffer_handle through the callback cleanup path, and then continues processing the next operation sequence number to avoid pending tasks permanently occupying capacity and resources.

[0127] Duplicate load requests for the same key-value pair can be merged into a single in-transit load task and registered with multiple waiting parties. A load task must not override an unfinished unload task of the same version. Upon returning the completion result, the key-value pair, version number, and operation sequence number are verified; late results with mismatches only trigger resource release and do not update the cache state.

[0128] When an inference request depends on multiple layers or multiple key-value blocks, the key-value management and callback registration module 102 only notifies the inference scheduling module 101 to continue execution after all required data has been successfully verified and marked as available. If loading fails, processing can continue according to cache miss, recalculation, or other preset rollback methods.

[0129] See Figure 4 After task submission (401), dependency checks (402) are performed. Tasks with satisfied dependencies are written to a unique priority task queue (403); tasks with unsatisfied dependencies are registered in the dependency record (406) of the status management table. Complete tasks are maintained in a pending task table indexed by task_id, and are checked again after the preceding operations are completed. Dependency record (406) and the pending task table are not additional sub-queues for worker threads to retrieve tasks. The worker thread pool (404) retrieves the highest-priority task currently scheduled from the unique priority task queue (403) and accesses the SSD (405).

[0130] The unique priority task queue 403 can use a max-heap, an indexed heap, or other data structure that allows retrieval of the highest-priority element by the scheduling key. Queue access is protected by synchronization mechanisms such as mutexes and condition variables; when the queue is empty, worker threads wait for new tasks or close notifications.

[0131] The priority task queue controls the order in which worker threads acquire or submit tasks to the asynchronous I / O interface. Once a low-priority task begins execution or is submitted to the SSD, it is not automatically paused or canceled by subsequent high-priority tasks; the SSD controller can also schedule requests according to its own strategy. Therefore, the priority scheduling in this application is a non-preemptive scheduling of pending tasks.

[0132] To prevent low-priority tasks from remaining unprocessed for extended periods, you can set an aging increment for queuing time or set minimum service quotas for different task types. When memory pressure is high, you can also increase the scheduling priority of unloading tasks, preventing tasks releasing memory from being permanently blocked by continuously arriving loading tasks.

[0133] In one implementation of a synchronous thread pool, after a worker thread retrieves a task from the unique priority task queue 403, it calls `pread` or `pwrite` to perform blocking read / write operations. The Python task submission thread can return immediately after the task is enqueued, thus the read / write operation is performed asynchronously relative to the Python business layer. The worker thread handles short reads, short writes, and interrupted by signals, generating a callback descriptor after completing the expected number of bytes or acknowledging failure.

[0134] In one asynchronous I / O implementation, the C++ I / O execution module uses io_uring or POSIX AIO. The submitting thread constructs an asynchronous request based on the task description, writes the task_id to the request-associated field, and saves the task status and buffer_handle in the in-transit task table. The completion thread reads the completion event, checks the actual number of bytes completed and the error code based on the task_id, and then generates a callback description. The submitting thread and the completion thread may not be the same thread, and neither accesses Python objects.

[0135] The two I / O implementations described above are alternative implementations. In either implementation, the thread performing SSD I / O operations, handling native completion events, and constructing callback descriptors does not access Python objects or call Python / C APIs.

[0136] The system sets a preset capacity C_task for the total number of tasks waiting for dependencies in the pending task table, tasks in the unique priority task queue, and in-transit I / O tasks, and sets a preset capacity C_callback for the C++ callback queue in the callback execution module 106. When the total number of the aforementioned three types of tasks reaches C_task, the Python business layer can wait for available capacity within a preset submission timeout, or receive a status indicating that the task is full and postpone submission, without silently discarding the task. If a callback has been registered but the task has not been registered or enqueued due to full capacity, submission timeout, or closure, the submission thread, while holding the GIL, unregisters the callback_id and releases the buffer owner.

[0137] When the C++ callback queue reaches its capacity limit, the I / O completion thread waits for the callback execution thread in callback execution module 106 to release the queue space, while keeping the completion results and related buffers valid. Simultaneously, the system suspends receiving new I / O tasks, allowing backpressure to propagate along the task submission direction. After the callback execution thread creates a batch snapshot, it releases the queue lock and waits for the Global Interpreter Lock (GIL) without holding the lock.

[0138] See Figure 5 The callback batch processing includes waiting for the callback queue 501, reading the earliest enqueue time 502, judging whether the quantity threshold N or time threshold T is met 503, forming a batch snapshot 504, obtaining the GIL and executing the callbacks in sequence 505, and releasing the GIL and starting the next batch 506.

[0139] When the C++ callback queue is empty, the callback execution thread waits for a non-empty notification, and the time threshold is not activated. After the first callback descriptor is enqueued, its enqueue time is obtained and recorded using a monotonic clock, serving as the current waiting starting point. Both the enqueue time and the current time are obtained from the same monotonic clock to avoid system clock adjustments affecting threshold judgment. The waiting starting point does not use the completion time of the previous batch execution.

[0140] If the number of pending callback descriptors reaches N, the callback execution thread, protected by the queue synchronization mechanism, removes N descriptors from the head of the queue at once to form the current batch. If the waiting time of the earliest pending descriptor reaches T, the callback execution thread removes no more than N pending descriptors from the head of the queue to form the current batch. N is a positive integer greater than 1, and T is a positive integer in microseconds; if the configuration value is expressed in milliseconds, it is first converted to microseconds.

[0141] After the current batch is formed, the callback execution thread releases the queue synchronization mechanism and then acquires the GIL. While waiting for the GIL and executing the current batch, worker threads or completion threads that process I / O completion results can still submit new descriptors to the C++ callback queue; the new descriptors are left for processing in the next batch, and the next batch uses the enqueue time of the earliest descriptor to be processed at that time as the starting point for waiting.

[0142] After the callback execution thread acquires the GIL, it parses the `callback_id` according to the order in which the descriptors entered the C++ callback queue and executes the corresponding Python callback functions. This order represents the order in which the results are enqueued and observed, and does not guarantee a strict chronological order of completion within the SSD. When business dependencies need to be maintained, verification is performed using key-value version numbers and operation sequence numbers.

[0143] Python callback functions are executed directly by the callback execution thread within the GIL holding section of the current batch. They only perform necessary memory state updates, completion marking, and wake-up notifications, without performing synchronous memory I / O. The callback execution thread cleans up the callback registration entries and buffer owners before releasing the GIL of the current batch.

[0144] If M callbacks are processed cumulatively, forming J non-empty batches, then the number of times the GIL is explicitly retrieved is J. This number is approximately 1 / N of the callback-by-call method only when each batch contains N callbacks. The time threshold T constrains the batch aggregation waiting time; the actual callback completion time also includes the time required for thread scheduling, waiting for the GIL, and executing preceding callbacks.

[0145] I / O completion results should include at least success, failure, timeout, and canceled statuses, and record the actual number of bytes completed and the error code. For short reads, short writes, or recoverable transient errors, the system will re-execute only if the operation is idempotent, following a preset retry count R. After the retry count is exceeded, the task result will be marked as an error; loading or deleting a task will roll back to stored (305), and unloading a task will roll back to available (303), preventing key-value entries from remaining in a transitional state for an extended period.

[0146] When a callback function throws an exception, the callback execution thread records the callback_id, task_id, and exception state. Based on the task type, it removes the key-value entry from the loading, storing, or deleting state and clears or normalizes the Python interpreter's exception state. It then continues processing other callbacks in the same batch, releasing the corresponding registry entries and buffer references in the cleanup path. The failure of a single callback does not result in the loss of other completed results in the current batch.

[0147] For requests that are cancelled or completed late, the system determines whether to update the status based on the key-value version number, generation, and operation sequence number. Logical cancellation means the original request will not be woken up again, but does not indicate that the underlying I / O has terminated. After issuing a cancellation request for asynchronous I / O, the system continues to maintain the buffer_handle and its Python owner until it receives a completion event for the original request or a clear cancellation completion result, confirming that the I / O backend will no longer access the buffer. Completion results deemed expired only trigger resource cleanup and do not wake up cancelled requests.

[0148] The C++ I / O execution module has three shutdown states: running, empty, and stopped. The Python shutdown thread, while holding the Global Interpreter Lock (GIL), first sets the module from the running state to the empty state, rejects new tasks, iterates through the pending task table and the unique priority task queue, generates cancellation results for tasks that have not yet been executed, and cleans up the corresponding callback_id and buffer owner. For submitted asynchronous I / O, after issuing a cancellation request, it waits for the original request completion event or the cancellation completion result, and only releases the buffer_handle after confirming that the I / O backend will no longer access the buffer.

[0149] After setting the empty state, the Python shutdown thread releases the GIL in the waiting section where it does not access Python objects, allowing the callback execution thread to acquire the GIL, process completed results, and release the callback registration item and Python buffer owner. After the pending task table, in-transit tasks, callback descriptors, and local batches are all cleared, the shutdown thread waits for and joins the I / O worker threads, completion threads, and callback execution threads without holding the GIL, and then reacquires the GIL before returning to the Python business layer. Only after the callback execution thread has completely exited is the Python interpreter allowed to enter the finalization phase.

[0150] See Figure 6 The cache processing device includes a task creation unit 601, a submission unit 602, a C++ I / O execution unit 603, a callback queue unit 604, a batch control unit 605, and a GIL callback unit 606.

[0151] The task creation unit 601, located in the Python business layer, is used to register Python callback functions while holding the GIL (Global Interpreter Lock) and maintain references to these functions until the callback is completed, the task terminates, or the submission fails, when it detects that a key-value pair (KV) cache needs to be loaded from or unloaded from the SSD. It also obtains the corresponding callback identifier and generates a C++ I / O task description item that does not contain Python objects. The C++ I / O task description item includes at least the task type, KV identifier, SSD storage path, data length, memory buffer address or memory buffer handle, and callback identifier. The task creation unit 601 is also used to establish a mapping between the owner of the memory buffer and the memory buffer address or handle, and maintain a reference to the owner until it is confirmed that the I / O backend will no longer access the memory buffer. Specifically, this task creation unit 601, located in the Python business layer, is used to register Python callback functions while holding the GIL, maintain references to them, establish a mapping between the buffer owner and the buffer address or buffer_handle, maintain a reference to the owner until it is confirmed that the I / O backend will no longer access the buffer, and generate a C++ I / O task description item that does not contain Python objects.

[0152] Submission unit 602 is used to submit C++ I / O task descriptions to the task receiving interface of the C++ I / O execution unit via the pybind11 interface, and return to the Python business layer before the completion of the current I / O task. Specifically, the Python thread calling the pybind11 interface only releases the GIL during the execution of C++ code sections that do not access Python objects, and reacquires the GIL before returning to the Python business layer. More specifically, submission unit 602 is used to submit task descriptions to the task receiving interface of the C++ I / O execution unit 603 via the pybind11 interface, ensuring that Python object operations only occur in code sections holding the GIL.

[0153] The C++ I / O execution unit 603 includes a task receiving interface, a task queue, one or more worker threads, and an optional completion thread. It is used to have the task receiving interface write C++ I / O task descriptions that meet the enqueueing conditions into the task queue; worker threads retrieve C++ I / O task descriptions from the task queue; and, without holding the GIL and accessing Python objects or calling Python / C APIs, perform synchronous SSD I / O operations or submit asynchronous SSD I / O requests based on the C++ I / O task descriptions. The worker thread or completion thread that processes the completion result constructs a callback description that does not contain a Python object but includes at least a callback identifier and task status, and pushes the callback description into the C++ callback queue. Specifically, the C++ I / O execution unit 603 is used to perform synchronous SSD I / O operations or submit asynchronous SSD I / O requests without holding the GIL and accessing Python objects, and the worker thread or completion thread that processes the completion result constructs a callback description that does not contain a Python object and pushes the callback description into the callback queue unit 604.

[0154] The callback queue unit 604 includes a C++ callback queue and a callback execution thread independent of the worker thread and the completion thread. It receives callback descriptions pushed into the C++ callback queue and, when the number of pending callback descriptions in the C++ callback queue reaches a preset threshold greater than one, or when the C++ callback queue is not empty and the waiting time of the earliest enqueued pending callback description reaches a preset time threshold in microseconds, removes one or more callback descriptions from the head of the C++ callback queue at once to form the current callback batch without holding the GIL. After forming the current callback batch, it acquires the GIL, retrieves and executes the corresponding Python callback function based on the callback identifier in the current callback batch, and completes callback registration cleanup after callback completion or task termination, before releasing the GIL. Specifically, the callback queue unit 604 receives callback descriptions and records their enqueue order and enqueue time. The batch control unit 605 forms a current batch of no more than N descriptions when the quantity or time threshold is met, and releases the queue synchronization mechanism before waiting for the GIL.

[0155] GIL callback unit 606 is used to obtain the GIL after the current batch is formed, retrieve the Python callback function based on callback_id, perform state updates, notifications and resource cleanup, and release the GIL uniformly after the batch processing is completed.

[0156] The units in the device can be implemented by software modules executed by a processor, or by a combination of software and dedicated hardware. The division of units is used to describe functional relationships and does not require corresponding independent physical components.

[0157] See Figure 7 The electronic device may be a server or computing node for deploying large language model inference services, including a processor 701, a memory 702, a computer program 703, a graphics processor 704, a system bus 705, an SSD 706, and a network interface 707.

[0158] The processor 701 executes the computer program 703 in the memory 702, implementing task creation, native task scheduling, I / O completion processing, and batch callbacks. The graphics processor 704 is used to perform large language model inference calculations, the SSD 706 is used to store unloaded KV cache data, and the network interface 707 is used to receive inference requests or exchange data with other nodes.

[0159] Processor 701, memory 702, graphics processor 704, SSD 706, and network interface 707 communicate via system bus 705 or a corresponding interconnect structure. The electronic device may include one or more processors, one or more SSDs, and other storage tiers; this application does not limit the specific number of hardware components or the type of bus.

[0160] Computer-readable storage media may include read-only memory, random access memory, solid-state drive, magnetic disk, optical disk, or other non-transitory media capable of storing computer programs. When the computer program is executed by a processor, it causes the electronic device to implement any of the aforementioned method implementations.

[0161] The technical features described above can be combined without contradiction. For synchronization mechanisms, storage formats, and inference framework interfaces not specifically described, equivalent methods that enable those skilled in the art to achieve the same functionality can be used.

Claims

1. A caching method for large language models, characterized in that, Applied to a process that has enabled the Global Interpreter Lock (GIL), the process having a service layer and a transport layer, the method includes: When the business layer detects that it needs to load the key-value pair (KV) cache from the solid-state drive (SSD) or unload the KV cache to the SSD, while holding the GIL, it registers the callback function corresponding to this I / O task and obtains the corresponding callback identifier, and generates an I / O task description item that does not contain Python objects; Submit the I / O task description to the transport layer, and release the GIL after submission; The transport layer's worker thread executes the SSD's I / O requests based on the I / O task description without holding the GIL; After the SSD's I / O request is completed, the transport layer constructs a callback description that does not contain a Python object and pushes the callback description into the callback queue; wherein, the callback description includes at least the callback identifier and the task status of this I / O task; The callback queue is monitored by a callback execution thread independent of the worker thread, and one or more callback description items are removed from the callback queue to form the current callback batch when the batch processing trigger condition is met. Obtain the GIL, execute the corresponding callback functions in batches according to the callback description items in the current callback batch, and release the GIL after execution.

2. The caching method for large language models according to claim 1, characterized in that, The business layer is a Python business layer, and the transport layer is a C++ I / O execution module; The Python business layer communicates with the C++ I / O execution module via the pybind11 interface; The I / O task description item includes at least the task identifier, task type, key-value identifier, SSD storage path, data length, memory buffer address or memory buffer handle, and the callback identifier; The memory buffer remains valid until it is confirmed that the I / O backend will no longer access the memory buffer.

3. The caching method for large language models according to claim 1, characterized in that, The I / O task description item further includes task type and task priority. The task type includes load KV cache task and unload KV cache task. The I / O task description item includes basic priority and scheduling priority. The method further includes: When generating the I / O task description item, the basic priority is determined according to the task type, and the basic priority is used as the initial scheduling priority; wherein, the basic priority of the loading KV cache task is higher than the basic priority of the unloading KV cache task.

4. The caching method for large language models according to claim 3, characterized in that, Before submitting the I / O task description item to the transport layer, the method further includes: Based on the basic priority, the scheduling priority is determined according to at least one of the urgency of the corresponding I / O request and the reuse frequency of the corresponding KV cache; The urgency level is determined at least based on the waiting time or remaining time budget of the I / O request; the reuse frequency is determined based on the number of times the KV cache corresponding to the KV identifier is accessed within a preset statistical window.

5. The caching method for large language models according to claim 4, characterized in that, The method further includes: A priority task queue is maintained within the transport layer; wherein, the worker thread retrieves the highest priority I / O task description from the priority task queue and executes it, and the priority task queue is the only task queue maintained within the transport layer, and there are no additional sub-queues for worker threads to retrieve tasks.

6. The caching method for large language models according to claim 1, characterized in that, The batch processing triggering conditions include: The number of pending callback descriptions in the callback queue reaches a preset threshold N, where N is a positive integer greater than 1; or, The callback queue is not empty, and the waiting time calculated from the enqueue time of the earliest pending callback description item entering the callback queue reaches a preset time threshold T; where T is a positive integer in microseconds.

7. The caching method for large language models according to claim 6, characterized in that, The method further includes: When the number of pending callback description items reaches the preset number threshold N, N callback description items are removed from the callback queue at once to form the current callback batch; When the waiting time reaches the preset time threshold T, the currently pending callback description items, no more than N, are removed from the callback queue at once to form the current callback batch.

8. The caching method for large language models according to claim 1, characterized in that, The method further includes: When executing the callback functions in the current callback batch, they are executed sequentially according to the order in which each callback description item in the current callback batch entered the callback queue; If any callback function encounters an exception, the exception is recorded and the remaining callback functions in the current callback batch are executed.

9. The caching method for large language models according to claim 1, characterized in that, The SSD I / O operation is submitted through an asynchronous I / O interface, or executed through a synchronous I / O interface; In any I / O mode, the thread performing the SSD I / O operation does not hold the GIL and does not access the objects in the business layer.

10. A caching processing device for large language models, characterized in that, Applied to a process that has enabled the Global Interpreter Lock (GIL), the process having a service layer and a transport layer, the device includes: The registration unit is used to register a callback function corresponding to the current I / O task and obtain the corresponding callback identifier when the business layer detects that it is necessary to load the key-value pair KV cache from the solid-state drive SSD or unload the KV cache to the SSD, while holding the GIL; and to generate an I / O task description item that does not contain Python objects. A release unit is used to submit the I / O task description item to the transport layer and release the GIL after submission; An execution unit is configured to execute the I / O request of the SSD according to the I / O task description item through the worker thread of the transport layer without holding the GIL; The construction unit is configured to, after the I / O request of the SSD is completed, construct a callback description item without containing a Python object by the transport layer, and push the callback description item into the callback queue; wherein, the callback description item includes at least the callback identifier and the task status of the current I / O task; The monitoring unit is used to monitor the callback queue through a callback execution thread independent of the worker thread, and remove one or more callback description items from the callback queue to form the current callback batch when the batch processing trigger condition is met; The acquisition unit is used to acquire the GIL, execute the corresponding callback functions in batches according to the callback description items in the current callback batch, and release the GIL after execution.

11. An electronic device, characterized in that, The device includes at least one processor and a memory connected to the at least one processor, the memory storing a computer program that, when executed by the at least one processor, causes the electronic device to implement the caching processing method for large language models as described in any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 9.

Images