Task processing method, computing cluster and computing device

Abstract

The embodiment of the invention provides a task processing method, a computing cluster and computing equipment. The method is applied to an acceleration calculation unit, the acceleration calculation unit comprises a plurality of AI accelerators, the AI accelerators comprise a first accelerator, and the accelerators are connected with a shared memory pool based on a cache coherence protocol. The first accelerator obtains historical intermediate state data corresponding to the target reasoning task from a first address field of the shared memory pool; the first accelerator calculates a new lexical element and first intermediate state data corresponding to the new lexical element based on the historical intermediate state data; the first accelerator stores the first intermediate state data into the first address field. By adopting the method, a plurality of AI accelerators can access the shared memory pool through the shared framework, so that efficient sharing and multiplexing of the intermediate state data among different AI accelerators are realized, and the cache utilization rate and the reasoning efficiency are improved.

Description

Technical Field

[0001] This application relates to the field of large language model technology, and in particular to a task processing method, computing cluster, and computing device. Background Technology

[0002] Large language model inference tasks are widely used in intelligent scenarios such as online customer service and smart assistant business applications, where the accuracy and timeliness of inference results are extremely important. As application scenarios become increasingly complex and model sizes continue to expand, the computing power and memory capacity of a single node are no longer sufficient to meet inference demands. Therefore, a distributed inference architecture is adopted, dividing the large language model and deploying it across multiple computing nodes, so that computational fragments of different or the same layer of the model are distributed across the processing units of different computing nodes.

[0003] However, in collaborative inference across compute nodes, the key-value cache is typically maintained locally by the compute node that generated it. When other nodes need to access this key-value cache to perform subsequent inference tasks, it needs to be migrated to the compute node where the current task resides via a high-speed network. For example, compute node A generates a key-value cache A1 for a certain inference task and stores it in its local memory; when compute node B needs to reuse this key-value cache A1, it needs to migrate key-value cache A1 to compute node B via a high-speed network. This cross-node transfer significantly increases communication overhead, leading to increased latency and reduced inference efficiency. Summary of the Invention

[0004] This application provides a task processing method, a computing cluster, and a computing device. Through a shared framework, multiple AI accelerators can access a shared memory pool, thereby achieving efficient sharing and reuse of intermediate state data among different AI accelerators, improving cache utilization and inference efficiency.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions: In a first aspect, embodiments of this application provide a task processing method applied to an accelerated computing unit, the accelerated computing unit including multiple AI accelerators, the multiple AI accelerators including a first accelerator, the multiple accelerators being connected to a shared memory pool based on a cache coherence protocol, the method including: in response to processing a target inference task, the first accelerator obtaining historical intermediate state data corresponding to the target inference task from a first address segment of the shared memory pool; the first accelerator calculating new lexical units and the first intermediate state data corresponding to the new lexical units based on the historical intermediate state data; the first accelerator storing the first intermediate state data in the first address segment.

[0006] Based on this solution, a shared memory pool is introduced among multiple AI accelerators, and a cache coherence protocol is used to enable collaborative access to shared data, allowing different accelerators to work together to complete inference tasks. In this way, intermediate state data generated during inference (such as key-value cache) is uniformly stored in the shared memory pool, achieving cross-accelerator data sharing and reuse. This avoids redundant transmission and repeated computation of intermediate data, significantly improves cache utilization, reduces communication overhead, and thus improves overall task processing efficiency and response speed.

[0007] In one possible implementation, the method further includes: a first accelerator or a second accelerator calculating intermediate state data based on the input prompt word; the first accelerator or the second accelerator storing the intermediate state data in a first address segment to form historical intermediate state data.

[0008] Based on this scheme, the above process corresponds to the filling stage in large language model inference. In this stage, the model needs to perform a one-time forward computation on the entire input prompt word to generate a complete context representation and corresponding intermediate state data. By uniformly storing this cached data in a designated address segment of the shared memory pool, namely the first address segment, this historical state can be directly read and reused for subsequent decoding stages (regardless of whether it is executed by the same accelerator or another accelerator) without recalculation. This achieves efficient collaboration between filling and decoding across different accelerators, thereby improving overall task processing efficiency and response speed.

[0009] In another possible implementation, the first or second accelerator stores intermediate state data in a first address segment to form historical intermediate state data, including: for the i-th word in the input prompt word, the first or second accelerator determines the first storage location of the i-th word in the first address segment based on the starting physical address of the first address segment and the fixed storage space size occupied by the intermediate state data corresponding to each word; after all words in the input prompt word have been processed, the first or second accelerator updates the number of currently processed words.

[0010] Based on this scheme, a structured and directly addressable shared storage layout is constructed for the filling stage, enabling different AI accelerators to accurately read and write intermediate state data at the granular level of tokens. Since the storage offset corresponding to each token can be directly calculated from the starting physical address of the shared memory segment and a predefined fixed storage step size, efficient parallel writing is supported, and low-overhead random access capability is provided for the subsequent decoding stage. Simultaneously, by maintaining metadata such as the "number of processed tokens" in the shared memory, the system can accurately define the range of valid historical states, ensuring that only legal and complete context information is reused during the decoding stage, avoiding invalid access.

[0011] In another possible implementation, in response to processing the target inference task, the first accelerator obtains historical intermediate state data corresponding to the target inference task from the first address segment of the shared memory pool, including: the first accelerator obtains the position index corresponding to the current word in the target inference task from the meta database in the shared memory pool; the first accelerator or other accelerator obtains historical intermediate state data from the first address segment based on the position index.

[0012] Based on this scheme, when processing the target inference task, the first accelerator queries the meta-database for the position index corresponding to the historical context upon which the current lexical depends, and then directly accesses the corresponding region in the first address segment of the shared memory pool based on the index to obtain the required historical intermediate state data. In this way, by introducing a meta-database into the shared memory pool, a mapping relationship between the logical identifier (position index) and the physical storage location can be established for each lexical in the inference task. This allows the first accelerator to directly exchange data with the shared memory pool without any intermediate copying, achieving true "zero-copy" sharing, thereby improving the overall efficiency and response speed of task processing.

[0013] In another possible implementation, the first accelerator stores the first intermediate state data in the first address segment, including: the first accelerator determines the second storage location of the new word in the first address segment based on the starting physical address and the fixed storage space occupied by the first intermediate state data corresponding to the new word; writes the first intermediate state data generated by the new word into the second storage location; and the first accelerator updates the number of currently processed words.

[0014] Based on this scheme, after generating a new lexical unit, the first accelerator directly calculates the second storage location corresponding to the new lexical unit based on the starting physical address of the first address segment of the shared memory pool and the fixed storage space occupied by the intermediate state data corresponding to each lexical unit. It then writes the newly generated first intermediate state data to this location and updates the "current number of processed lexical units" recorded in the shared memory. In this way, the fixed slot layout achieves efficient and ordered writing of intermediate state data and low-overhead addressing, effectively improving task processing efficiency and response speed.

[0015] In another possible implementation, the method further includes: a second accelerator acquiring a target inference task; the second accelerator acquiring a predicted resource allocation result for processing the target inference task based on the sequence length of the target inference task; the predicted resource allocation result including a first storage space required for processing the target inference task; wherein the first storage space corresponds to a first address segment; and the second accelerator generating a resource allocation request based on the first storage space required for processing the target inference task.

[0016] Based on this scheme, fine-grained memory management at the session level is achieved by pre-determining the predicted resource allocation results based on the target inference task and generating resource allocation requests accordingly. This allows memory resources to be allocated immediately upon session creation and quickly released upon session termination, thus supporting a dynamic and efficient allocation and reclamation mechanism and improving the resource utilization of the shared memory pool.

[0017] In another possible implementation, after the first accelerator or other accelerator generates a resource allocation request based on the storage capacity required for the target inference task, the method further includes: a second accelerator sending a resource allocation request to the management node; wherein the resource allocation request includes a task identifier for the target inference task, a first storage space required to execute the target inference task, and a user identifier for inputting the target inference task; wherein the user identifier is associated with the inference task level, so that the management node prioritizes multiple inference requests from different users based on the inference task level and processes high-priority users first.

[0018] Based on this solution, by embedding multi-dimensional task metadata into resource allocation requests and utilizing the inference task level associated with user identifiers, management nodes can implement differentiated priority scheduling for concurrent inference requests from different users. This enables deep collaboration between resource allocation, task semantics, and users, thereby improving overall resource scheduling efficiency in multi-tenant, high-concurrency scenarios.

[0019] In another possible implementation, after the first accelerator or other accelerator generates a resource allocation request based on the storage capacity required by the target inference task, the method further includes: the second accelerator sending a resource allocation request to the management node; wherein the resource allocation request includes the task identifier of the target inference task, the first storage space required to execute the target inference task, and the service quality level of the target inference task; wherein the service quality level is used to characterize the priority of the target inference task.

[0020] Based on this solution, by including multi-dimensional metadata such as task identifier, required first storage space, and service quality level in the resource allocation request, and combining it with the inference task level determined by the service quality level, the management node can implement fine-grained priority scheduling for concurrent inference requests from different users, effectively improving the overall resource utilization efficiency of the system.

[0021] In another possible implementation, the shared memory pool has multiple shared memory modules, and multiple accelerators and multiple shared memory modules are connected to a switch based on a consistent caching protocol; in response to processing the target inference task, the first accelerator obtains the historical intermediate state data corresponding to the target inference task from the first address segment of the shared memory pool, including: the first accelerator determines the first address segment based on the target inference task; the access request of the first accelerator is routed to the target storage module through the switch to obtain the historical intermediate state data, wherein the first address segment is located within the target storage module.

[0022] Based on this solution, by dividing the shared memory pool into multiple physically distributed shared memory modules and combining switches and cache coherency protocols to build a high-bandwidth, low-latency interconnect architecture, not only is scalable access to large-scale intermediate state data achieved, but also fine-grained storage resource allocation is supported.

[0023] Secondly, embodiments of this application also provide a task inference method applied to a management node. The method includes: the management node receiving a resource allocation request sent by a second accelerator; the management node determining a first address segment based on the resource allocation request and the available capacity of the shared memory pool; wherein the first address segment is a continuous address segment in the shared memory pool, and the first address segment is used to store historical intermediate state data corresponding to the target inference task.

[0024] Based on this scheme, resource allocation requests from accelerators are received and processed uniformly by the management node. A contiguous address space (i.e., the first address segment) is dynamically partitioned based on the real-time available capacity of the shared memory pool to store historical intermediate state data for specific inference tasks. This achieves centralized and refined control over shared memory resources. This improves the accuracy and adaptability of memory allocation, thereby significantly enhancing task execution efficiency and overall system performance.

[0025] In another possible implementation, the following is also included: the management node creates a task record corresponding to the target inference task in the metadata database; the task record stores context information related to the target inference task, including the task identifier of the target inference task, the starting physical address of the target inference task, and the number of lexical units being processed in the target inference task.

[0026] Based on this scheme, after allocating memory space for the inference task, a dedicated record is proactively created and maintained in the metadata database. This record stores key context information such as the task identifier and starting physical address. This improves the response speed for subsequent location and access to intermediate state data, ensuring the efficient operation of the inference task.

[0027] In another possible implementation, the following is also included: the management node receives a release request sent by the first accelerator; the release request includes the task identifier of the target inference task; based on the task identifier of the target inference task, the management node releases the task record corresponding to the target inference task in the metadata database and the historical intermediate state data corresponding to the target inference task in the shared memory pool.

[0028] Based on this scheme, when an inference task is completed or terminated, the accelerator sends a release request containing the task identifier to the management node, which then synchronously releases the resources occupied by that task. In this way, by introducing a task-level resource release mechanism, the coordinated reclamation of shared memory resources and metadata is achieved, ensuring the synchronous cleanup of storage resources and control information, effectively preventing memory leaks and metadata redundancy, and improving the long-term operational stability of the system.

[0029] In another possible implementation, the first address segment is determined based on the resource allocation request and the available capacity of the shared memory pool, including: determining the first address segment based on the resource allocation request, wherein the first address segment is a contiguous address space in the shared memory pool; and sending the target resource allocation result of the target inference task to the second accelerator based on the first address segment.

[0030] Based on this scheme, upon receiving a resource allocation request, the management node first allocates a contiguous address space from the available capacity of the shared memory pool as the first address segment, according to the storage requirements specified in the request (such as the required initial storage space size). This satisfies the target inference task's storage requirements for historical intermediate state data. In this way, contiguous address allocation ensures efficient and low-overhead access to the intermediate state data by subsequent accelerators. Subsequently, the information in this first address segment is encapsulated into the target resource allocation result and returned to the second accelerator to guide its subsequent data writing and inference execution.

[0031] Thirdly, embodiments of this application also provide a computing device, including: a processor and a memory; the processor and the memory are coupled; the memory is used to store program instructions; the processor is used to execute the program instructions to perform the method as described in any of the first aspects above.

[0032] Fourthly, embodiments of this application provide a chip for performing the methods described in any of the first aspects above.

[0033] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a computer, implement the method as described in any of the first aspects.

[0034] In a sixth aspect, embodiments of this application provide a program product including a computer program that, when executed by a processor, implements the method as described in any of the first aspects. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of a distributed inference architecture for a task processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the framework of a task processing method provided in an embodiment of this application; Figure 3 This is provided by the embodiments of this application. Figure 2 A schematic diagram of a computing cluster within the framework shown; Figure 4 This is provided by the embodiments of this application. Figure 2 A schematic diagram of a computing device with the frame shown; Figure 5 This is a first flowchart illustrating a task processing method provided in an embodiment of this application; Figure 6 This is a second flowchart illustrating a task processing method provided in an embodiment of this application; Figure 7 This is a schematic diagram of the signaling flow of a task processing method provided in an embodiment of this application; Figure 8 This is a schematic diagram of an interface for obtaining a target reasoning task provided in an embodiment of this application; Figure 9 This is a third flowchart illustrating a task reasoning method provided in an embodiment of this application; Figure 10 This is a signaling flow diagram of another task processing method provided in an embodiment of this application; Figure 11 This is a fourth flowchart illustrating a task reasoning method provided in an embodiment of this application; Figure 12 This is the fifth flowchart of a task reasoning method provided in an embodiment of this application; Figure 13 This is the sixth flowchart of a task reasoning method provided in an embodiment of this application; Figure 14 This is a schematic diagram of a process for establishing a communication connection between a shared memory pool and a target processing unit, provided in an embodiment of this application. Figure 15 This is a schematic diagram of a task reasoning device provided in an embodiment of this application; Figure 16 This is a schematic diagram of another task reasoning device provided in an embodiment of this application; Figure 17This is a schematic diagram of another task reasoning device provided in the embodiments of this application; Figure 18 This is a schematic diagram of a computing device provided in an embodiment of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. To facilitate a clear description of the technical solutions of the embodiments of this application, the use of terms such as "first," "second," etc., in the embodiments of this application is for illustrative purposes and to distinguish the objects being described. There is no particular order between them, nor does it indicate a specific limitation on the number of devices in the embodiments of this application, and they do not constitute any limitation on the embodiments of this application.

[0037] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of this application.

[0038] It should be noted that many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below.

[0039] The following explanations of the technical terms mentioned in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0040] Large language models (LLMs) are neural network models based on deep learning techniques with a large number of parameters (usually hundreds of millions to hundreds of billions or even more). By pre-training on massive amounts of text data, they have the ability to understand and generate natural language and can be applied to various natural language processing and target reasoning tasks such as text generation, question answering, translation, and summarization.

[0041] Key-value cache (KV cache) refers to a mechanism within a large language model where each input token generates a corresponding key and value vector during the attention process. To avoid repeatedly calculating the attention results of historical tokens when generating each new token, the large language model caches the keys and values ​​of already processed tokens, forming a key-value cache.

[0042] Compute Express Link (CXL) is a high-performance, low-latency open interconnect technology primarily used to improve communication efficiency between processors and components such as accelerators and memory expansion devices.

[0043] CXL shared memory is a memory sharing mechanism based on the Compute Fast Interconnect protocol that supports efficient, low-latency, and cache-consistent sharing across nodes. In this embodiment, a shared memory pool is used as an example for illustration.

[0044] The embodiments of this application will now be described with reference to the accompanying drawings.

[0045] In large-scale model inference scenarios, due to the sheer size of the model, a single computing node often struggles to handle its computational and storage demands. Therefore, a distributed inference architecture is commonly employed, where the large language model is broken down and deployed across a computing cluster composed of multiple computing devices. Through task decomposition and parallel processing, the overall throughput and response efficiency of the system are improved.

[0046] The following is based on Figure 1 We will use an example to illustrate the distributed inference structure.

[0047] Figure 1 This is a schematic diagram of a distributed inference architecture for a task processing method provided in an embodiment of this application.

[0048] like Figure 1 As shown, a general distributed inference architecture is implemented by deploying a large language model on three computing devices (Node 1, Node 2, and Node 3). This is achieved by dividing the large language model into three main logical parts and deploying them on different computing nodes. The computing devices exchange necessary data through efficient network communication protocols to ensure the smooth operation of the entire inference process.

[0049] Figure 2 This is a schematic diagram of the framework of a task processing method provided in an embodiment of this application.

[0050] like Figure 2 As shown in the embodiments of this application, a task processing method framework based on a distributed architecture is provided. This framework includes a management node, a computing resource pool (accelerated computing units), a CXL switch, and a shared memory pool. The components in this framework achieve efficient interconnection and data communication through the CXL switch; that is, multiple AI accelerators and multiple shared memory modules are connected to the switch based on a consistent caching protocol.

[0051] The computing resource pool (accelerated computing unit) consists of multiple AI accelerators supporting a consistent caching protocol. Each AI accelerator can participate in different inference tasks as an independent computing unit. Multiple AI accelerators can be deployed on the same computing device or on different computing devices, such as the first accelerator deployed on computing device X and the second accelerator deployed on computing device Y. Optionally, the AI ​​accelerator includes any of the following: graphics processing unit (GPU), tensor processing unit (TPU), neural processing unit (NPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or other hardware acceleration units supporting deep learning computing.

[0052] Each AI accelerator can directly access multiple shared memory modules in the shared memory pool, enabling unified read and write operations on the shared storage space. The shared memory pool consists of multiple shared memory modules, forming a globally visible, high-throughput, low-latency shared memory pool. Each shared memory module can be allocated to store historical intermediate state data for a single inference task, specifically carrying the intermediate state data generated during inference—the key-value cache in the attention mechanism. In other words, accelerator access requests are routed to the target storage module via a switch to retrieve historical intermediate state data.

[0053] The management node acts as the global manager in this framework, responsible for key functions such as resource configuration, task scheduling, and monitoring of the entire system. Its deployment is highly flexible: it can run on a standalone computing device, such as computing device Z, utilizing the CPU resources of that device to execute scheduling logic and system management tasks; or it can be integrated into the onboard controller of the CXL switch, leveraging the switch's hardware capabilities to directly optimize data paths and communication efficiency.

[0054] When executing the target inference task, the second accelerator generates a resource allocation request based on the target inference task and sends it to the management node. The management node determines the resource allocation result based on the resource allocation request and the available capacity of the shared memory pool to ensure that intermediate state data during the inference process can be effectively stored. The second accelerator calculates intermediate state data based on the input prompt words and stores the intermediate state data in the first address segment to form historical intermediate state data, which can be reused in the subsequent decoding stage. Further, the first accelerator obtains the historical intermediate state data corresponding to the target inference task from the first address segment of the shared memory pool; based on the historical intermediate state data, the first accelerator calculates new lexical units and the first intermediate state data corresponding to the new lexical units; the first accelerator stores the first intermediate state data in the first address segment.

[0055] In summary, the shared framework enables multiple AI accelerators to access the shared memory pool, thereby achieving efficient sharing and reuse of intermediate state data among different AI accelerators, improving cache utilization and inference efficiency.

[0056] The above Figure 2 The framework shown offers flexible deployment options, allowing for both overall deployment at the computing device level and fine-grained deployment at the hardware module level. The following section will combine these options with... Figure 3 and Figure 4 This section provides a detailed explanation of these two typical deployment patterns.

[0057] Figure 3 This is provided by the embodiments of this application. Figure 2 The diagram illustrates a computing cluster within the framework shown. Figure 3 As shown, the computing cluster 300 includes multiple computing devices 301, a shared switch 302, and a shared memory pool 303. Specifically, the shared switch 302 is connected to each computing device 301 (first computing device X, second computing device Y, and third computing device Z) and the shared memory pool 303 to ensure efficient interconnection and data communication. Each computing device 301 includes multiple AI accelerators; for example, the first accelerator is assumed to be deployed on the second computing device Y, and the second accelerator is assumed to be deployed on the third computing device Z. Each computing device 301 also includes a management node, for example, the management node is assumed to be deployed on the first computing device X. In addition, a shared memory pool 303 exists, consisting of multiple shared memory modules forming a globally visible, high-throughput, low-latency shared memory pool specifically used to store intermediate state data generated during inference. In this deployment configuration, each computing device 301 accesses the memory access nodes in the shared memory pool 303 through the shared switch 302, thereby achieving unified read and write access to the global shared storage space.

[0058] Figure 4This is provided by the embodiments of this application. Figure 2 The diagram shows a computing device with the frame shown.

[0059] like Figure 4 As shown, the computing device 400 integrates a management node 401, an accelerated computing unit 402, and a shared memory pool 403. The accelerated computing unit 402 includes multiple AI accelerators, including a first accelerator 4021 and a second accelerator 4022. The shared memory pool 403 is connected to the multiple AI accelerators via a cache coherence protocol. The first accelerator 4021 obtains historical intermediate state data corresponding to the target inference task from a first address segment of the shared memory pool, and calculates new lexical units and their corresponding first intermediate state data based on this historical intermediate state data, storing the first intermediate state data in the first address segment. The second accelerator 4021 calculates intermediate state data based on the input prompt word and stores it in the first address segment to form historical intermediate state data. The management node 401 determines the first address segment based on resource allocation requests and the available capacity of the shared memory pool; the first address segment is a continuous address segment within the shared memory pool, used to store the historical intermediate state data corresponding to the target inference task.

[0060] It should be noted that, Figure 3 The illustration only shows a scenario where the management node is deployed on a separate computing device (e.g., computing device Z), and does not show its deployment within the onboard controller of the CXL switch. In this example, the management node relies on the local CPU resources of computing device Z to perform global management functions such as system-level resource configuration, task scheduling, and runtime status monitoring. Although not shown in the figure, according to the design of this embodiment, the management node can also be flexibly integrated into the controller of the CXL switch, thereby directly participating in data path optimization and communication scheduling using the switch hardware.

[0061] The specific implementation process of the task processing method is explained in detail below with reference to the accompanying drawings.

[0062] Figure 5 This is a first flowchart illustrating a task processing method provided in an embodiment of this application.

[0063] like Figure 5 As shown, applied to Figure 2 The task processing framework shown includes a management node, a first accelerator, a second accelerator, a CXL switch, and a shared memory pool. The task processing method includes the following steps: S1: The second accelerator acquires the target inference task.

[0064] The target inference task refers to the specific task selected for execution by the second accelerator (such as a GPU) from one or more raw task requests awaiting inference processing received in the initial stage, after priority filtering. These task requests are usually initiated by users and include natural language tasks that require the participation of a large language model (LLM), such as question answering, text generation, and summary extraction.

[0065] Figure 6 This is a second flowchart illustrating a task processing method provided in an embodiment of this application.

[0066] like Figure 6 As shown, step S1 includes steps S11-S12.

[0067] S11: The terminal device responds to the user's input operation and obtains the target inference task.

[0068] Figure 7 This is a schematic diagram of the signaling flow of a task processing method provided in an embodiment of this application.

[0069] like Figure 7 As shown, step S11 includes S1001-S1002.

[0070] S1001: The UI module of the terminal device responds to the user's input operation and obtains the target inference task.

[0071] Figure 8 This is a schematic diagram of an interface for obtaining a target reasoning task provided in an embodiment of this application.

[0072] like Figure 3 and Figure 8 As shown in (a), the UI module of the terminal device responds to the user's input operation (such as entering "locallhost / ........html") when opening the inference interface in browser page 1, and displays as shown in (a). Figure 8 The first interface 8 is shown in (b) of the diagram. The first interface 8 includes an input box 81 for the user to input a reasoning request, i.e., to submit a target reasoning task.

[0073] For example, the UI module of the terminal device responds to the target reasoning task A of “continuing the writing of “Ilove”” entered by the user in the input box 21 of the first interface 2.

[0074] S12: The terminal device sends the target inference task to the second accelerator.

[0075] Continue to combine Figure 7 As shown, step S12 includes step S1002.

[0076] S1002: The UI module of the terminal device sends the target inference task to the second accelerator.

[0077] Following the example above, the UI module sends the target inference task A to the first computing unit.

[0078] S2: The second accelerator determines the predicted resource allocation result of the target inference task based on the sequence length of the target inference task.

[0079] The predicted resource allocation result is used to characterize the first storage space required to execute the target inference task. The first storage space corresponds to the first address segment, which is a contiguous address segment within the shared memory pool. This first address segment is used to store historical intermediate state data corresponding to the target inference task. The following continues... Figure 4 Further explanation is provided regarding the prediction resource allocation results for the specific acquisition target inference task.

[0080] In one example, continue combining Figure 7 As shown, step S2 includes step S1003.

[0081] S1003: The second accelerator determines the first storage space for the target inference task based on the sequence length of the target inference task.

[0082] For example, the first storage space for the target reasoning task can be determined by combining formula (1): Formula (1) in, The total context length for the target reasoning task includes the number of input cue words and the expected maximum number of output words. This is the fixed storage space occupied by the intermediate state data corresponding to the word element.

[0083] Optionally, the expected maximum number of output tokens is determined based on at least one of the following: the maximum generation length specified by the user when initiating the inference request; the maximum generation length corresponding to the default generation limit preset by the user or this type of task; and the configuration constraints of the inference engine (e.g., low-latency mode limits the output to no more than 64 tokens, and long text mode allows a maximum of 2048 tokens). It should be noted that, in the case of multiple factors mentioned above, the expected maximum number of output tokens can be determined based on a priority strategy or a minimum value principle, which will not be elaborated upon here.

[0084] Continuing with the above example, let's take the target reasoning task A, which is to "continue writing 'I love'", as an example. Assume that the length of its input sequence is 2 (i.e., it contains two words: "I" and "love"), and the expected maximum number of output words is 3. Based on the above formula (1), the second accelerator calculates and determines the first storage space A1 required for task A.

[0085] After the second accelerator determines the first storage space required to process the target inference task, it executes step S1004.

[0086] S1004: The second accelerator generates a resource allocation request based on the first storage space required to process the target inference task.

[0087] It should be noted that the resource allocation request includes not only the initial storage space required to process the target inference task, but also resource information corresponding to the target inference task. This information assists the management node in effectively scheduling and allocating resources when receiving multiple accelerator requests simultaneously. Specifically, the resource information includes a task identifier (Sequence_ID) and priority information for the target inference task. The task identifier (Sequence_ID) uniquely identifies the input token sequence corresponding to a target inference task, ensuring that the intermediate state data resources of different target inference tasks are isolated from each other. The priority information is associated with the inference task level, enabling the management node to prioritize tasks based on their levels and process higher-priority users first.

[0088] Optionally, priority information may be presented in one of the following forms: User identifiers, such as tenant IDs or user IDs, are used to associate inference task levels, enabling management nodes to prioritize multiple inference requests from different users based on inference task levels and process higher-priority users first. For example, VIP users have higher levels than non-VIP users.

[0089] The Quality of Service (QoS) level is a scheduling priority policy identifier mapped from the inference task level associated with the user identifier. It is used to characterize the resource contention priority of the inference task in a multi-tenant concurrent environment. Specifically, based on the user identifier, the corresponding inference task level is pre-configured or dynamically queried, and the inference engine type specified by the task (such as low-latency mode, high-precision mode, or long-context mode) is comprehensively considered to jointly map the two into a specific QoS level.

[0090] S1005: The second accelerator sends a resource allocation request to the management node.

[0091] The management node may include a KV Cache manager, which can be a hardware module or a software program, such as a dedicated circuit, FPGA, ASIC, or deployed as a software service running on the CPU. The specific implementation form is not limited and can be flexibly selected according to system performance requirements and hardware platform.

[0092] Continuing with the example above, the second accelerator generates a resource allocation request A3 based on the first storage space A1 and resource information A2, and sends the resource allocation request to the management node.

[0093] S3: The management node determines the resource allocation result for the target inference task based on the resource allocation request and the available capacity of the shared memory pool.

[0094] The resource allocation results for the target reasoning task include the first address segment.

[0095] Continue to combine Figure 6 As shown, step S3 includes step S31.

[0096] S31: The management unit verifies the matching relationship of the preset verification rules of the resource information in the resource allocation request based on the resource allocation request.

[0097] Continue to combine Figure 6 As shown, step S31 includes steps S311-S312.

[0098] S311: The management node verifies the matching relationship between the parameter information in the resource allocation request and the preset verification rules.

[0099] Continue to combine Figure 7 As shown, step S311 includes step S1006.

[0100] S1006: The management node verifies the first storage space and resource information and their matching relationship with the preset verification rules.

[0101] The management node performs compliance checks on the primary storage space, task identifier, and priority information to determine if they conform to preset verification rules. These preset verification rules ensure the rationality, security, and quality of service in resource allocation. If a task is marked as a "high-priority session" (e.g., from a member user), but the overall resource quota of its tenant is low, policy restrictions may be triggered. If the requested resource amount exceeds the current system's capacity (e.g., "high session level, but the system cannot provide the required memory"), the verification will fail.

[0102] Continuing with the example above, let's assume the target reasoning task is the user. A Input, if user A For non-member users, Sequence_ID A The first storage space has been determined. The management node will check whether the request meets all preset policies, such as the maximum memory quota for non-member users and the limit on the number of concurrent tasks.

[0103] S312: In the case of a perfect match, the management node determines the first address segment from the shared memory pool based on the resource allocation request and the available capacity of the shared memory pool.

[0104] The first address segment is a contiguous address space in the shared memory pool, used to store historical intermediate state data corresponding to the target inference task.

[0105] Continue to combine Figure 7 As shown, step S312 includes step S1007.

[0106] S1007: In the case of a complete match, the management unit determines the first address segment from the shared memory pool.

[0107] The first address segment is determined by finding a contiguous physical address space in the shared memory pool that meets the capacity requirements. Specifically, the management node addresses a contiguous free region with a starting physical address of Saddr and a size of size within the available address range of the shared memory pool, and allocates this region as the first address segment for the target inference task to store its corresponding historical intermediate state data. Continuing with the example above, after successful verification, the management node finds a region with a starting physical address of 0x8000_0000 and a capacity of 128MB within the available address range of the shared memory pool, and uses this address and capacity as the allocation result (i.e., the first address segment) for subsequent writing of intermediate state data and cross-device sharing.

[0108] After resource allocation is completed, in order to support state tracking, cache access and task management in the subsequent inference process, the management node also needs to establish metadata records and runtime contexts corresponding to the inference task in the system. The following will continue to explain in conjunction with step S1008.

[0109] S1008: The management node creates a task record corresponding to the target inference task in the metadata database.

[0110] The management node first creates a structured metadata record in the metadata database. This metadata database can be deployed in a shared memory pool or on the computing device where the management node resides. At the core of the metadata database is a globally visible metadata table (as shown in Table 1), which centrally stores key information about all active or completed inference tasks.

[0111] Task records are used to store context information related to the target inference task. Each record corresponds to a unique inference task and includes the following fields: task identifier (such as Sequence_ID), allocated starting physical address (S_addr), current write position (pos_size), task status (Status), etc.

[0112] S_addr is used to characterize the starting position of the target inference task in the target region of the shared memory pool.

[0113] Status is used to indicate the execution status of the current token during the inference process, such as "waiting", "computing", and "result cached", for scheduling and resource management. Among them, "result cached" can include the active and inactive states corresponding to intermediate state data.

[0114] For ease of explanation, the metadata table is presented in the form of Table 1.

[0115] Table 1 ; S4: The management unit sends the resource allocation results to the second accelerator.

[0116] Continue to combine Figure 7 As shown, step S4 includes step S1009.

[0117] S1009: The management node sends the resource allocation results to the second accelerator.

[0118] It should be noted that while the management node creates the task record corresponding to the target inference task, the AI ​​accelerator (first accelerator or second accelerator) creates and maintains a lightweight session context data structure in local memory. This structure is used to efficiently track key operational states during the execution of the inference task. The context contains the task's unique identifier session_id, the allocated shared memory starting physical address S_addr, and the number of tokens currently written to the intermediate state data pos_size (initially 0, and increasing as the inference process progresses). Its core function is to support the management unit in quickly calculating the write position, identifying the session, and making scheduling decisions, avoiding frequent access to the metadata database in shared memory, thereby reducing control latency and improving system response efficiency.

[0119] S5: Based on the resource allocation results, the second accelerator performs inference calculations on each word in the inference task in sequence during the forward inference phase, and stores the generated intermediate state data in the first address segment.

[0120] The forward inference process is the core computational flow used by deep learning models to generate prediction results during the deployment phase. The model processes input words one by one. When processing each word, each network layer calculates and generates the corresponding key and value cache, i.e., intermediate state data (KV Cache), through a self-attention mechanism to save historical context information.

[0121] Continue to combine Figure 6 As shown, step S5 includes steps S51-S52.

[0122] S51: The second accelerator calculates intermediate state data based on the input prompt words.

[0123] Continue to combine Figure 7 As shown, step S52 includes step S1010.

[0124] S1010: The second accelerator feeds the input prompts into the large model to calculate intermediate state data.

[0125] Specifically, the second accelerator feeds the cue word sequence into the large model, calculates the key and value vectors of each attention head layer by layer, and organizes them into complete intermediate state data (KV Cache) according to the sequence dimension, which serves as intermediate state data.

[0126] S52: The second accelerator stores intermediate state data in the first address segment of the shared memory pool to form historical intermediate state data.

[0127] The shared memory pool is divided into multiple shared memory modules, or memory blocks, each potentially connected to the target processing unit via different CXL link configurations (e.g., link width, hop count). Due to differences in data transmission path length and available bandwidth, the latency of the target processing unit accessing different storage areas varies significantly. Specifically, the access requests from the first accelerator are routed to the target storage module via a switch to obtain historical intermediate state data. To improve the efficiency of writing intermediate state data and overall system performance, the topology and historical access latency information between the AI ​​accelerator and each memory access node can be obtained before the write operation to select the optimal write path, thereby improving cache write efficiency and overall system performance.

[0128] Continue to combine Figure 7 As shown, step S52 includes steps S1011-S1015.

[0129] S1011: The second accelerator acquires historical access latency information and topology relationships.

[0130] Historical access latency information refers to the actual access latency data of the first computing unit to various storage areas in the shared memory pool within a preset historical time period. Topology is used to characterize the connection path between the target processing unit and the shared memory pool in the hardware interconnect structure. For example, as... Figure 2As shown, the first computing unit is connected to the shared memory pool via a CXL switch. In this architecture, all data transmission must be forwarded through the CXL switch, but different memory access nodes may be connected to different ports of the shared memory pool, resulting in different physical path lengths and bandwidth capabilities between them and the first computing unit.

[0131] For example, the second accelerator acquires historical access latency information H and topology relationship K.

[0132] S1012: The second accelerator analyzes the actual response latency and bandwidth utilization of each path based on historical access latency information and topology relationships.

[0133] Continuing with the example above, firstly, based on the topology K, a set of hardware connection paths from the first computing unit to each storage region in the shared memory pool is identified, such as path P1 and path P2. Secondly, for the historical access latency information H, historical access records corresponding to each identified path (such as P1 and P2) are extracted, and path-specific latency data is obtained for each path. (corresponding to P1) and (Corresponding to P2), the actual response latency and bandwidth utilization of each path (path P1 and path P2) are obtained.

[0134] S1013: The second accelerator determines the target path based on the actual response latency and bandwidth utilization.

[0135] Continuing with the example above, the KV Cache Manager determines the target path O based on the actual response latency and bandwidth utilization.

[0136] S1014: The second accelerator, based on the target path, stores the intermediate state data (i.e., key-value cache, KV Cache) generated during the inference process into the first address segment allocated for the target inference task in the shared memory pool.

[0137] Specifically, for the i-th word in the input prompt word, the second accelerator determines the first storage location of the i-th word in the first address segment based on the starting physical address of the first address segment and the fixed storage space occupied by the intermediate state data corresponding to each word.

[0138] For example, the first storage location can be determined by formula (2): Formula (2) The starting physical address assigned to the target inference task. The fixed storage space occupied by the intermediate state data corresponding to a single word.

[0139] After processing the entire prompt word sequence, the second accelerator updates the "current number of processed tokens" (denoted as pos_size) corresponding to the task and writes this value into the corresponding task record in the metadata database to indicate the write offset position of newly generated tokens in the next decoding stage.

[0140] Continuing with the example above, let's take task A as an example. Assume its S_addr is 0x8000_0000 and each word occupies 64KB. Then the physical address of the first word ("I") is 0x8000_0000, and the physical address of the second word ("love") is 0x8000_0000 + 64KB = 0x8001_0000.

[0141] S1015: After completing the processing of the entire prompt word sequence, the second accelerator updates the task record in the metadata database corresponding to the target inference task.

[0142] The task record includes "the number of tokens processed so far".

[0143] To ensure the timeliness of metadata information in the metadata table of the shared memory pool, after writing the recalculated intermediate state data into the shared memory pool, the task records in the metadata database of the shared memory pool are further updated synchronously to ensure that subsequent tasks can correctly query, locate and reuse the cached data.

[0144] The task record includes the position index for each lexical unit, with the position indices being S_addr and... This facilitates subsequent location tracking.

[0145] Continuing with the example above, the first computing unit writes the physical storage addresses corresponding to each term in the target inference task A into the task record of the metadata database: the physical address of the first term "I" (Position=0) is 0x8000_0000, and the physical address of the second term "love" (Position=1) is 0x8001_0000 (i.e., 0x8000_0000+64KB). Before entering the decoding stage, it is necessary to ensure that the prefill stage has been successfully completed and that the generated intermediate state data has been completely written into the shared memory pool. Specifically, after the second accelerator completes the prefill calculation for the target inference task, it sends a status update to the management node, marking the status field of the target inference task in the task record as "Ready for Decoding". Based on this status field, combined with the service quality level of the target inference task and the current resource load, the management node schedules the task to the first accelerator (i.e., the dedicated decoding accelerator) for subsequent processing. The first accelerator only starts the decoding process after receiving the scheduling instruction from the management node, thus ensuring data consistency and the correctness of the execution sequence.

[0146] S6: During the decoding phase, the first accelerator, in response to processing the target inference task, retrieves the historical intermediate state data corresponding to the target inference task from the first address segment of the shared memory pool.

[0147] Continue to combine Figure 6 As shown, step S6 includes steps S61-S62.

[0148] S61: During the decoding phase, the first accelerator, in response to processing the target inference task, retrieves the position index corresponding to the current lexical in the target inference task from the metadata in the shared memory pool.

[0149] Continue to combine Figure 7 As shown, step S61 includes step S1016.

[0150] S1016: During the decoding phase, the first accelerator, in response to the task identifier of the target inference task, retrieves the position index corresponding to the current lexical in the target inference task from the metadata in the shared memory pool.

[0151] The location index includes the starting physical address Saddr and the fixed storage space size occupied by each token. To efficiently identify the task to which a lexical belongs, a two-dimensional "task-position" addressing mechanism is adopted: each lexical is determined by its unique task identifier (such as Sequence_ID) and its position (pos_size) in the task sequence.

[0152] Continuing with the example above, during the decoding phase, the first accelerator queries the corresponding resource context based on the Sequence_ID of the current processing task to obtain the allocated Saddr and the number of currently processed tokens (i.e., pos_size).

[0153] S62: The first accelerator retrieves historical intermediate state data from the first address segment of the shared memory pool based on the location index.

[0154] Continue to combine Figure 7 As shown, step S62 includes step S1017.

[0155] S1017: The first accelerator retrieves historical intermediate state data from the first address segment of the shared memory pool based on the location index.

[0156] Continuing with the example above, the first accelerator, based on the location index, calculates the physical address of the required historical intermediate state data (i.e., key-value cache) within the first address segment of the shared memory pool using formula (1), and directly reads the corresponding intermediate state data. For example, for task A (Sequence_ID = A), if Saddr = 0x8000_0000, =64KB, and the word “love” at position i=1 needs to be read. Then the first accelerator calculates the physical address as 0x8000_0000+1×64KB=0x8001_0000, and reads the corresponding intermediate state data from this address.

[0157] S7: The first accelerator calculates new lexical units and their corresponding first intermediate state data based on historical intermediate state data.

[0158] Continue to combine Figure 7 As shown, step S7 includes step S1018.

[0159] S1018: The first accelerator calculates new lexical units and the corresponding first intermediate state data based on historical intermediate state data.

[0160] It should be noted that during the subsequent reasoning process based on historical intermediate state data and the current input tokens, the second computing unit will also write the newly generated first intermediate state data into the shared memory pool until all reasoning tasks are completed. This process is consistent with the mechanism described in step S6 above, and will not be repeated here.

[0161] It should be noted that if historical intermediate state data is not obtained from the shared memory pool, the first accelerator needs to execute the target inference task to generate intermediate state data for each word, using this as historical intermediate state data. Based on the generated historical intermediate state data and the current input word, subsequent inference continues until all inference tasks are completed. In other words, the first accelerator needs to re-execute the target inference task during the decoding stage to generate intermediate state data for each word, and continue subsequent inference based on the generated cache.

[0162] S8: The first accelerator stores the first intermediate state data into the first address segment.

[0163] Continue to combine Figure 7 As shown, S8 includes steps S1019-S1021.

[0164] S1019: The first accelerator determines the second storage location of the new word in the first address segment based on the fixed storage space occupied by the starting physical address and the intermediate state data corresponding to the new word.

[0165] Alternatively, the second storage location can be determined using formula (3): Formula (3) in, This indicates the position index of the token to be written in its sequence. Its value is equal to the number of tokens that the task has already processed, i.e., the position where the next newly generated token will be written. S_addr is the starting physical address of the first address segment, and per_token_size is the fixed storage space occupied by the intermediate state data corresponding to each token.

[0166] S1020: Write the first intermediate state data corresponding to the newly generated word element into the second storage location.

[0167] The specific details of generating the first intermediate state data in step S1020 can be found in steps S51 and S52 above, and will not be repeated here.

[0168] S1021: The first accelerator updates the task record in the metadata database corresponding to the target inference task.

[0169] The task record includes the number of lexical units that have been processed so far.

[0170] After the write operation is complete, the first accelerator updates the "current number of processed tokens" for the target inference task to reflect the increase in sequence length and provide the correct write offset for the next decoding stage.

[0171] It should be noted that the intermediate state data read during the decoding phase is not limited to the data generated and stored in the first address segment of the current task during the prefill phase. In specific scenarios, such as task reuse, context continuation, or shared inference, the decoding phase can also read intermediate state data belonging to the same user or other tasks that has not yet been released from other address segments to achieve functions such as context inheritance, long sequence concatenation, or multi-task collaborative inference.

[0172] During the decoding phase, step S8 can be repeated until a complete output sequence satisfying the termination condition is generated. In each iteration, the first accelerator generates a new term, writes its corresponding intermediate state data into shared memory, and updates the number of processed terms. Once it is determined that the target inference task has completed all inference processes, the resource reclamation phase begins.

[0173] S9: After completing all inference tasks, the management node releases the relevant data stored in the shared memory pool of the target inference task.

[0174] Continue to combine Figure 7 As shown, step S9 includes steps S1022-S1023.

[0175] S1022: The management node receives a release request sent by the first accelerator.

[0176] The release request includes the task identifier of the target inference task.

[0177] Optionally, a release request is generated when the termination condition is met. Specifically, the termination condition includes at least one of the following: reaching the maximum generation length, encountering the end-of-life symbol EOS, or user-initiated termination.

[0178] S1023: The management node releases the task record corresponding to the target inference task in the metadata database and the historical intermediate state data corresponding to the target inference task in the shared memory pool based on the task identifier of the target inference task.

[0179] In one example, after the management node detects that the first accelerator has completed task inference, it can send a memory release request to the resource management module of the shared memory pool. Specifically, the controller of the shared memory pool first verifies the task execution status, confirming that the task has been completed and no other unit is accessing the data; after successful verification, the corresponding first address segment is marked as "free" and added to the shared memory pool's memory area to be shared, for subsequent tasks to reuse; finally, a release success response is returned to the management node, completing resource reclamation.

[0180] In summary, by establishing a shared framework, multiple AI accelerators can access a shared memory pool, thereby enabling efficient sharing and reuse of intermediate state data among different AI accelerators, improving cache utilization and inference efficiency.

[0181] In the aforementioned embodiment based on the Prefill-Decoding (PD) separation architecture, the system deploys two AI accelerators: the second accelerator performs the computation in the Prefill stage, and the first accelerator performs the computation in the Decoding stage, thereby achieving stage decoupling and resource optimization of the computational tasks. In addition, this application also provides an alternative implementation method that is not separate (i.e., Prefill and Decoding are executed sequentially on the same accelerator). In this mode, a single AI accelerator (e.g., the first accelerator) sequentially completes the prompt word filling computation and subsequent autoregressive decoding, without requiring cross-device task migration or state synchronization. The following is a combination of... Figure 9 An example is provided.

[0182] Figure 9 This is a third flowchart illustrating a task processing method provided in an embodiment of this application.

[0183] like Figure 9 As shown, the task processing method includes the following steps: S01: The first accelerator acquires the target inference task.

[0184] Figure 10 This is a schematic diagram of the signaling flow of another task processing method provided in the embodiments of this application.

[0185] like Figure 10 As shown, step S01 includes steps S2001-S2002.

[0186] S2001: The UI module of the terminal device responds to the user's input operation and obtains the target inference task.

[0187] S2002: The UI module of the terminal device sends the target inference task to the second accelerator.

[0188] The specific details of steps S2001-S2002 can be found in steps S1001-S1002 above, and will not be repeated here.

[0189] S02: The first accelerator determines the predicted resource allocation result of the target inference task based on the sequence length of the target inference task.

[0190] like Figure 10 As shown, step S02 includes steps S2003-S2005.

[0191] S2003: The first accelerator determines the first storage space for the target inference task based on the sequence length of the target inference task.

[0192] S2004: The first accelerator generates a resource allocation request based on the first storage space required to process the target inference task.

[0193] S2005: The first accelerator sends a resource allocation request to the management node.

[0194] The specific details of steps S2003-S2005 can be found in steps S1003-S1005 above, and will not be repeated here.

[0195] S03: The management node determines the resource allocation result for the target inference task based on the resource allocation request and the available capacity of the shared memory pool.

[0196] like Figure 10 As shown, step S03 includes steps S2006-S2008.

[0197] S2006: The management node verifies the first storage space and resource information and their matching relationship with the preset verification rules.

[0198] S2007: In the case of a complete match, the management unit determines the first address segment from the shared memory pool.

[0199] S2008: The management node creates a task record corresponding to the target inference task in the metadata database.

[0200] The specific details of steps S2006-S2008 can be found in steps S1006-S1008 above, and will not be repeated here.

[0201] S04: The management unit sends the resource allocation results to the first accelerator.

[0202] Continue to combine Figure 10 As shown, step S04 includes step S2009.

[0203] S2009: The management node sends the resource allocation results to the second accelerator.

[0204] The specific details of step S209 can be found in step S1009 above, and will not be repeated here.

[0205] S05: Based on the resource allocation results, the first accelerator performs inference calculations on each word in the inference task in sequence during the forward inference phase, and stores the generated intermediate state data in the first address segment.

[0206] Continue to combine Figure 10 As shown, step S05 includes steps S2010-S2015.

[0207] S2010: The first accelerator feeds the input prompts into the large model to calculate intermediate state data.

[0208] S2011: The first accelerator acquires historical access latency information and topology relationships.

[0209] S2012: The first accelerator analyzes the actual response latency and bandwidth utilization of each path based on historical access latency information and topology relationships.

[0210] S2013: The first accelerator determines the target path based on the actual response latency and bandwidth utilization.

[0211] S2014: The first accelerator, based on the target path, stores the intermediate state data (i.e., key-value cache, KV Cache) generated during the inference process into the first address segment allocated for the target inference task in the shared memory pool.

[0212] S2015: After completing the processing of the entire prompt word sequence, the second accelerator updates the task record in the metadata database corresponding to the target inference task.

[0213] The specific details of steps S2010-S2015 can be found in steps S1010-S1015 above, and will not be repeated here.

[0214] S06: During the decoding phase, the first accelerator, in response to processing the target inference task, retrieves the historical intermediate state data corresponding to the target inference task from the first address segment of the shared memory pool.

[0215] Continue to combine Figure 10 As shown, step S06 includes steps S2016-S2017.

[0216] S2016: During the decoding phase, the first accelerator, in response to the task identifier of the target inference task, retrieves the position index corresponding to the current lexical in the target inference task from the metadata in the shared memory pool.

[0217] S2017: The first accelerator retrieves historical intermediate state data from the first address segment of the shared memory pool based on the location index.

[0218] The specific details of steps S2016-S2017 can be found in steps S1016-S1017 above, and will not be repeated here.

[0219] S07: The first accelerator calculates new lexical units and the corresponding first intermediate state data based on historical intermediate state data.

[0220] Continue to combine Figure 10 As shown, step S07 includes step S2018.

[0221] S2018: The first accelerator calculates new lexical units and the corresponding first intermediate state data based on historical intermediate state data.

[0222] The specific details of step S2018 can be found in step S1018 above, and will not be repeated here.

[0223] S08: The first accelerator stores the first intermediate state data into the first address segment.

[0224] Continue to combine Figure 10 As shown, step S08 includes steps S2019-S2021.

[0225] S2019: The first accelerator determines the second storage location of the new word in the first address segment based on the fixed storage space occupied by the starting physical address and the intermediate state data corresponding to the new word.

[0226] S2020: Write the first intermediate state data corresponding to the newly generated word element into the second storage location.

[0227] S2021: The first accelerator updates the task record in the metadata database corresponding to the target inference task.

[0228] The specific details of steps S2019-S2021 can be found in steps S1019-S1021 above, and will not be repeated here.

[0229] S9: After completing all inference tasks, the management node releases the relevant data stored in the shared memory pool of the target inference task.

[0230] Continue to combine Figure 10 As shown, step S09 includes steps S2022-S2023.

[0231] S2022: The management node receives a release request sent by the first accelerator.

[0232] S2023: Based on the task identifier of the target inference task, the management node releases the task record corresponding to the target inference task in the metadata database and the historical intermediate state data corresponding to the target inference task in the shared memory pool.

[0233] The specific details of steps S2022-S2023 can be found in steps S1022-S1023 above, and will not be repeated here.

[0234] In summary, by establishing a shared framework, multiple AI accelerators can access a shared memory pool, thereby enabling efficient sharing and reuse of intermediate state data among different AI accelerators, improving cache utilization and inference efficiency.

[0235] Corresponding to the above-described task reasoning method embodiments, this application also provides another embodiment of a reasoning method. This embodiment is applied to an accelerated computing unit, which includes multiple AI accelerators, including a first accelerator. The following describes... Figure 11 An example is provided.

[0236] Figure 11 This is the fourth flowchart of a task reasoning method provided in an embodiment of this application.

[0237] like Figure 11 As shown, this is applied to an accelerated computing unit, which includes multiple AI accelerators, each including a first accelerator. These accelerators are connected to a shared memory pool based on a cache coherence protocol. The method includes: S111: In response to processing the target inference task, the first accelerator obtains the historical intermediate state data corresponding to the target inference task from the first address segment of the shared memory pool.

[0238] The specific details of step S111 can be found in step S6 above, and will not be repeated here.

[0239] S112: The first accelerator calculates new lexical units and the corresponding first intermediate state data based on historical intermediate state data.

[0240] The specific details of step S112 can be found in step S7 above, and will not be repeated here.

[0241] S113: The first accelerator stores the first intermediate state data into the first address segment.

[0242] The details of step S113 can be found in step S8 above, and will not be repeated here.

[0243] In summary, the shared framework enables multiple AI accelerators to access the shared memory pool, thereby achieving efficient sharing and reuse of intermediate state data among different AI accelerators, improving cache utilization and inference efficiency.

[0244] Corresponding to the above-described task reasoning method embodiments, this application also provides another embodiment of a reasoning method. This embodiment is applied to an accelerated computing unit, which includes multiple AI accelerators, including a second accelerator. The following describes the embodiment in conjunction with... Figure 12 An example is provided.

[0245] Figure 12 This is the fifth flowchart of a task reasoning method provided in an embodiment of this application.

[0246] like Figure 12As shown, this is applied to an accelerated computing unit, which includes multiple AI accelerators, each including a second accelerator. These accelerators are connected to a shared memory pool based on a cache coherence protocol. The method includes: S121: The second accelerator acquires the target inference task.

[0247] The specific details of step S121 can be found in step S1 above, and will not be repeated here.

[0248] S122: The second accelerator obtains the resource allocation results for the target inference task based on the sequence length of the target inference task.

[0249] The predicted resource allocation results include the initial storage space required to process the target inference task.

[0250] The specific details of step S122 can be found in step S2 above, and will not be repeated here.

[0251] S123: The second accelerator generates a resource allocation request based on the first storage space required to process the target inference task and sends it to the management node.

[0252] The specific details of step S123 can be found in steps S3-S4 above, and will not be repeated here.

[0253] S124: The second accelerator receives the resource allocation results sent by the management node.

[0254] S125: The second accelerator calculates intermediate state data based on the input prompt words.

[0255] The specific details of steps S124-S125 can be found in steps S4-S5 above, and will not be repeated here.

[0256] S126: The second accelerator stores intermediate state data in the first address segment to form historical intermediate state data.

[0257] The specific details of step S126 can be found in step S6 above, and will not be repeated here.

[0258] In summary, the shared framework enables multiple AI accelerators to access the shared memory pool, thereby achieving efficient sharing and reuse of intermediate state data among different AI accelerators, improving cache utilization and inference efficiency.

[0259] Corresponding to the above-described task reasoning method embodiments, this application also provides another embodiment of a reasoning method, which is applied to a management node. The following describes the embodiment in conjunction with... Figure 13 An example is provided.

[0260] Figure 13This is the sixth flowchart of a task reasoning method provided in an embodiment of this application.

[0261] like Figure 13 As shown, this is applied to the management node, which manages multiple AI accelerators connected to a shared memory pool. The task processing methods include: S131: The management unit receives a resource allocation request sent by the second accelerator.

[0262] The resource allocation request includes the first storage space and resource information for the target inference task.

[0263] The specific details of step S131 can be found in step S3 above, and will not be repeated here.

[0264] S132: The management node determines the first address segment based on the resource allocation request and the available capacity of the shared memory pool.

[0265] The first address segment is a continuous address segment in the shared memory pool, which is used to store historical intermediate state data corresponding to the target inference task.

[0266] The specific details of step S132 can be found in step S3 above, and will not be repeated here.

[0267] S133: The management unit sends the resource allocation results to the second accelerator.

[0268] The specific details of step S133 can be found in step S4 above, and will not be repeated here.

[0269] In summary, by using a shared framework, multiple AI accelerators can access a shared memory pool, thereby enabling efficient sharing and reuse of intermediate state data among different AI accelerators, improving cache utilization and inference efficiency.

[0270] To achieve efficient access between the AI ​​accelerator and the shared memory pool, a communication connection needs to be established between them during the initialization phase of the distributed inference system. This connection includes not only physical link connectivity but also address space mapping. The following section will combine... Figure 14 An example is provided.

[0271] Figure 14 This is a schematic diagram illustrating a process for establishing a communication connection between a shared memory pool and an AI accelerator, as provided in an embodiment of this application.

[0272] like Figure 14 As shown, the method for establishing a communication connection between the shared memory pool and the AI ​​accelerator includes the following steps: S141: Establish a connection with the pre-configured shared memory pool and obtain the base address and total space of the shared memory pool.

[0273] Continue to combine Figure 14 As shown, step S141 includes steps S1411-S1413.

[0274] S1411: Get the base address and total size of the initial CXL shared pool.

[0275] The hardware resource configuration information for initializing the CXL shared device is queried and obtained through the firmware interface. This information is used to centrally store and manage intermediate state data (KV Cache) generated by multiple computing devices or processing units during the execution of target inference tasks, supporting cross-device, low-latency data sharing and access. The hardware resource configuration information includes the base address and the total space size. The base address refers to the starting position of the initialized CXL shared device in the system's physical address space, and the total space size refers to the total capacity of the initialized CXL shared pool, i.e., the available capacity.

[0276] For example, the base address "0x1A2B3C4D_0000" and the total space size "128GB" of the initial CXL shared device are obtained.

[0277] Since the memory resources of the CXL shared pool mentioned above are not all used to execute the target inference task, a portion of it needs to be allocated specifically for storing intermediate state data during the inference process.

[0278] S1412: Based on the base address and total space, determine the target memory region (shared memory pool) in the CXL shared pool for performing the target inference task.

[0279] The shared memory pool is a logical management area in the CXL shared device used by the AI ​​accelerator. This logical management area includes a dedicated sub-area for intermediate state data and a metadata sub-area.

[0280] Continuing with the example above, based on the base address "0x1A2B3C4D_0000" of the initialized CXL shared device and the total space size "128GB", the initialized CXL shared device is divided into cache region F. The total space size of cache region F is M=32GB, its starting physical address is 0x1A2B3C4D_0000 (inherited from the base address of the CXL shared device), and its ending address is 0x1A2B3C4D_0000 + 32GB - 1.

[0281] S1413: Map the shared memory pool to the address spaces of multiple AI accelerators and management nodes respectively.

[0282] Continuing with the example above, cache region F (starting physical address 0x1A2B3C4D_0000, size 32GB) is mapped into the virtual address space of multiple AI accelerators and management nodes.

[0283] In summary, directly mapping the shared memory pool to the address spaces of multiple AI accelerators and management nodes avoids the communication overhead caused by cross-node data migration and improves inference efficiency.

[0284] Corresponding to the aforementioned embodiments of the task processing method, this application also provides embodiments of the task processing method apparatus.

[0285] Figure 15 This is a schematic diagram of a task processing device provided in an embodiment of this application.

[0286] like Figure 15 As shown, the task inference device 1500 is applied to a first accelerator, which includes a first acquisition module 1501, a first inference module 1502, and a storage module 1503. The first acquisition module 1501 is used to acquire historical intermediate state data corresponding to the target inference task from a first address segment of a shared memory pool in response to processing the target inference task. The first inference module 1502 is used to calculate new lexical units and their corresponding first intermediate state data based on the historical intermediate state data. The storage module 1503 is used to store the first intermediate state data in the first address segment.

[0287] In one possible implementation, the inference module 1502 is used to, in response to processing the target inference task, obtain the position index corresponding to the current word in the target inference task from the meta-database in the shared memory pool; and based on the position index, obtain historical intermediate state data from the first address segment of the shared memory pool.

[0288] In one possible implementation, the storage module 1503 is used to determine the second storage location of the new word in the first address segment based on the starting physical address and the fixed storage space occupied by the intermediate state data corresponding to the new word; write the first intermediate state data corresponding to the newly generated word to the second storage location; and update the number of currently processed words.

[0289] Corresponding to the aforementioned embodiments of the task reasoning method, this application also provides embodiments of a task processing method apparatus.

[0290] Figure 16 This is a schematic diagram of another task reasoning device provided in an embodiment of this application.

[0291] like Figure 16As shown, the task inference device 1600 is applied to the second accelerator, which includes: a second acquisition module 1601, a generation module 1602, a sending module 1603, and a second inference module 1604.

[0292] The second acquisition module 1601 is used to acquire the predicted resource allocation result for processing the target inference task based on the sequence length of the target inference task; the predicted resource allocation result includes the first storage space required for processing the target inference task; wherein the first storage space corresponds to the first address segment; the generation module 1602 is used to generate a resource allocation request based on the first storage space required for processing the target inference task.

[0293] In one possible implementation, the sending module 1603 is used to send a resource allocation request to the management node; wherein the resource allocation request includes a task identifier of the target inference task, a model type for executing the target inference task, a first storage space required to execute the target inference task, and a user identifier for inputting the target inference task; wherein the user identifier is associated with the inference task level, so that the management node prioritizes multiple inference requests from different users based on the inference task level and processes high-priority users first.

[0294] In one possible implementation, the second inference module 1604 is used to calculate intermediate state data based on the input prompt words; and store the intermediate state data in the first address segment to form historical intermediate state data.

[0295] Corresponding to the aforementioned embodiments of the task reasoning method, this application also provides embodiments of a task processing method apparatus.

[0296] Figure 17 This is a schematic diagram of another task reasoning device provided in the embodiments of this application.

[0297] like Figure 17 As shown, the task inference device 1700 is applied to the management node, which includes: a receiving module 1701, a determining module 1702, a creating module 1703, and a releasing module 1704.

[0298] The receiving module 1701 is used to receive the resource allocation request sent by the second accelerator; the determining module 1702 is used to determine the first address segment based on the resource allocation request and the available capacity of the shared memory pool; wherein, the first address segment is a continuous address segment in the shared memory pool, and the first address segment is used to store the historical intermediate state data corresponding to the target inference task.

[0299] In one possible implementation, module 1703 is created to create a task record corresponding to the target inference task in the metadata database. The task record stores context information related to the target inference task, including the task identifier of the target inference task, the starting physical address of the target inference task, and the number of lexical units being processed in the target inference task.

[0300] In one possible implementation, the receiving module 1701 is used to receive a release request sent by the first accelerator; the release request includes the task identifier of the target inference task; the release module 1704 is used to release the task record corresponding to the target inference task in the metadata database and the historical intermediate state data corresponding to the target inference task in the shared memory pool based on the task identifier of the target inference task.

[0301] In one possible implementation, the determining module 1702 is used to determine a first address segment based on a resource allocation request, wherein the first address segment is a contiguous address space in a shared memory pool; and based on the first address segment, to send the target resource allocation result of the target inference task to the second accelerator.

[0302] Corresponding to the above embodiments, this application also provides a different... Figure 4 The computing device implementation method of the architecture shown.

[0303] Figure 18 This is a schematic diagram of a computing device provided in an embodiment of this application.

[0304] like Figure 18 As shown, the computing device 1800 includes a processor 1801 and a memory 1802. Exemplarily, the computing device 1800 may also include a communications interface 1803 and a communications bus 1804.

[0305] The processor 1801, memory 1802, and communication interface 1803 communicate with each other via communication bus 1804. The communication interface 1803 may include a transmitter and receiver for communicating with other devices or communication networks, and may be a wired interface (port), such as a fiber distributed data interface (FDDI) or a gigabit Ethernet interface (GE).

[0306] In some embodiments, the processor 1801 is used to execute program 1805, specifically performing the relevant steps in the above-described inference task execution method embodiments. Specifically, program 1805 may include program code, which includes computer-executable instructions.

[0307] For example, processor 1801 may be a central processing unit (CPU), a graphics processing unit (GPU) (AI accelerator), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement some embodiments of this application. Computing device 1800 may include one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs. The CPU may be a single-core CPU or a multi-core CPU.

[0308] In some embodiments, memory 1802 is used to store program 1805. Memory 1802 may include high-speed random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device.

[0309] Specifically, program 1505 can be called by processor 1801 to cause computing device 1800 to perform processing task generation operations.

[0310] Some embodiments of this application provide a computer-readable storage medium storing at least one executable instruction that, when executed on a computing device 1800, causes the computing device 1800 to perform the task processing method described above.

[0311] For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device.

[0312] This application provides a chip system in several embodiments, which is applied to a server. The chip system includes one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via lines. The interface circuits are used to receive signals from the server's memory and send signals to the processors, the signals including computer instructions stored in the memory. When the server processor executes the computer instructions, the server performs various steps in the task processing method shown in the above-described method embodiments.

[0313] The beneficial effects that the readable storage medium provided in some embodiments of this application can achieve can be referred to the beneficial effects in the corresponding task reasoning and execution methods provided above, and will not be repeated here.

[0314] It should be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0315] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0316] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0317] For the purposes of this specification, "computer-readable medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0318] More specific examples of computer-readable media (a non-exhaustive list) include the following: electrical connections having one or more wires (electronic devices), portable computer disks (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM).

[0319] Furthermore, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory. It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof.

[0320] In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc. The above embodiments are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of this application should be included within the scope of protection of this application.

Claims

1. A task processing method, characterized in that, The method is applied to an accelerated computing unit, which includes multiple AI accelerators, including a first accelerator. The multiple accelerators are connected to a shared memory pool based on a cache coherence protocol. The method includes: In response to processing the target inference task, the first accelerator obtains historical intermediate state data corresponding to the target inference task from the first address segment of the shared memory pool; The first accelerator calculates new lexical units and the corresponding first intermediate state data based on the historical intermediate state data. The first accelerator stores the first intermediate state data in the first address segment.

2. The task processing method according to claim 1, characterized in that, The method further includes: The first or second accelerator calculates intermediate state data based on the input prompt words; The first accelerator or the second accelerator stores the intermediate state data in the first address segment to form the historical intermediate state data.

3. The task processing method according to claim 2, characterized in that, The first accelerator or the second accelerator stores the intermediate state data in the first address segment to form the historical intermediate state data, including: For the i-th word in the input prompt word, the first accelerator or the second accelerator determines the first storage location of the i-th word in the first address segment based on the starting physical address of the first address segment and the fixed storage space occupied by the intermediate state data corresponding to each word. After all the tokens in the input prompt word have been processed, the first accelerator or the second accelerator updates the number of tokens that have been processed.

4. The task processing method according to any one of claims 1-3, characterized in that, In response to processing the target inference task, the first accelerator obtains historical intermediate state data corresponding to the target inference task from the first address segment of the shared memory pool, including: In response to processing the target inference task, the first accelerator obtains the position index corresponding to the current word in the target inference task from the meta database in the shared memory pool; The first accelerator obtains the historical intermediate state data from the first address range based on the location index.

5. The task processing method according to any one of claims 1-4, characterized in that, The first accelerator stores the first intermediate state data in the first address segment, including: The first accelerator determines the second storage location of the new word in the first address segment based on the fixed storage space occupied by the starting physical address and the first intermediate state data corresponding to the new word; The first accelerator writes the first intermediate state data generated by the new lexical into the second storage location; The first accelerator updates the number of currently processed tokens.

6. The task processing method according to claim 1, characterized in that, The method further includes: The second accelerator acquires the target inference task; The second accelerator obtains a predicted resource allocation result for processing the target inference task based on the sequence length of the target inference task; the predicted resource allocation result includes a first storage space required for processing the target inference task; wherein, the first storage space corresponds to the first address segment; The second accelerator generates a resource allocation request based on the first storage space required for the target inference task.

7. The task processing method according to claim 6, characterized in that, After generating the resource allocation request, the following is also included: The second accelerator sends a resource allocation request to the management node; The resource allocation request includes a task identifier for the target inference task, a first storage space required to execute the target inference task, and a user identifier for inputting the target inference task. The user identifier is associated with the inference task level, so that the management node prioritizes multiple inference requests from different users based on the inference task level and processes high-priority users first.

8. The task processing method according to claim 1, characterized in that, The shared memory pool contains multiple shared memory modules, and the multiple AI accelerators are connected to the multiple shared memory modules via a consistent caching protocol. In response to processing the target inference task, the first accelerator obtains historical intermediate state data corresponding to the target inference task from a first address segment of the shared memory pool, including: The first accelerator determines the first address range based on the target inference task; The access request of the first accelerator is routed to the target storage module through the switch to obtain the historical intermediate state data, wherein the first address range is located within the target storage module.

9. A task processing method applied to a management node, characterized in that, The method includes: The management node receives a resource allocation request sent by the second accelerator; The management node determines a first address segment based on the resource allocation request and the available capacity of the shared memory pool; wherein, the first address segment is a continuous address segment in the shared memory pool, and the first address segment is used to store historical intermediate state data corresponding to the target inference task.

10. The task processing method according to claim 9, characterized in that, Also includes: The management node creates a task record corresponding to the target inference task in the metadata database; The task record stores context information related to the target reasoning task, including the task identifier of the target reasoning task, the starting physical address of the target reasoning task, and the number of lexical units being processed in the target reasoning task.

11. The task processing method according to claim 9 or 10, characterized in that, Also includes: The management node receives a release request sent by the first accelerator; the release request includes the task identifier of the target inference task; Based on the task identifier of the target inference task, the management node releases the task record corresponding to the target inference task in the metadata database and the historical intermediate state data corresponding to the target inference task in the shared memory pool.

12. The task processing method according to any one of claims 9-11, characterized in that, Determining the first address segment based on the resource allocation request and the available capacity of the shared memory pool includes: The management node determines the first address segment based on the resource allocation request, whereby the first address segment is a contiguous address space within the shared memory pool. The management node sends the target resource allocation result of the target inference task to the second accelerator based on the first address range.

13. A computing cluster, characterized in that, include: Management node; An accelerated computing unit includes multiple AI accelerators, among which a first accelerator is included; A shared memory pool, which is used to connect to the multiple AI accelerators based on a cache coherence protocol; The first accelerator is configured to obtain historical intermediate state data corresponding to the target inference task from the first address segment of the shared memory pool, calculate new lexical units and the first intermediate state data corresponding to the new lexical units based on the historical intermediate state data, and store the first intermediate state data in the first address segment.

14. The computing cluster according to claim 13, characterized in that, The computing cluster also includes switches; The shared memory pool includes multiple shared memory modules; the multiple AI accelerators and the multiple shared memory modules are connected to the switch based on a consistent caching protocol.

15. A computing device, characterized in that, include: Management node; An accelerated computing unit includes multiple AI accelerators, among which a first accelerator is included; A shared memory pool, which is connected to the multiple AI accelerators based on a cache coherence protocol; The first accelerator is used to obtain historical intermediate state data corresponding to the target inference task from the first address segment of the shared memory pool, and calculate new lexical units and the first intermediate state data corresponding to the new lexical units based on the historical intermediate state data, and store the first intermediate state data in the first address segment.

Images