A large-scale model-based staged inference method and apparatus for multi-core particle systems

Abstract

This application provides a large-model phased inference method and apparatus based on a multi-core system. The large-model phased inference method based on a multi-core system includes: responding to an inference task of obtaining a large model, allocating a first core and a second core among multiple cores; the first core is used to perform a pre-filling stage in the inference task; the second core is used to perform a decoding stage in the inference task; allocating a target storage area in a storage space; the target storage area is used to store key-value caches generated during the processing of the inference task; when the inference task is in the pre-filling stage, the first core performs forward computation on the inference task, and the generated key-values ​​are stored and written to the target storage area based on a first mapping relationship; when the inference task is processing the decoding stage, the second core reads the key-value cache in the target storage area according to a second mapping relationship, and performs decoding computation based on the key-value cache to obtain the processing result of the inference task.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and integrated circuit technology, and in particular to a large-model phased inference method and apparatus based on a multi-core system. Background Technology

[0002] The large model inference process includes a pre-filling stage and a decoding stage. The pre-filling stage requires parallel processing of the input sequence, which has a high computational load. The decoding stage generates output word by word in an autoregressive manner, which accesses the key-value cache more frequently and is more sensitive to storage bandwidth and access latency.

[0003] In related technologies, a unified processor resource is typically used to alternately execute the pre-filling stage and the decoding stage. Since the two stages have significant differences in computational intensity, memory access bandwidth and cache requirements, competition arises between computational resources, cache resources and bandwidth resources, making it difficult to achieve efficient resource utilization and affecting the overall performance of the inference task. Summary of the Invention

[0004] In view of this, embodiments of this application provide a large-model phased inference method and apparatus based on a multi-core system to solve the above problems.

[0005] The first aspect of this application provides a large-model phased inference method based on a multi-core system, applied to a multi-core system comprising multiple cores interconnected and sharing storage space. The method includes: in response to an inference task to obtain a large model, allocating a first core and a second core among the multiple cores; the first core being used to perform a pre-filling phase in the inference task; the second core being used to perform a decoding phase in the inference task; allocating a target storage area in the storage space; the target storage area being used to store key-value caches generated during the processing of the inference task; establishing a first mapping relationship between the first core and the target storage area, and a second mapping relationship between the second core and the target storage area; when the inference task is in the pre-filling phase, performing forward computation on the inference task through the first core, and storing and writing the generated key-values ​​into the target storage area based on the first mapping relationship; when the inference task is processing the decoding phase, reading the key-value cache in the target storage area through the second core according to the second mapping relationship, performing decoding computation based on the key-value cache, and obtaining the processing result of the inference task.

[0006] According to embodiments of this application, allocating a first core and a second core among multiple cores includes: obtaining task feature information of an inference task; the task feature information includes the length of the input text and the expected length of the output text of the inference task; obtaining system state information of the multi-core system; the system state information includes the core utilization rate, request queuing depth, storage space occupancy rate, and power consumption status of the multiple cores; evaluating a first resource requirement of the inference task in the pre-filling stage and a second resource requirement in the decoding stage based on the task feature information and the system state information; allocating the first core among the multiple cores based on the first resource requirement; and allocating the second core among the multiple cores based on the second resource requirement.

[0007] According to an embodiment of this application, allocating a target storage area in the storage space includes: obtaining model structure information of a large model; the model structure information includes the number of model layers, the number of attention heads, and the data precision; calculating the key-value cache capacity required for the inference task based on the model structure information and the length of the input text of the inference task; and dividing the corresponding target storage area in the storage space according to the key-value cache capacity.

[0008] According to embodiments of this application, the first mapping relationship includes at least one of the following: an address mapping relationship from the virtual address space of the first core to the physical address space of the target storage region; an access path mapping relationship from the first core to the target storage region; the access path mapping relationship is used to determine the data transmission path of the first core to access the target storage region through the core interconnect network; a write permission mapping relationship of the first core to the target storage region; and the write permission mapping relationship is used to identify the writable status of each storage location in the target storage region by the first core.

[0009] According to an embodiment of this application, performing forward computation on an inference task using a first core particle and writing the generated key-value pairs into a target storage area based on a first mapping relationship includes: reading the model weights of a large model using the first core particle, performing forward computation on the input terms of the inference task based on the model weights, and generating key-value cache data corresponding to attention computation at each layer; determining the target write address of the key-value cache data in the target storage area according to the address mapping relationship in the first mapping relationship; determining the data transmission path of the first core particle accessing the target storage area through a core particle interconnection network according to the access path mapping relationship in the first mapping relationship; and writing the key-value cache data to the target write address via the data transmission path through a storage access engine within the first core particle; the storage access engine is used to write the generated key-value cache data into the target storage area in a streaming manner while the first core particle performs forward computation.

[0010] According to embodiments of this application, the second mapping relationship includes at least one of the following: an address mapping relationship from the virtual address space of the second core to the physical address space of the target storage region; an access path mapping relationship from the second core to the target storage region; the access path mapping relationship is used to determine the data transmission path of the second core to access the target storage region through the core interconnect network; a read permission mapping relationship of the second core to the target storage region; and a read permission mapping relationship is used to identify the readable status of each storage location in the target storage region by the second core.

[0011] According to an embodiment of this application, the method further includes: determining the target read address of key-value cache data stored in the target storage area according to the address mapping relationship in the second mapping relationship; determining the data transmission path of the second core particle to access the target storage area through the core particle interconnection network according to the access path mapping relationship in the second mapping relationship; and reading the key-value cache in the target storage area through the second core particle according to the second mapping relationship, and performing decoding calculation based on the key-value cache to obtain the processing result of the inference task, including: reading key-value cache data from the target read address through the second core particle via the data transmission path; and performing autoregressive decoding calculation based on the key-value cache data through the second core particle to generate output tokens one by one to obtain the processing result of the inference task.

[0012] According to embodiments of this application, the method further includes: obtaining a power consumption upper limit for the multi-core system; the power consumption upper limit is the maximum power consumption allowed for the multi-core system under normal operating conditions; when the inference task is in the pre-filling stage, reducing the operating frequency or operating voltage of the second core and increasing the operating frequency or operating voltage of the first core based on the power consumption upper limit, so that the power consumption of the first core is greater than the power consumption of the second core; when the inference task switches to the decoding stage, reducing the operating frequency or operating voltage of the first core and increasing the operating frequency or operating voltage of the second core based on the power consumption upper limit, so that the power consumption of the second core is greater than the power consumption of the first core; wherein the sum of the power consumption of the first core and the power consumption of the second core does not exceed the power consumption upper limit.

[0013] According to an embodiment of this application, the method further includes: during the execution of the inference task, obtaining performance indicators of the first core and the second core; determining whether the current resource configuration meets preset performance requirements based on the performance indicators; and performing a resource adjustment operation if the current resource configuration does not meet the preset performance requirements. The resource adjustment operation includes at least one of the following: reallocating the power consumption quotas of the first core and the second core; adjusting the storage space size of the target storage area; or adding a third core to the inference task, the third core being used to share the pre-filling computation of the first core or the decoding computation of the second core.

[0014] A second aspect of this application provides a large model phased inference apparatus based on a multi-core system, comprising: a storage space; a multi-core system including multiple cores interconnected and sharing the storage space; a processor; and a processor for executing the above-described large model phased inference method based on a multi-core system.

[0015] By adopting the technical solution of this application, the first core and the second core are allocated to the pre-filling stage and the decoding stage of the inference task, respectively, in a multi-core system. This allows the two stages to be undertaken by different physical computing resources, thereby avoiding the resource contention problem that occurs when the unified processor resources are used to alternately execute the two stages in related technologies.

[0016] Since the first and second cores can be configured with computing power and memory access capabilities independently according to the characteristics of their respective stages, the high parallel computing requirements of the pre-filling stage and the frequent cache access requirements of the decoding stage can be met in a targeted manner, solving the problem of low resource utilization efficiency caused by the difference in resource requirements between the two stages in related technologies.

[0017] Furthermore, by allocating a target storage area in the shared storage space specifically for storing key-value cache, and establishing a first mapping relationship and a second mapping relationship between the first core and the second core and the target storage area, the key-value cache generated by the first core during the pre-filling stage can be directly written to the target storage area based on the first mapping relationship, while the second core can directly read the key-value cache from the target storage area according to the second mapping relationship during the decoding stage. This achieves efficient data transfer between the pre-filling stage and the decoding stage through the shared storage space, eliminating the need for additional cross-node migration or copying operations of the key-value cache during stage switching, reducing the latency overhead of stage switching, and improving the overall execution efficiency and resource utilization of the inference task. Attached Figure Description

[0018] The above and other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 This is a schematic diagram of the structure of a multi-core particle system provided in an embodiment of this application;

[0020] Figure 2 A flowchart illustrating a large-model staged inference method based on a multi-core system provided in this application embodiment;

[0021] Figure 3 A schematic diagram illustrating power management and data flow in a multi-core system, provided as an embodiment of this application;

[0022] Figure 4This is a timing simulation waveform diagram of power consumption management and phase switching in a multi-core system provided in an embodiment of this application. Detailed Implementation

[0023] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0027] The following describes, with reference to the accompanying drawings, a method and apparatus for staged inference of a large model based on a multi-core system according to embodiments of this application.

[0028] Please refer to Figure 1 , Figure 1 This is a schematic diagram of a multi-core system provided in an embodiment of this application.

[0029] like Figure 1 As shown, the multi-core system includes multiple cores that are interconnected and share storage space.

[0030] Specifically, this multi-chip system employs a multi-chip module-based packaging structure, constructed on a packaging substrate, on which multiple computing chips are packaged. For ease of explanation, Figure 1The diagram schematically labels some of the computational particles as pre-filled particles and decoding particles. Pre-filled particles are used to perform the pre-filling phase of the inference task, and decoding particles are used to perform the decoding phase of the inference task.

[0031] The package structure incorporates shared memory resources that can be accessed by multiple computing chips. These shared memory resources may include an in-package shared cache, an on-chip static random access memory cache, or internal caches within each computing chip. Multiple computing chips can interconnect with the package substrate, shared memory resources, or other computing chips via microbumps and high-speed inter-chip physical interfaces.

[0032] The encapsulation structure also connects to an external dynamic random access memory (DRAM) via a storage control interface. The external DRAM is used to store model parameters and key-value cache data.

[0033] Furthermore, the multi-core system also includes a power management module, which is located within the package or works in conjunction with the package, to control the supply voltage, clock frequency, and power budget allocation of each computing core. In addition, the multi-core system also includes a core scheduling module, a task distribution module, and a cache transfer module, used to schedule computing cores, distribute inference tasks, and transfer cached data between different computing cores.

[0034] It should be noted that the above packaging structure can also be implemented based on 2.5D packaging or 3D packaging. In the 2.5D packaging implementation, silicon interposers, silicon bridges, or other high-density interconnect structures can be introduced; in the 3D packaging implementation, computing chips, cache chips, or storage chips can be stacked and integrated through vertical interconnect structures.

[0035] It should be noted that in edge device deployment scenarios for related technologies, due to the scarcity of hardware resources, a single processor cannot simultaneously meet the high computational intensity requirements of the pre-filling stage and the high memory access pressure requirements of the decoding stage. While in a cloud cluster environment, although pre-filling and decoding tasks can be separated and deployed using multiple independent devices, data transmission between these devices relies on network communication, resulting in high communication latency and bandwidth bottlenecks.

[0036] The multi-core system provided in this application separates pre-filling and decoding at the core level, enabling resource-constrained edge devices to complete the division of labor and cooperation between the pre-filling and decoding stages within a single package structure. Since the pre-filled core and the decoding core are interconnected via a high-speed inter-core physical interface, the data transmission latency between cores can be reduced compared to network communication between multiple devices in a cloud cluster.

[0037] In the above Figure 1 Based on this, please refer to Figure 2 , Figure 2 A flowchart illustrating a phased inference method for a large model based on a multi-core system, provided in this application embodiment.

[0038] like Figure 2 As shown, the above-mentioned large-model phased inference method based on multi-core particle systems may specifically include the following steps.

[0039] In step S210, in response to the inference task of obtaining a large model, a first core and a second core are allocated among multiple cores; the first core is used to perform the pre-filling stage in the inference task; the second core is used to perform the decoding stage in the inference task.

[0040] Step S220: Allocate a target storage area in the storage space; the target storage area is used to store the key-value cache generated when processing inference tasks;

[0041] Step S230: Establish a first mapping relationship between the first core and the target storage region, and a second mapping relationship between the second core and the target storage region;

[0042] Step S240: When the inference task is in the pre-filling stage, the inference task is forward-computed through the first core, and the generated key-value storage is written to the target storage area based on the first mapping relationship.

[0043] In step S250, during the decoding stage of the inference task, the second core reads the key-value cache in the target storage area according to the second mapping relationship, and performs decoding calculation based on the key-value cache to obtain the processing result of the inference task.

[0044] In step S210, the first core refers to the core allocated among multiple cores for processing pre-filled calculations. It can be understood as a computing unit specifically responsible for processing the input sequence, used to perform parallel calculations on the input prompts for the reasoning task and generate corresponding key-value cache data.

[0045] In inference tasks, the pre-filling stage refers to the stage in the large model inference process where the input cue word sequence is computed for the first time. It can be understood as the process of processing the complete input cue words at once and generating initial context information.

[0046] Similarly, the second core refers to the core allocated among multiple cores for processing decoding calculations. It can be understood as a computing unit specifically responsible for generating the output sequence, used to generate output tokens one by one based on the generated key-value cache data.

[0047] The decoding stage in inference tasks refers to the stage in the large model inference process that gradually generates the output sequence. It can be understood as an autoregressive generation process that predicts and outputs words one by one based on the input context.

[0048] In one feasible implementation, the load can be allocated according to the current load status of multiple cores, selecting a core with a lower load as the first core and another core with a lower load as the second core, so as to achieve load balancing among the cores.

[0049] In another feasible implementation, the chips can be allocated according to their computational characteristics. Chips with higher computing power and suitable for handling large-scale parallel computing are allocated as the first chip, and chips with low latency and suitable for handling autoregressive computing are allocated as the second chip.

[0050] It should be noted that the first core and the second core can be different physical cores, or the same physical core can perform different functional roles at different times. During the execution of the inference task, the allocation of the first core and the second core can be dynamically adjusted according to the actual computing needs.

[0051] In step S220, the target storage area refers to the storage area reserved for the current inference task in the shared storage space. It can be understood as a dedicated space for temporarily storing intermediate data during the inference process. It is used to store key-value cache data generated by the inference task in the pre-filling stage and the decoding stage, so that the first core and the second core can access and reuse this data.

[0052] Among them, key-value cache refers to the key vector and value vector data generated by the attention calculation of each layer during the inference process of a large model. It can be understood as a data structure that records the context information of the input sequence and the generated sequence.

[0053] In one feasible implementation, the data size of the key-value cache can be estimated based on the length of the input sequence and the expected output length of the inference task, and a target storage area of ​​the corresponding size can be allocated in the shared storage space.

[0054] In another feasible implementation, the size and location of the target storage area can be dynamically allocated based on the current available capacity of the shared storage space and the priorities of multiple inference tasks to improve the utilization of storage space.

[0055] It should be noted that the target memory region can be allocated to one or more locations within the package's shared cache, on-chip static random access memory cache, internal cache of each compute chip, or external dynamic random access memory. After the inference task is completed, the target memory region can be released for use by other inference tasks.

[0056] In step S230, the first mapping relationship refers to the address correspondence between the first core and the target storage area. It can be understood as the correspondence configuration between the local address space of the first core and the physical address of the target storage area, which enables the first core to write the key-value cache data generated in the pre-filling stage to the target storage area.

[0057] Similarly, the second mapping relationship refers to the address correspondence between the second core and the target storage area. It can be understood as the correspondence configuration between the local address space of the second core and the physical address of the target storage area, which enables the second core to read key-value cache data from the target storage area and reuse it during the decoding stage.

[0058] In one feasible implementation, the first mapping relationship and the second mapping relationship can be established by configuring the address translation tables of the first and second cores respectively to map their virtual addresses to the physical addresses of the target storage area.

[0059] In another feasible implementation, the first mapping relationship and the second mapping relationship can be established by configuring the routing of the high-speed inter-core physical interface to set the data path and transmission protocol for the first core and the second core to access the target storage area.

[0060] It should be noted that the first and second mapping relationships point to the same target storage area, allowing the key-value cache data written by the first memory chip to be directly read by the second memory chip without the need for data transfer between different memory chips. During the inference task execution, read and write permissions can be configured for the first mapping relationship, and read permissions can be configured for the second mapping relationship, according to access permission requirements.

[0061] In step S240, the input prompt word sequence can be computed in parallel using the first core. After the attention calculation of each layer is completed, the corresponding key-value cache is generated in sequence. Then, the physical address of the target storage area is determined according to the first mapping relationship, and the key-value cache of each layer is written to the target storage area in layer order.

[0062] In another feasible implementation, while the first core completes the forward computation of each layer, a streaming write method can be adopted. After each layer of key-value cache computation is completed, the key-value cache of that layer is written to the corresponding position of the target storage area through the first mapping relationship, so as to reduce the cache occupancy inside the first core.

[0063] It should be noted that, since the pre-filling stage requires processing the complete input sequence, the forward computation performed in the first core is a computationally intensive operation with high parallelism. When writing to the key-value cache, synchronous or asynchronous writing methods can be selected based on the access bandwidth and latency characteristics of the target storage area. In the case of asynchronous writing, a write buffer can be set up inside the first core, and the key-value cache can be written to the target storage area all at once after all the computations in the pre-filling stage are completed.

[0064] In step S250, the physical address of the target storage area can be determined by the second core according to the second mapping relationship, the key-value caches generated in the pre-filling stage can be read, and then attention calculation can be performed based on the read key-value caches and the current input words to generate the output words of the current step. The above process is repeated until the end flag is generated or the preset output length is reached, and finally the processing result of the inference task is obtained.

[0065] In another feasible implementation, the key-value cache in the target storage area can be read into the internal cache of the second core before the second core performs the decoding calculation for the first time. Then, in the subsequent autoregressive generation process, the existing key-value cache is read directly from the internal cache, and the newly generated key-value cache for each step is appended to the target storage area and the internal cache to reduce repeated access to the shared storage space.

[0066] It's important to note that the decoding phase involves memory-intensive operations; generating each token requires accessing all existing key-value caches. Directly accessing the target storage area through the second mapping avoids moving the key-value cache from the first to the second core, thus reducing data transfer overhead. During decoding, newly generated key-value caches in the second core can also be written to the target storage area for use in subsequent decoding steps. After the inference task is completed, the key-value cache data in the target storage area can be retained or released as needed.

[0067] By adopting the technical solution of this application, the first core and the second core are allocated to the pre-filling stage and the decoding stage of the inference task, respectively, in a multi-core system. This allows the two stages to be undertaken by different physical computing resources, thereby avoiding the resource contention problem that occurs when the unified processor resources are used to alternately execute the two stages in related technologies.

[0068] Since the first and second cores can be configured with computing power and memory access capabilities independently according to the characteristics of their respective stages, the high parallel computing requirements of the pre-filling stage and the frequent cache access requirements of the decoding stage can be met in a targeted manner, solving the problem of low resource utilization efficiency caused by the difference in resource requirements between the two stages in related technologies.

[0069] Furthermore, by allocating a target storage area in the shared storage space specifically for storing key-value cache, and establishing a first mapping relationship and a second mapping relationship between the first core and the second core and the target storage area, the key-value cache generated by the first core during the pre-filling stage can be directly written to the target storage area based on the first mapping relationship, while the second core can directly read the key-value cache from the target storage area according to the second mapping relationship during the decoding stage. This achieves efficient data transfer between the pre-filling stage and the decoding stage through the shared storage space, eliminating the need for additional cross-node migration or copying operations of the key-value cache during stage switching, reducing the latency overhead of stage switching, and improving the overall execution efficiency and resource utilization of the inference task.

[0070] Based on the above embodiments, as an optional embodiment, step S210 may further include the following steps.

[0071] Step S310: Obtain task feature information for the reasoning task; the task feature information includes the length of the input text and the expected length of the output text for the reasoning task.

[0072] Step S320: Obtain system status information of the multi-core system; system status information includes core utilization rate, request queuing depth, storage space occupancy rate and power consumption status of multiple cores;

[0073] Step S330: Based on task feature information and system status information, evaluate the first resource requirement of the reasoning task in the pre-filling stage and the second resource requirement in the decoding stage.

[0074] Step S340: Allocate the first core particle among multiple core particles based on the first resource requirement;

[0075] Step S350: Allocate the second core among multiple cores based on the second resource requirement.

[0076] In step S310, the task feature information refers to the set of parameters describing the computational load characteristics of the current inference task. The input text length refers to the number of lemmas contained in the input prompt word sequence received by the inference task.

[0077] In one feasible implementation, the length of the input text can be read directly from the task request when the inference task is submitted, and the expected length of the output text can be obtained according to the maximum generated length specified by the user or the default configuration of the application scenario.

[0078] Optionally, the task feature information may also include at least one of the following: service level objectives for the inference task, model type information, batch size or latency constraints, to support a more granular resource allocation strategy.

[0079] In step S320, system status information refers to the set of monitoring data that reflects the current operating status of the multi-core system, and real-time status parameters that characterize the availability of system resources and load pressure.

[0080] Here, chip utilization refers to the proportion of processing power currently used by each chip to execute computing tasks. Request queuing depth refers to the number of pending inference tasks in the task queue. Storage space utilization refers to the proportion of shared storage resources that have been allocated or used relative to the total capacity. Power consumption status refers to the current total power consumption level of the multi-chip system and the power consumption distribution of each chip.

[0081] In one feasible implementation, the utilization rate of each chip can be periodically collected by a performance counter set inside each chip, the request queuing depth can be read by a task scheduling module, the storage space occupancy rate can be obtained by a storage controller, and the power consumption status can be monitored by a power management module. The above information can be summarized to form system status information.

[0082] It should be noted that system status information can be acquired in real time when an inference task arrives, or it can be queried based on monitoring data collected periodically in the background. When multiple inference tasks are executed concurrently, the system status information can reflect the overall resource usage of all active tasks.

[0083] In step S330, the first resource requirement refers to the number of kernels, computing power, and storage bandwidth required for the inference task to complete the pre-filling stage computation. The second resource requirement refers to the number of kernels, memory access capability, and cache capacity required for the inference task to complete the decoding stage computation, which can be understood as the amount of resources required to achieve the target performance in the decoding stage.

[0084] In one feasible implementation, task feature coefficients can be calculated based on the input text length and the expected output text length, wherein the task feature coefficients characterize the ratio of computational load between the pre-filling stage and the decoding stage.

[0085] For example, the task feature coefficient γ can be calculated using the following formula:

[0086]

[0087] Among them, L in L represents the length of the input text. outε represents the expected output text length, and ε is a very small constant to prevent the denominator from being zero. When the task feature coefficient γ is large, it indicates that the input sequence of the current inference task is long while the expected output sequence is short, the computational load of the pre-filling stage is high, and the first resource requirement is correspondingly large; when the task feature coefficient γ is small, it indicates that the input sequence of the current inference task is short while the expected output sequence is long, the computational load of the decoding stage is high, and the second resource requirement is correspondingly large.

[0088] In step S340, the number of cores required to undertake the pre-filling task can be determined based on the first resource requirement, and then the core with the current low utilization rate and sufficient computing power can be selected as the first core from multiple cores.

[0089] In one feasible implementation, a dynamic pre-filling resource allocation ratio ρ can be set. The value of the dynamic pre-filling resource allocation ratio ρ is determined based on the task characteristic coefficient γ, the request queuing depth, and the core utilization rate. Then, the first core quantity Np is calculated based on the number of active cores N participating in the current inference task. The calculation formula is as follows:

[0090] ;

[0091] Specifically, when the task characteristic coefficient γ is large, the value of the dynamic prefilling resource allocation ratio ρ is increased, which increases the number of first core particles allocated to the prefilling stage; when the task characteristic coefficient γ is small, the value of the dynamic prefilling resource allocation ratio ρ is decreased, which reduces the number of first core particles allocated to the prefilling stage.

[0092] Optionally, the allocation of the first core can be completed before the inference task begins execution, or it can be dynamically determined at the start of the pre-filling phase. When system resources are limited, the actual number of first cores allocated may be less than the ideal number assessed in the first resource requirement evaluation. In this case, resource constraints can be accommodated by extending the execution time of the pre-filling phase or adjusting the batch size.

[0093] In step S350, the number of chips required to undertake the decoding task can be determined according to the second resource requirement, and then the chip with the current low utilization rate and the memory access capability that meets the requirements can be selected as the second chip from multiple chips.

[0094] In one feasible implementation, the second core number Nd can be calculated based on the number of active cores N participating in the current inference task and the dynamic pre-filling resource allocation ratio ρ, using the following formula:

[0095]

[0096] The sum of the number of second core particles Nd and the number of first core particles Np is equal to the number of active core particles N, thereby achieving a complete division of the core particle resources participating in the inference task into a pre-filling stage and a decoding stage.

[0097] Optionally, the allocation of the second core can be completed simultaneously with the allocation of the first core, or it can be allocated during or after the pre-filling phase, to support different task scheduling strategies.

[0098] By adopting the above-described technical solution of this embodiment, before allocating the first and second cores, the task characteristic information of the inference task and the system state information of the multi-core system are obtained first. Based on both, the resource requirements of the pre-filling stage and the decoding stage are evaluated, enabling core allocation to be combined with task characteristics and the actual system conditions. By dynamically determining the pre-filling resource allocation ratio according to the ratio of the input text length to the expected output text length, the core quantity ratio of the two stages can be adaptively adjusted for inference tasks with different input-output length combinations. This ensures that both the computationally intensive pre-filling stage and the memory-intensive decoding stage can obtain hardware resources matching their load characteristics, thereby improving the adaptability of the multi-core system to different types of inference tasks.

[0099] Based on the above embodiments, as an optional embodiment, step S220 may further include the following steps.

[0100] Step S410: Obtain the model structure information of the large model; the model structure information includes the number of model layers, the number of attention heads, and the data precision;

[0101] Step S420: Calculate the key-value cache capacity required for the inference task based on the model structure information and the length of the input text for the inference task;

[0102] Step S430: Based on the key-value cache capacity, divide the corresponding target storage area in the storage space.

[0103] In step S410, the model structure information refers to the configuration data that describes the parameters of the large model network architecture, which can be understood as a set of structural parameters that determine the model calculation process and data scale.

[0104] In one feasible implementation, model structure information such as the number of model layers, the number of attention heads, and the data precision can be read from the model configuration file when loading a large model.

[0105] In another feasible implementation, the model structure information of commonly used large models can be pre-parsed and cached during the initialization phase of the multi-core system. When an inference task is received, the corresponding model structure information can be directly queried and obtained according to the model identifier specified by the task.

[0106] In step S420, the key-value cache capacity refers to the storage space required to store all key-value cache data of the inference task. It can be understood as the total number of bytes to accommodate the key vector and value vector data of each layer. It is used to determine the size of the storage area that needs to be reserved for the current inference task in the shared storage space.

[0107] In one feasible implementation, the total capacity required to store the key vectors and value vectors of each layer can be calculated based on the number of model layers, the number of attention heads, the length of the input text, and the number of bytes corresponding to the data precision.

[0108] In another feasible implementation, a reserved space can be added when calculating the basic capacity. The reserved space is used to store the key-value cache data corresponding to the newly generated lexical units in the decoding stage, so that a target storage area sufficient to accommodate all key-value cache data of the entire inference task can be allocated at once before the pre-filling stage begins.

[0109] In step S430, based on the calculated key-value cache capacity, an available storage area can be found in the shared storage space, and the found storage area can be marked as the target storage area for the current inference task.

[0110] In one feasible implementation, a contiguous allocation strategy can be adopted in the shared storage space to find a contiguous available storage area with a capacity not less than the key-value cache capacity, and record the starting address and length information of the contiguous storage area as the address description information of the target storage area.

[0111] In another feasible implementation, a segmented allocation strategy can be adopted in the shared storage space to decompose the key-value cache capacity into multiple storage segments, allocate each storage segment in different locations in the shared storage space, and then organize the multiple scattered storage segments into a logically continuous target storage area through an address mapping table.

[0112] Optionally, when the shared storage space includes multiple tiers, different parts of the target storage area can be allocated in the in-package shared cache or external dynamic random access memory based on the key-value cache capacity and the access characteristics of the storage resources at each tier.

[0113] By adopting the above-described technical solution of this embodiment, the key-value cache capacity required for the inference task is calculated based on the model structure information of the large model and the length of the input text of the inference task. Then, the corresponding target storage area is divided in the storage space according to the calculated key-value cache capacity, so that the size of the target storage area can accurately match the amount of key-value cache data actually generated by the inference task. This avoids the problem of insufficient storage space caused by allocation that is too small and the problem of waste of storage resources caused by allocation that is too large, thereby improving the utilization efficiency of the shared storage space.

[0114] Based on the above embodiments, as an optional embodiment, the first mapping relationship includes at least one of the following:

[0115] The address mapping relationship between the virtual address space of the first core and the physical address space of the target storage region;

[0116] The access path mapping relationship from the first core to the target storage area; the access path mapping relationship is used to determine the data transmission path of the first core to access the target storage area through the core interconnect network;

[0117] The write permission mapping relationship between the first core and the target storage area; the write permission mapping relationship is used to identify the writable status of each storage location in the target storage area for the first core.

[0118] The address mapping relationship refers to the conversion rules between the virtual address used by the first core and the actual physical address of the target storage area, which enables the first core to access the target storage area distributed in different physical locations in the shared storage space with a unified virtual address space.

[0119] In one feasible implementation, an address mapping table can be configured in the address translation unit inside the first core. The address mapping table records the correspondence between the virtual address segments of the first core and the physical address segments of the target storage area. When the first core accesses a virtual address, the address translation unit converts the virtual address into the corresponding physical address according to the address mapping table.

[0120] The access path mapping relationship refers to the routing configuration of the physical links and intermediate nodes traversed in the core interconnection network when the first core accesses the target storage area. It is used to establish a definite communication path for data transmission from the first core to the target storage area in the interconnection topology of a multi-core system.

[0121] In one feasible implementation, a routing entry from the first die to the target storage region can be configured in the routing table of the die interconnect network based on the physical location of the first die and the target storage region within the package. The routing entry specifies the high-speed inter-die physical interface, interconnect bridge, or intermediate layer interconnect resource that the access request needs to pass through.

[0122] The write permission mapping relationship refers to the write operation permission configuration of the first core to different address ranges in the target storage area, which is used to prevent unauthorized data write operations when multiple cores share access to the same storage space.

[0123] In one feasible implementation, a write permission identifier can be configured for the first core in the access control table of the target storage area. The write permission identifier specifies that the first core has write permission to the entire address range or a portion of the address range of the target storage area. When the first core initiates a write operation, the storage controller determines whether the write operation is allowed based on the write permission identifier in the access control table.

[0124] Optionally, the first mapping relationship may include all three mapping relationships mentioned above, or only one or two of them. When only address mapping relationships are included, the access path can be determined by the default routing rules of the chip interconnect network, and write permissions can be defaulted to allow writing. When multiple mapping relationships are included, they work together to achieve access control of the first chip to the target storage area.

[0125] By adopting the above-described technical solution of this embodiment, and by establishing a first mapping relationship including address mapping, access path mapping, and write permission mapping, the first core can transparently access the target storage area using virtual addresses without needing to concern itself with its physical distribution. Furthermore, write permission control ensures the security of data writing, thereby providing complete address translation, routing, and permission control support for the first core to write key-value cache data to the target storage area during the pre-filling stage.

[0126] Based on the above embodiments, as an optional embodiment, step S240 may further include the following steps.

[0127] Step S510: Read the model weights of the large model through the first core, and perform forward computation on the input words of the inference task based on the model weights to generate key-value cache data corresponding to the attention computation of each layer;

[0128] Step S520: Determine the target write address of the key-value cache data in the target storage area according to the address mapping relationship in the first mapping relationship;

[0129] Step S530: Determine the data transmission path of the first chip to access the target storage area through the chip interconnect network according to the access path mapping relationship in the first mapping relationship;

[0130] In step S540, the key-value cache data is written to the target write address via the data transmission path through the storage access engine within the first core. The storage access engine is used to write the generated key-value cache data to the target storage area in a streaming manner while the first core performs forward computation.

[0131] In step S510, model weights refer to the parameter matrices in each layer of the large model network, which can be understood as the weight parameters and bias parameters required to perform forward computation.

[0132] In one feasible implementation, model weights can be read from an external dynamic random access memory and loaded into the internal cache of the first core. Then, embedding layer calculation, layer-by-layer forward propagation, and attention calculation are performed on the input word sequence. After the attention calculation of each layer is completed, the corresponding key vector and value vector are generated as the key-value cache data of that layer.

[0133] In step S520, the target write address refers to the physical address location within the target storage area where the key-value cache data should be written, which can be understood as the specific storage location of the key-value cache data in the shared storage space.

[0134] In one feasible implementation, the virtual address of the key-value cache data in the first core virtual address space can be calculated based on the currently calculated layer index, sequence position, and key-value vector dimension. Then, the virtual address can be converted into a physical address in the target storage area through address mapping to obtain the target write address.

[0135] In step S530, the data transmission path refers to the sequence of physical links through which key-value cache data is transmitted from the first core to the target storage area. It can be understood as the specific channels and nodes through which the data flows in the core interconnection network.

[0136] In one feasible implementation, a transmission path from the first core to the storage resource of the target storage area can be determined based on the routing table entries configured in the access path mapping relationship. The transmission path includes the output interface of the first core, intermediate switching nodes in the core interconnection network, and the input interface of the target storage area.

[0137] In step S540, the storage access engine refers to the hardware unit inside the first core that is responsible for performing memory read and write operations, and is used to process the write operation of key-value cache data in parallel while the first core performs forward computation.

[0138] In one feasible implementation, after the first core completes the attention calculation for each layer, the generated key-value cache data can be stored in the write buffer inside the first core. Then, the storage access engine reads the key-value cache data from the write buffer and sends it to the target write address via a determined data transmission path, thereby realizing the pipelined parallel execution of computation and write operations.

[0139] It should be noted that the storage access engine can support concurrent processing of multiple write requests. When the first core comprises multiple computing units generating key-value cache data at different layers or sequence positions in parallel, the storage access engine can simultaneously receive and process write requests from multiple computing units. When the target storage area is located in multiple different physical locations, the storage access engine can dynamically adjust the sending rate of write requests based on the access bandwidth of each physical location.

[0140] By adopting the above-described technical solution of this embodiment, the target write address is determined according to the address mapping relationship, and the data transmission path is determined according to the access path mapping relationship. This enables the first core to accurately locate the storage location of the key-value cache data and select a suitable transmission channel. Then, the key-value cache data is written to the target storage area in a streaming manner through the storage access engine. This achieves parallel execution of forward computation and data writing, improves the execution efficiency of the pre-filling stage and the utilization rate of the cache resources inside the first core.

[0141] Based on the above embodiments, as an optional embodiment, the second mapping relationship includes at least one of the following:

[0142] The address mapping relationship between the virtual address space of the second core and the physical address space of the target storage region;

[0143] The access path mapping relationship from the second core to the target storage area; the access path mapping relationship is used to determine the data transmission path of the second core to access the target storage area through the core interconnect network.

[0144] The second core part maps read permissions to the target storage area; the read permission mapping relationship is used to identify the readable status of each storage location in the target storage area for the second core part.

[0145] The address mapping relationship in the second mapping relationship refers to the conversion rule between the virtual address used by the second core and the actual physical address of the target storage area.

[0146] In one feasible implementation, an address mapping table can be configured in the address translation unit inside the second core. The address mapping table records the correspondence between the virtual address segments of the second core and the physical address segments of the target storage area. When the second core accesses a virtual address, the address translation unit converts the virtual address into the corresponding physical address according to the address mapping table.

[0147] The access path mapping relationship in the second mapping relationship refers to the routing configuration of the physical links and intermediate nodes traversed in the chip interconnect network when the second chip accesses the target storage area.

[0148] In one feasible implementation, a routing entry from the second chip to the target storage region can be configured in the routing table of the chip interconnect network based on the physical location of the second chip and the target storage region within the package. The routing entry specifies the high-speed inter-chip physical interface, interconnect bridge, or intermediate layer interconnect resource that the access request needs to pass through.

[0149] The read permission mapping relationship in the second mapping relationship refers to the read operation permission configuration of the second core for different address ranges in the target storage area.

[0150] In one feasible implementation, a read permission identifier can be configured for the second core in the access control table of the target storage area. The read permission identifier specifies that the second core has read permission for the entire address range or a portion of the address range of the target storage area. When the second core initiates a read operation, the storage controller determines whether the read operation is allowed based on the read permission identifier in the access control table.

[0151] Optionally, the second mapping relationship and the first mapping relationship point to the same target storage area, so that the key-value cache data written by the first core can be directly read by the second core. The second mapping relationship can be established at the same time as the first mapping relationship between the first core and the target storage area, or it can be established during the pre-filling stage or after the pre-filling stage is completed.

[0152] By adopting the above-described technical solution of this embodiment, by establishing a second mapping relationship including address mapping relationship, access path mapping relationship and read permission mapping relationship, the second core can transparently access the target storage area in the form of virtual address without paying attention to its physical distribution. The access path mapping relationship selects a suitable data transmission channel for the second core to improve the access efficiency of key-value cache data in the decoding stage. The read permission control ensures the security and isolation of data access.

[0153] Based on the above embodiments, as an optional embodiment, the above method may further include the following steps.

[0154] Step S610: Determine the target read address of the key-value cache data stored in the target storage area according to the address mapping relationship in the second mapping relationship;

[0155] Step S620: Determine the data transmission path of the second core to access the target storage area through the core interconnect network according to the access path mapping relationship in the second mapping relationship;

[0156] Step S630: Read key-value cache data from the target read address via the data transmission path through the second core;

[0157] In step S640, the second core performs autoregressive decoding calculation based on the key-value cache data to generate output tokens one by one, thereby obtaining the processing result of the inference task.

[0158] In step S610, the target read address refers to the physical address location of the key-value cache data within the target storage area.

[0159] In one feasible implementation, the virtual address of the key-value cache data in the second core virtual address space can be calculated based on the model layer index, sequence position, and key-value vector dimension that need to be accessed during the decoding stage. Then, the virtual address is converted into a physical address in the target storage area through the address mapping relationship in the second mapping relationship to obtain the target read address.

[0160] In step S620, the data transmission path refers to the sequence of physical links through which key-value cache data is transmitted from the target storage area to the second core. It can be understood as the specific channels and nodes through which the data flows in the core interconnection network.

[0161] In one feasible implementation, a transmission path from the storage resource where the target storage area is located to the second core can be determined based on the routing table entries configured in the access path mapping relationship in the second mapping relationship. The transmission path includes the output interface of the target storage area, the intermediate switching node in the core interconnection network, and the input interface of the second core.

[0162] In step S630, a read request can be generated based on the determined target read address through the storage access engine inside the second core, and then the read request can be sent to the target storage area via the determined data transmission path. The key-value cache data returned by the target storage area can be received and stored in the cache inside the second core.

[0163] In one feasible implementation, before the decoding calculation is performed for the first time in the decoding stage, all key-value cache data generated in the pre-filling stage in the target storage area can be read in batches and cached in the internal cache of the second core. Then, in the subsequent autoregressive generation process, the existing key-value cache data can be read directly from the internal cache of the second core to reduce repeated access to the shared storage space.

[0164] It should be noted that after generating new output tokens during the decoding stage, the second core will calculate the key-value cache data corresponding to the new tokens. The key-value cache data corresponding to the new tokens can be written into the internal cache of the second core or appended to the target storage area for use in subsequent decoding steps.

[0165] In step S640, the model weights of the large model can be read through the second core, and attention calculation can be performed on the current input word and the key-value cache data read from the target storage area to generate the output word and the corresponding key-value cache data of the current step. Then, the output word is used as the input word for the next step, and the above calculation process is repeated until an end flag is generated or the preset output length is reached to obtain the processing result of the inference task.

[0166] By adopting the above-described technical solution of this embodiment, the target read address and data transmission path are determined according to the address mapping relationship and access path mapping relationship in the second mapping relationship, respectively. This enables the second core to accurately locate the storage location of the key-value cache data and select a suitable transmission channel. Then, the second core reads the key-value cache data from the target read address through the determined data transmission path and performs autoregressive decoding calculation, thereby realizing efficient access and reuse of the key-value cache data generated in the pre-filling stage during the decoding stage.

[0167] Based on the above embodiments, as an optional embodiment, the above-described large model phased inference method based on multi-core particle systems may further include the following steps.

[0168] Step S710: Obtain the power consumption limit of the multi-core system; the power consumption limit is the maximum power consumption allowed for the multi-core system under normal operating conditions.

[0169] Step S720: When the inference task is in the pre-filling stage, reduce the operating frequency or operating voltage of the second core based on the power consumption limit, and increase the operating frequency or operating voltage of the first core, so that the operating power consumption of the first core is greater than that of the second core.

[0170] In step S730, when the inference task is switched to the decoding stage, the operating frequency or operating voltage of the first chip is reduced based on the power consumption limit, and the operating frequency or operating voltage of the second chip is increased so that the power consumption of the second chip is greater than that of the first chip.

[0171] The sum of the power consumption of the first chip and the power consumption of the second chip does not exceed the upper limit of power consumption.

[0172] Please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating power management and data flow in a multi-core system, as provided in an embodiment of this application.

[0173] like Figure 3As shown, the multi-core system includes a tidal power management module, a first core, a second core, and a shared storage resource pool. The tidal power management module, located on the top of the package, controls the operating frequency and voltage of the first and second cores. The first core is labeled as a pre-filled core, and the second core is labeled as a decoding core; they communicate with each other via a high-speed inter-core physical interface. The shared storage resource pool, located below the first and second cores, stores key-value cache data and supports the transfer of key-value cache between different cores. Both the first and second cores can access the shared storage resource pool through their respective weights and data loading channels. The first core receives instruction input, and the second core generates decoding output.

[0174] In step S710, the power consumption limit refers to the power consumption constraint value determined by the multi-core system based on its heat dissipation design and power supply capability.

[0175] In one feasible implementation, the power consumption limit of the multi-core system can be read from the tidal power management module, which is pre-configured by the tidal power management module based on the package's heat dissipation and power supply capabilities.

[0176] In step S720, when the inference task enters the pre-filling stage, the tidal power management module can send a control command to the second chip to reduce the operating frequency or operating voltage, thereby adjusting the second chip to a low-power state. At the same time, it can send a control command to the first chip to increase the operating frequency or operating voltage, thereby adjusting the first chip to a high-performance state.

[0177] Combination Figure 3 As shown, during the pre-filling stage, the first core receives the instruction input and performs forward calculation, writing the generated key-value cache data into the shared storage resource pool through the data loading channel. At this time, the first core is in a high-power operating state, while the second core does not participate in the calculation. The operating frequency or operating voltage of the second core is reduced by the tidal power management module, so that the second core enters a low-power state.

[0178] In one feasible implementation, the power budget that can be allocated to the first chip can be calculated by the tidal power management module based on the power consumption limit and the computing load requirements of the pre-filling phase. Then, the operating frequency and operating voltage of the first chip are determined according to the power budget. The operating frequency of the second chip is reduced to the minimum operating frequency or the operating voltage is reduced to the minimum voltage of the holding state, so that the first chip can obtain a higher power budget to support high parallel computing.

[0179] In step S730, when the inference task switches from the pre-filling stage to the decoding stage, the tidal power management module can send a control command to the first chip to reduce the operating frequency or operating voltage, thereby adjusting the first chip to a low-power state. At the same time, it can send a control command to the second chip to increase the operating frequency or operating voltage, thereby adjusting the second chip to a high-performance state.

[0180] Combination Figure 3 As shown, during the decoding stage, the second chip reads key-value cache data from the shared storage resource pool through the data loading channel, performs autoregressive decoding calculations and generates decoding outputs. At this time, the second chip is in a high-power operating state, while the first chip has completed the pre-filling calculation. The operating frequency or operating voltage of the first chip is reduced by the tidal power management module, so that the first chip enters a low-power state.

[0181] In one feasible implementation, the power budget that can be allocated to the second chip can be calculated by the tidal power management module based on the power consumption limit and the memory access load requirements during the decoding stage. Then, the operating frequency and operating voltage of the second chip are determined according to the power budget. The operating frequency of the first chip is reduced to the minimum operating frequency or the operating voltage is reduced to the minimum voltage of the hold state, so that the second chip can obtain a higher power budget to support frequent memory access and autoregressive calculations.

[0182] Optionally, in steps S720 and S730, the adjustment of the operating frequency and the operating voltage can be performed simultaneously, or only the operating frequency or only the operating voltage can be adjusted. When only the operating frequency is adjusted, the operating voltage can be kept constant, and power consumption control can be achieved through frequency regulation; when only the operating voltage is adjusted, the operating frequency can be adjusted synchronously to maintain the stability of the chip operation.

[0183] By adopting the above-described technical solution in this embodiment, the power consumption limit of the multi-core system is obtained through the tidal power management module, and the operating frequency or operating voltage of the first and second cores are adjusted in the pre-filling and decoding stages, respectively. This allows the cores undertaking the computational tasks of the current inference stage to obtain a higher power budget and operating performance under the power consumption limit constraint, while the cores that are not currently involved in computation enter a low-power state to release the power budget. This achieves dynamic allocation of power budget between different inference stages, and, in conjunction with the shared storage resource pool, enables the flow of key-value cache data between different cores, thereby improving the computational performance and resource utilization efficiency of the multi-core system under a fixed power budget.

[0184] Based on the above embodiments, as an optional embodiment, the above-described large model phased inference method based on multi-core particle systems may further include the following steps.

[0185] Step S810: During the execution of the inference task, obtain the performance indicators of the first core and the second core.

[0186] Step S820: Determine whether the current resource configuration meets the preset performance requirements based on performance indicators;

[0187] Step S830: If the current resource configuration does not meet the preset performance requirements, perform a resource adjustment operation;

[0188] The resource adjustment operations include at least one of the following:

[0189] The power consumption quotas of the first and second chips are reallocated;

[0190] Adjust the storage space size of the target storage area;

[0191] A third core is added to the inference task. The third core is used to share the pre-filling computation of the first core or the decoding computation of the second core.

[0192] In step S810, the performance index refers to the parameters that reflect the working status and operating efficiency of the first and second cores during the inference task execution process.

[0193] In one feasible implementation, the performance indicators of the first and second cores can be periodically collected by the performance monitoring unit inside each core. The performance indicators include computing throughput, storage access latency, computing unit utilization, storage bandwidth utilization, or task execution time.

[0194] It should be noted that the performance metrics of the first core mainly reflect the computational efficiency of the pre-filling stage, including the parallel computing throughput and the write speed of key-value cache data; the performance metrics of the second core mainly reflect the operational efficiency of the decoding stage, including the read latency of key-value cache data and the response speed of word-by-word generation.

[0195] In step S820, the collected performance metrics can be compared with preset performance requirements to determine whether each performance metric has reached a preset threshold. If the preset performance requirements include a computing throughput requirement, it is determined whether the currently collected computing throughput has reached a preset throughput threshold; if the preset performance requirements include an access latency requirement, it is determined whether the currently collected storage access latency is lower than a preset latency threshold.

[0196] In step S830, if the current resource configuration does not meet the preset performance requirements, an appropriate resource adjustment operation can be selected based on the specific deviation of the performance indicators.

[0197] In the first type of resource adjustment operation, the reallocation of power consumption quotas for the first and second cores refers to adjusting the power consumption budget ratios allocated to the first and second cores while keeping the power consumption cap unchanged. If the computing throughput of the first core does not meet the preset requirements, the power consumption quota of the first core can be increased, while the power consumption quota of the second core can be decreased simultaneously. Conversely, if the access latency of the second core exceeds the preset requirements, the power consumption quota of the second core can be increased, while the power consumption quota of the first core can be decreased simultaneously.

[0198] In the second type of resource adjustment operation, adjusting the storage space size of the target storage region refers to increasing or decreasing the shared storage space capacity allocated to the target storage region based on the actual storage needs of the inference tasks. When the storage capacity of the key-value cache data is close to the upper limit of the target storage region, the storage space size of the target storage region can be increased to avoid performance degradation caused by insufficient storage space; when the actual usage of the key-value cache data is much smaller than the capacity of the target storage region, the storage space size of the target storage region can be reduced, and the freed storage space can be allocated to other inference tasks.

[0199] In the third resource adjustment operation, adding a third core to the inference task refers to selecting a third core from the idle cores of the multi-core system to join the execution of the inference task when the computing power of the first or second core is insufficient to meet the preset performance requirements. If the computational throughput in the pre-filling stage is insufficient, a third core can be selected to share the pre-filling computation of the first core, dividing the input sequence into multiple sub-sequences, with the first and third cores performing parallel forward computation on different sub-sequences respectively. If the response speed in the decoding stage is insufficient, a third core can be selected to share the decoding computation of the second core, distributing the decoding operations of multiple inference tasks to the second and third cores for parallel execution.

[0200] By adopting the above-described technical solution of this embodiment, the performance indicators of the first and second cores are obtained during the execution of the inference task. Based on the performance indicators, it is determined whether the current resource configuration meets the preset performance requirements. When the current resource configuration does not meet the preset performance requirements, resource adjustment operations, including reallocating power consumption quotas, adjusting the storage space size, or adding a third core, are performed. This achieves dynamic monitoring of the inference task execution process and adaptive adjustment of resource configuration.

[0201] Please refer to Figure 4 , Figure 4 This is a timing simulation waveform diagram of power consumption management and phase switching in a multi-core system provided in an embodiment of this application.

[0202] Based on the above embodiments, in order to further verify the effectiveness of the large model phased inference method based on multi-core particle system, this embodiment provides a timing simulation verification of hardware and software co-operation based on the analysis model.

[0203] like Figure 4 As shown, Figure 4 The upper part of the graph shows the power consumption variation curves, with the horizontal axis representing time (in milliseconds) and the vertical axis representing power consumption (in watts). The graph includes four curves: the pre-filled core cluster power consumption curve, the decoding core cluster power consumption curve, the system instantaneous total power consumption curve, and the thermal design power limit curve. During the pre-filling phase, the pre-filled core cluster power consumption curve remains at a high level, while the decoding core cluster power consumption curve remains at a low level. At the phase switching point (e.g., approximately 10 milliseconds on the time axis), the tidal power management module performs a power quota flipping operation, causing the pre-filled core cluster power consumption to plummet to a low level, while the decoding core cluster power consumption rises to a high level. Throughout the entire inference task execution, the system's instantaneous total power consumption remains consistently below the thermal design power limit.

[0204] like Figure 4 As shown, Figure 4 The lower half of the diagram is a digital logic timing waveform, comprising five timing signal lines: prefill computation status signal, direct memory access streaming write status signal, on-chip network interrupt handshake signal, decoding computation status signal, and shared memory resource in-situ read signal. During the prefill phase, the prefill computation status signal remains high, and the direct memory access streaming write status signal immediately follows, indicating that the key-value cache data write operation and the prefill computation operation overlap in time, hiding the data write latency. At the stage switching point, the on-chip network interrupt handshake signal generates a narrow pulse, indicating the completion of the prefill phase and notifying the decoding chip to begin the decoding phase. During the decoding phase, the decoding computation status signal periodically generates pulses, representing the word-by-word autoregressive computation process. The shared memory resource in-situ read signal pulses synchronously with the decoding computation status signal, indicating that the decoding chip reads key-value cache data from the shared memory resource during each autoregressive computation.

[0205] It should be noted that, as Figure 4 As shown, the direct memory access streaming write status signal and the pre-fill computation status signal largely overlap in time, indicating that the write operation of key-value cache data can be performed synchronously during the pre-fill computation, without waiting for the pre-fill computation to be completed before writing, thus reducing the overall latency of the pre-fill stage. The on-chip network interrupt handshake signal has an extremely narrow pulse width, indicating that only control signals need to be transmitted between chips during the stage switching process, without the need for large-scale data transmission, reducing the time overhead of stage switching.

[0206] It should be noted that, as Figure 4 As shown, during the decoding stage, the generation of the in-situ read signal of the shared storage resource indicates that the decoding chip directly reads the key-value cache data generated in the pre-filling stage from the shared storage resource. There is no need to move the key-value cache data from the pre-filling chip to the decoding chip, thus realizing a zero-copy data handover method.

[0207] Optionally, such as Figure 4 The simulation results show that by performing power quota flipping at the stage switching point through the tidal power management module, the instantaneous total power consumption of the system can always be kept below the thermal design power limit, avoiding the problem of total power consumption exceeding the limit caused by the pre-filled chip and the decoding chip being in a high-power state at the same time. By hiding the data write latency through direct memory access streaming write operation, realizing fast stage switching through narrow pulse interrupt handshake signal, and realizing zero-copy data handover through in-place read method, the computing resources can be fully utilized within the thermal design power limit, and the first word response latency of the inference task can be compressed.

[0208] pass Figure 4 The simulation verification shown demonstrates the effectiveness of the large-model phased inference method based on multi-core systems described in this embodiment in terms of power consumption management, data flow, and phase switching. It verifies that the method can achieve efficient execution and rapid switching between the pre-filling phase and the decoding phase within a strictly limited thermal design power range through reasonable hardware and software coordination, thereby improving the overall inference performance of the multi-core system.

[0209] This application also provides a large-model staged inference device based on a multi-core system, including:

[0210] Storage space;

[0211] A multi-core system, which consists of multiple cores that are interconnected and share storage space;

[0212] Processor; processes the large-model phased inference method based on multi-core systems in the above embodiments.

[0213] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A method for staged inference of large models based on multi-core particle systems, characterized in that, Applied to a multi-core system, the multi-core system comprising multiple cores interconnected and sharing storage space, the method includes: In response to the inference task of obtaining a large model, a first core and a second core are allocated among the plurality of cores; the first core is used to perform the pre-filling stage in the inference task; and the second core is used to perform the decoding stage in the inference task. Allocate a target storage area in the storage space; the target storage area is used to store the key-value cache generated when processing the inference task. Establish a first mapping relationship between the first core and the target storage region, and a second mapping relationship between the second core and the target storage region; When the inference task is in the pre-filling stage, the inference task is forward computed through the first core, and the generated key-value pairs are stored and written into the target storage area based on the first mapping relationship. In the decoding stage of the inference task, the second core reads the key-value cache in the target storage area according to the second mapping relationship, and performs decoding calculations based on the key-value cache to obtain the processing result of the inference task.

2. The method of claim 1, wherein, The allocation of the first core and the second core among the plurality of cores includes: Obtain the task feature information of the reasoning task; the task feature information includes the length of the input text and the expected length of the output text of the reasoning task; Obtain the system status information of the multi-core system; the system status information includes the core utilization rate, request queuing depth, storage space occupancy rate, and power consumption status of the multiple cores; Based on the task feature information and the system status information, assess the first resource requirement of the inference task in the pre-filling stage and the second resource requirement in the decoding stage; Based on the first resource requirement, a first core particle is allocated among the plurality of core particles; The second core is allocated among the plurality of cores based on the second resource requirement.

3. The method of claim 1, wherein, The allocation of the target storage region in the storage space includes: Obtain the model structure information of the large model; the model structure information includes the number of model layers, the number of attention heads, and the data precision; Based on the model structure information and the length of the input text for the inference task, calculate the key-value cache capacity required for the inference task; Based on the key-value cache capacity, the corresponding target storage area is divided in the storage space.

4. The method of claim 1, wherein, The first mapping relationship includes at least one of the following: The address mapping relationship between the virtual address space of the first chip and the physical address space of the target storage region; The access path mapping relationship from the first chip to the target storage area; the access path mapping relationship is used to determine the data transmission path of the first chip to access the target storage area through the chip interconnect network. The write permission mapping relationship of the first chip to the target storage area; The write permission mapping relationship is used to identify the writable status of each storage location in the target storage area for the first chip.

5. The method of claim 4, wherein, The step of performing forward computation on the inference task through the first core and storing and writing the generated key-value pairs into the target storage area based on the first mapping relationship includes: The model weights of the large model are read through the first core, and forward computation is performed on the input lexical units of the inference task based on the model weights to generate key-value cache data corresponding to the attention computation of each layer. Based on the address mapping relationship in the first mapping relationship, determine the target write address of the key-value cache data in the target storage area; Based on the access path mapping relationship in the first mapping relationship, the data transmission path of the first chip to access the target storage area through the chip interconnect network is determined; The key-value cache data is written to the target write address via the data transmission path through the storage access engine within the first core; the storage access engine is used to stream the generated key-value cache data to the target storage area while the first core performs forward computation.

6. The method of claim 1, wherein, The second mapping relationship includes at least one of the following: The address mapping relationship between the virtual address space of the second core and the physical address space of the target storage region; The access path mapping relationship from the second chip to the target storage area; the access path mapping relationship is used to determine the data transmission path of the second chip to access the target storage area through the chip interconnect network; The second chip's read permission mapping relationship to the target storage area; The read permission mapping relationship is used to identify the readable status of each storage location in the target storage area for the second core.

7. The method of claim 6, wherein, The method further includes: Based on the address mapping relationship in the second mapping relationship, determine the target read address of the key-value cache data stored in the target storage area; Based on the access path mapping relationship in the second mapping relationship, the data transmission path of the second chip to access the target storage area through the chip interconnect network is determined; The step of reading the key-value cache in the target storage area according to the second mapping relationship through the second core, and performing decoding calculations based on the key-value cache to obtain the processing result of the inference task includes: The key-value cache data is read from the target read address via the second core through the data transmission path; The second core performs autoregressive decoding calculations based on the key-value cache data, generating output tokens one by one to obtain the processing result of the inference task.

8. The method of claim 1, wherein, The method further includes: Obtain the power consumption limit of the multi-core system; the power consumption limit is the maximum power consumption value allowed to be consumed by the multi-core system under normal operating conditions. When the inference task is in the pre-filling stage, the operating frequency or operating voltage of the second chip is reduced based on the power consumption limit, and the operating frequency or operating voltage of the first chip is increased, so that the power consumption of the first chip is greater than that of the second chip. When the inference task switches to the decoding stage, the operating frequency or operating voltage of the first chip is reduced based on the power consumption limit, and the operating frequency or operating voltage of the second chip is increased, so that the power consumption of the second chip is greater than that of the first chip. Wherein, the sum of the power consumption of the first chip and the power consumption of the second chip does not exceed the power consumption limit.

9. The method of claim 1, wherein, The method further includes: During the execution of the inference task, the performance indicators of the first core and the second core are obtained; Determine whether the current resource configuration meets the preset performance requirements based on the performance indicators; If the current resource configuration does not meet the preset performance requirements, a resource adjustment operation will be performed. The resource adjustment operation includes at least one of the following: The power consumption quotas of the first chip and the second chip are reallocated; Adjust the storage space size of the target storage area; A third core is added to the inference task, which is used to share the pre-filling computation of the first core or the decoding computation of the second core.

10. A multi-core system based large model staged inference apparatus, comprising: include: Storage space; A multi-core system, comprising multiple cores interconnected and sharing the storage space; Processor; the processing is used to execute the large model phased inference method based on a multi-core system as described in any one of claims 1 to 9.

Images