Task processing apparatus, method and computing device

Abstract

The application discloses a task processing device, method and computing equipment, relates to the field of computer storage and natural language processing, and comprises a computing unit, which performs a multi-round computing task according to a target embedding vector of an input word element; a first storage unit used for storing embedding vectors of a plurality of word elements determined based on word element usage times; and a cache management unit configured to read the target embedding vector from the first storage unit and send the target embedding vector to the computing unit in the case that the target embedding vector is stored in the first storage unit; otherwise, determine a target storage unit and a storage address in which the target embedding vector is stored, and send the storage address to the computing unit. The technical problem of how to realize a computing task and improve computing speed in a limited resource scenario of an edge side device is solved, and the technical effect of reducing data access delay without expanding static storage resources in the device is achieved by the task processing device independently completing a whole-process reasoning.

Description

Technical Field

[0001] This application relates to the fields of computer storage and natural language processing technology, and in particular to a task processing apparatus, method and computing device. Background Technology

[0002] With the continuous development of computer technology, computing devices at the edge can deploy hardware acceleration architectures to improve the inference speed of computing tasks such as large language models. For example, edge devices can deploy hardware acceleration frameworks based on field programmable gate arrays (FPGAs).

[0003] In related technologies, in order to avoid the complex control flow and on-chip memory capacity limitations of hardware-accelerated architectures, many design schemes adopt a heterogeneous computing model. That is, the host's central processing unit (CPU) or coprocessor, such as the graphics processing unit (GPU), is responsible for performing the computationally intensive pre-filling stage. After that, the hardware-accelerated architecture (such as FPGA) is responsible for the memory bandwidth-constrained autoregressive decoding stage.

[0004] However, in scenarios where computing resources are severely limited for edge devices, these devices often only contain hardware architectures such as FPGAs and limited memory. Heterogeneous computing models are difficult to apply to resource-constrained computing devices, making it difficult to achieve LLM computing tasks at the edge. Summary of the Invention

[0005] This application provides a task processing apparatus, method, and computing device to at least solve the technical problems in the related art of how to implement computing tasks and how to improve the computing speed of computing tasks in scenarios where edge-side device resources are limited.

[0006] This application provides a task processing apparatus comprising: a computing unit configured to perform multiple rounds of computing tasks based on target embedding vectors of input lexical units and output predicted lexical units, wherein the input lexical units of the computing tasks include at least one of lexical units obtained by lexical transformation of user input text and predicted lexical units output from the previous round of computing tasks; a first storage unit configured to store embedding vectors of multiple lexical units determined based on the number of times lexical units are used; and a cache management unit configured to: read the target embedding vector from the first storage unit when the target embedding vector is stored in the first storage unit and send the target embedding vector to the computing unit; and determine the target storage unit and storage address where the target embedding vector is stored when the target embedding vector is not stored in the first storage unit and send the storage address to the computing unit, wherein the computing unit reads the target embedding vector from the target storage unit according to the storage address, and the capacity of the target storage unit is greater than the capacity of the first storage unit.

[0007] This application also provides a task processing method applied to the task processing apparatus described above. The method includes: using the computing unit of the task processing apparatus to perform multiple rounds of computation tasks based on the target embedding vector of the input lexical units, and outputting predicted lexical units, wherein the input lexical units of the computation tasks include at least one of the lexical units obtained after lexical transformation of the user input text and the predicted lexical units output by the previous round of computation tasks; using the cache management unit of the task processing apparatus to read the target embedding vector from the first storage unit when the target embedding vector is stored in the first storage unit, and send the target embedding vector to the computing unit; and when the target embedding vector is not stored in the first storage unit, determining the target storage unit and storage address where the target embedding vector is stored, and sending the storage address to the computing unit, wherein the computing unit reads the target embedding vector from the target storage unit according to the storage address, and the capacity of the target storage unit is greater than the capacity of the first storage unit.

[0008] This application also provides a computing device, including: the task processing apparatus as described above; at least one of a third storage unit and a fourth storage unit, configured to store embedding vectors of a plurality of lexical units, such that the task processing apparatus, when the first storage unit in the task processing apparatus does not include a target embedding vector, obtains the target embedding vector from at least one of the third storage unit and the fourth storage unit and performs a computing task using the target embedding vector.

[0009] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described task processing method.

[0010] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described task processing methods.

[0011] In this embodiment, the task processing device includes end-to-end components for implementing computational tasks, such as a cache management unit for managing the reading of embedding vectors, a first storage unit for storing a portion of the embedding vectors within the task processing device with limited resources, and a computation unit for computation. The entire process is independent of external processors, thus enabling the task processing device to independently complete the entire LLM inference process. Furthermore, using the first storage unit to store the embedding vectors determined based on the number of times each term is used, and having the cache management unit directly read the target embedding vector for computation from the first storage unit and return it to the computation unit, both minimize data access latency and reduce the impact on computational efficiency without expanding the static storage resources within the task processing device. Moreover, even when the first storage unit does not store the target embedding vector, the computation unit can still read the target embedding vector based on the storage address returned by the cache management unit, ensuring the smooth implementation of computational tasks with limited resources. Attached Figure Description

[0012] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 An architectural diagram of a task processing apparatus according to an embodiment of this application is shown schematically.

[0014] Figure 2 A schematic diagram illustrating address mapping information according to an embodiment of this application is shown.

[0015] Figure 3 The diagram illustrates an embedded vector read process under a four-level storage architecture according to an embodiment of this application.

[0016] Figure 4 The illustration schematically shows a data flow diagram of the embedded vectors stored in multiple storage units under a dynamically adjusted four-level storage architecture according to an embodiment of this application.

[0017] Figure 5 The schematic diagram illustrates a structure for parallel reading of multiple sub-embedded vectors according to an embodiment of this application.

[0018] Figure 6 A schematic diagram of the framework of a task processing apparatus according to an embodiment of this application is shown.

[0019] Figure 7 A flowchart illustrating a task processing method according to an embodiment of this application is shown schematically. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0021] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0022] With the continuous development of computer technology, large language models are widely used in natural language understanding, content generation and intelligent interaction, and their deployment needs are gradually shifting from cloud data centers to edge devices.

[0023] Heterogeneous computing models based on CPU / GPU and FPGA allocate computationally intensive tasks to the CPU / GPU for pre-processing before transmitting them to the FPGA. However, this "pre-processing-transmission" approach has two problems: First, it leads to significant and unavoidable data migration overhead. Moving the complete key-value cache from host memory to the FPGA's onboard memory via the high-speed Peripheral Component Interconnect Express (PCIe) bus not only consumes system bus bandwidth but also introduces transmission latency, increasing end-to-end inference latency. Second, this architecture heavily relies on the host CPU / GPU processors for pre-processing. In scenarios where computing resources are severely limited for edge devices, these devices often only contain hardware architectures such as FPGAs and limited memory. Heterogeneous computing models are difficult to apply to resource-constrained computing devices, making it difficult to implement LLM computing tasks at the edge.

[0024] Specifically, one of the challenges of deploying LLMs in computing devices (such as embedded FPGAs) is the high latency and bandwidth pressure caused by frequent accesses to external storage by the token embedding layer. In heterogeneous computing models or other related technologies, all embeddings are typically stored entirely in off-chip DDR memory or low-speed external SD cards. Each FPGA inference requires reading these embeddings through bandwidth-limited channels, and the access latency becomes a constraint on system performance. Especially during the decoding stage, the random access pattern of the FPGA's autoregressive generation process to embeddings makes data prefetching difficult. Simply expanding the FPGA's on-chip static cache to cache all parameters and vectors on-chip in pursuit of performance would further strain already limited logic and storage resources, impacting the hardware implementation of other computational modules in the model (such as attention mechanisms and feedforward networks).

[0025] To this end, this application proposes an on-chip cache architecture and dynamic scheduling method specifically for this layer, so as to enable computing devices to independently and efficiently complete the entire LLM inference process, including the pre-filling stage, within the task processing device (such as FPGA) when CPU / GPU is not included or is not used.

[0026] This application provides a task processing apparatus, comprising: a computing unit configured to perform multiple rounds of computing tasks based on target embedding vectors of input lexical units and output predicted lexical units, wherein the input lexical units of the computing tasks include at least one of lexical units obtained by lexical transformation of user input text and predicted lexical units output from the previous round of computing tasks; a first storage unit for storing embedding vectors of multiple lexical units determined based on the number of times lexical units are used; and a cache management unit configured to: read the target embedding vector from the first storage unit and send the target embedding vector to the computing unit when the target embedding vector is stored in the first storage unit; and determine the target storage unit and storage address where the target embedding vector is stored and send the storage address to the computing unit when the target embedding vector is not stored in the first storage unit, wherein the computing unit reads the target embedding vector from the target storage unit according to the storage address, and the capacity of the target storage unit is greater than the capacity of the first storage unit.

[0027] In this embodiment, the task processing device includes end-to-end components for implementing computational tasks, such as a cache management unit for managing the reading of embedding vectors, a first storage unit for storing a portion of the embedding vectors within the task processing device with limited resources, and a computation unit for computation. The entire process is independent of external processors, thus enabling the task processing device to independently complete the entire LLM inference process. Furthermore, using the first storage unit to store the embedding vectors determined based on the number of times each term is used, and having the cache management unit directly read the target embedding vector for computation from the first storage unit and return it to the computation unit, both minimize data access latency and reduce the impact on computational efficiency without expanding the static storage resources within the task processing device. Moreover, even when the first storage unit does not store the target embedding vector, the computation unit can still read the target embedding vector based on the storage address returned by the cache management unit, ensuring the smooth implementation of computational tasks with limited resources.

[0028] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] The specific application environment architecture or specific hardware architecture on which the execution of the task processing device and method depends is described herein.

[0030] Figure 1 A schematic diagram of a task processing apparatus according to an embodiment of this application is shown. Figure 1 As shown, the task processing device 110 includes a computing unit 101, a cache management unit 102, and a first storage unit 103.

[0031] The computation unit 101 is configured to perform multiple rounds of computation tasks based on the target embedding vectors of the input lexical units and output predicted lexical units. The input lexical units for the computation tasks include at least one of the lexical units obtained after lexical transformation of the user input text and the predicted lexical units output from the previous round of computation. The first storage unit 103 is used to store the embedding vectors of multiple lexical units determined based on the number of times each lexical unit is used. The cache management unit 102 is configured to: when the target embedding vector is stored in the first storage unit, read the target embedding vector from the first storage unit and send the target embedding vector to the computation unit; when the target embedding vector is not stored in the first storage unit, determine the target storage unit and its storage address where the target embedding vector is stored, and send the storage address to the computation unit. The computation unit reads the target embedding vector from the target storage unit according to the storage address, and the capacity of the target storage unit is greater than the capacity of the first storage unit.

[0032] The input and predicted terms refer to each round of computation performed by the computation unit. The input terms may be the same across multiple rounds of computation, the predicted terms may be the same across multiple rounds of computation, and the input and output terms may be the same across multiple rounds of computation, depending on the computation task performed by the computation unit.

[0033] During the first round of computation using the task processing device, the input terms may only include terms obtained after lexical transformation of the user input text. In subsequent rounds of computation, the input terms may include terms obtained after lexical transformation of the user input text, as well as the predicted terms output from the previous round. In one specific embodiment, the computation unit is configured to perform an autoregressive computation task, which can predict what the next term could be based on all input terms and output the predicted term as the predicted term.

[0034] For example, the user input text could be "I like to eat". After performing word segmentation and inputting it into the computing unit, the first predicted word segment could be "apple". After putting "apple" back into the original input word segment, the next round of input word segments is obtained, such as the input word segment corresponding to "I like to eat apple". At this time, with the support of "apple", the computing unit knows that the next word segment is highly likely to be "fruit".

[0035] When the computing unit performs multi-round computing tasks, it needs to use the embedding vectors of the input words to perform the calculations, and these embedding vectors are pre-calculated and stored.

[0036] The embedding vectors stored in the first storage unit 103 and the target storage unit are pre-calculated. The specific embedding vectors stored in the first storage unit 103 and the target storage unit can be dynamically changing or static. For example, the specific embedding vector in the target storage unit can be dynamically changing or static. Given the limited storage resources within the task processing device 110, the embedding vectors stored in the first storage unit 103 are dynamically changing, determined based on the number of times the word is used. The embedding vectors stored in the first storage unit may or may not include the target embedding vectors of the input words required for the current round of computation. The number of times a word is used characterizes the number of times each word or the embedding vector of a word is used during the execution of multiple rounds of computation tasks.

[0037] The target storage unit can be located either inside or outside the task processing device 110. The location of the target storage unit can be flexibly determined based on the resource constraints of the task processing device 110. For example, if the task processing device 110 has ample internal storage resources, both the target storage unit and the first storage unit 103 can be located inside the task processing device 110; conversely, the target storage unit can be located outside the task processing device 110. It is understood that even if the target storage unit is located outside the task processing device 110, this portion can be configured as a dedicated storage unit for the task processing device 110, and independent computation can be performed without the need for an external processor.

[0038] The target storage unit has a larger capacity than the first storage unit. Therefore, the amount of embedding vector data stored in the target storage unit can be greater than that stored in the first storage unit. This means that (if the capacity of the target storage unit allows), the embedding vector of a certain word not stored in the first storage unit can be stored in the target storage unit. Thus, even if the on-chip static resources of the task processing device 110 (such as the capacity of the first storage unit) are limited, the task processing device 110 can still complete the calculation using the embedding vectors stored in the target storage unit.

[0039] In this embodiment, before the computing unit performs calculations using the target embedding vectors of each input word, the cache management unit 102 determines the storage unit where each input word is located. In the case of the first storage unit 103 in the task processing device 110 (e.g., within the FPGA chip), the target embedding vector is directly read and transmitted to the computing unit. Thus, the computing unit 101 can directly perform calculations using the target embedding vector. This direct reading design allows the target embedding vector to be calculated to be sent to the computing unit in only 1-2 FPGA clock cycles. However, the embedding vector stored in the first storage unit 103 is dynamic. If the first storage unit 103 does not store the target embedding vector, the cache management unit responsible for storage can determine the target storage unit and its storage address where the target embedding vector is located. Then, the computing unit 101 can obtain the target storage unit and its storage address from the cache management unit 102, and read the target embedding vector from the target storage unit according to the storage address to perform the current round of calculation.

[0040] The computing unit 101 can be a pre-packaged, reusable hardware functional module used to perform specific computational tasks. For example, if the task processing device 110 is an FPGA, the computing unit 101 can be an Intellectual Property Core (IP) core used for computation. It is understood that there can be one or more computing units 101. If multiple units exist, they can pipeline the computational tasks at different stages of each round of computation, or pipeline the computational tasks in different rounds of multiple rounds. The specific parallel or serial task design can be determined according to actual needs. In one embodiment, the computing unit 101 can communicate with the target storage unit through an interface. The computing unit 101 can obtain the target embedding vector according to the storage address through this interface. For example, this interface can be an Advanced eXtensible Interface (AXI), such as the AXI4 interface.

[0041] The first storage unit 103 may be an inherent hardware resource of the task processing device 110. For example, the first storage unit 103 may be a block random access memory (BRAM). The cache management unit 102 may be a pre-packaged hardware unit used to perform address mapping and vector acquisition functions.

[0042] In this embodiment, the task processing device includes end-to-end components for implementing computational tasks, such as a cache management unit for managing the reading of embedding vectors, a first storage unit for storing a portion of the embedding vectors within the task processing device with limited resources, and a computation unit for computation. The entire process is independent of external processors, thus enabling the task processing device to independently complete the entire LLM inference process. Furthermore, using the first storage unit to store the embedding vectors determined based on the number of times each term is used, and having the cache management unit directly read the target embedding vector for computation from the first storage unit and return it to the computation unit, both minimize data access latency and reduce the impact on computational efficiency without expanding the static storage resources within the task processing device. Moreover, even when the first storage unit does not store the target embedding vector, the computation unit can still read the target embedding vector based on the storage address returned by the cache management unit, ensuring the smooth implementation of computational tasks with limited resources.

[0043] According to an embodiment of this application, the cache management unit is configured to determine whether the target embedding vector is stored in the first storage unit by: determining the address mapping information matching the input lexical based on the lexical identifier of the input lexical; reading the first address identifier bit from the address mapping information based on the first address range of the address mapping information; and determining that the target embedding vector is stored in the first storage unit if the storage unit indicated by the first address identifier bit is the first storage unit.

[0044] The address mapping table is used to maintain the correspondence between each token and the storage location information of its corresponding embedding vector. This storage location information includes storage units and / or storage addresses. For example, the core entries in the address mapping table are the token identifier (token_id) and its storage location information. The token_id is used to uniquely identify each input token.

[0045] Multiple address mapping entries in the address mapping table correspond to multiple tokens. Each address mapping entry is divided into multiple address ranges (i.e., multiple bit segments), and a specific address range is used to store specific information. The address mapping information may include token identifiers and a first address identifier bit. For example, the first address range is used to store the first address identifier bit, and another part of the address range is used to store the token_id.

[0046] In one specific embodiment, the token_id is represented using 16 bits to satisfy the requirement of all tokens supported by the LLM deployed in the computing unit. For example, if the tokenizer of a certain LLM contains 50,257 different token_ids, the reserved 16 bits can support 2 to the power of 16, a total of 65,536 token_ids. Since 65,536 > 50,257, the token_id entry in the address mapping table can meet the needs of most current models, such as the Transformer model.

[0047] The first address identifier bit is used to indicate the storage unit containing the embeddings of each token. The first address identifier bit can be represented using 2 bits. For example, taking a four-level cache as an example (the first storage unit is considered a level 1 cache, and the target storage unit can be any level in a three-level cache), 01, 02, 03, and 04 represent the storage units corresponding to the four-level cache, such as 01 representing the first storage unit. It is understood that a four-level cache is only an example; if there are two, three, or five or more levels of cache, it can all be implemented using 2 bits or more.

[0048] Upon reading the token_id of the input token, the cache management unit can determine the address mapping information matching the input token based on the specific storage value of the address range used to store the token_id in multiple address mapping information entries. After determining the address mapping information matching the token_id, it can directly read the first address identifier bit (such as 01, 02, 03, or 04 above) stored in the first address range within the address mapping information. Then, based on the information specifically expressed by the first address identifier bit, it determines whether the target embedding vector is stored in the first storage unit. For example, if the first address identifier bit is 01, the target embedding vector is stored in the first storage unit; otherwise, the target embedding vector is not stored in the first storage unit.

[0049] In the embodiments of this application, various information is stored through different address ranges of the address mapping table. When it is necessary to determine the target embedding vector of the token_id of the input token, the first address identifier bit is directly extracted from the first address range of the address mapping information that matches the token_id. The target embedding vector can be quickly located through simple addressing logic to determine whether it is stored in the first storage unit, thereby improving the speed of finding the target embedding vector and thus improving the overall task processing speed of the task processing device.

[0050] According to an embodiment of this application, the cache management unit is configured to determine the target storage unit and storage address storing the target embedding vector in the following manner: if the storage unit indicated by the first address identifier bit is not the first storage unit, the storage unit indicated by the first address identifier bit is taken as the target storage unit; according to the second address range of the address mapping information, the second address identifier bit is read from the address mapping information, and the address indicated by the second address identifier bit is determined as the storage address.

[0051] The address mapping information can also include a second address range to store the specific storage address of each embedding vector in each storage unit. This storage address can be indicated by a second address identifier bit. For example, the second address identifier bit can use 30 bits to represent the storage address of the current token_id in the target storage unit or the first storage unit.

[0052] For example, if the first address flag is 02, and it does not indicate the first memory cell (01), the memory cell corresponding to 02 can be directly used as the target memory cell. Similarly, if the first address flag is 03 or 04, the memory cell corresponding to 03 or 04 can be directly used as the target memory cell. Then, the second address flag is retrieved from the second address range, and the specific information indicated by the second address flag is used as the storage address.

[0053] It is understandable that if the target embedding vector is stored in the first storage unit, the second address flag of the address mapping information can store the storage address of the target embedding vector in the first storage unit to record the storage address of the target embedding vector.

[0054] In this embodiment, the cache management unit can associate the storage address of the target embedding vector in the first storage unit with the token_id based on a special design. This allows the target embedding vector to be retrieved directly from the token_id without needing to read the address from the second address range when it is stored in the first storage unit. For example, the token_id and the target embedding vector may have the same storage address in the first storage unit.

[0055] In the embodiments of this application, the storage unit where the target embedding vector is stored can be quickly determined by a single read operation on the first address identifier (a truncation operation of a specific address range). If the target embedding vector is located in the first storage unit, it can be directly read, improving the access speed. If the target embedding vector is not stored in the first storage unit, the storage address of the target embedding vector is quickly determined by a second read operation on the second address identifier (a second truncation operation of a specific address range). The entire process only requires two address read operations within a specific address range to locate and read the target embedding vector, significantly reducing the time spent on address lookup while ensuring storage efficiency, thus meeting the low-latency access requirements of computational tasks in large model inference scenarios.

[0056] Figure 2 A schematic diagram illustrating address mapping information according to an embodiment of this application is shown. Figure 2 As shown, the address mapping information is divided into four address ranges, from right to left: the third address range, the fourth address range, the first address range, and the second address range. These ranges are used to store the token_id, the original address, the first address identifier, and the second address identifier, respectively. The cache management unit can extract the necessary information from the information stored in each preset address range.

[0057] exist Figure 2In this context, the storage address indicated by the second address identifier can be considered as the transformed and mapped address under the multi-level caching architecture of the first storage unit and the target storage unit in this application, i.e., the address actually stored in a certain storage unit. The original address is 32 bits, used to indicate the address when storing all token_ids in a specific storage unit. It can be understood that, in order to ensure the accessibility of all token_ids and their embedding vectors, the full set of embedding vectors can be stored in a fixed storage unit, and this information can be stored in the address mapping table. Thus, if the target embedding vector in the first storage unit or the target storage unit is not accessible, the target embedding vector can be obtained based on the original address.

[0058] According to an embodiment of this application, the task processing apparatus further includes: at least one counter configured to: count the input words used by the computing unit in performing multiple rounds of computing tasks to obtain the word usage count of each input word; the cache management unit is further configured to: obtain the word usage count of each input word from the at least one counter; update the embedding vector of multiple words stored in the first storage unit using the embedding vector of multiple words stored in the target storage unit according to the word usage count of each input word; and update the address mapping table according to the updated embedding vector in the first storage unit.

[0059] The counter can be a hardware-implemented counting module used to track the number of times each input word is used in multiple rounds of computation. In one embodiment, a counter can be set for each word, and the counter is updated once for each word used as an input word in the computation process. Alternatively, a counter can be set for multiple words, and the counter corresponding to each word is updated once for each word used as an input word in the computation process.

[0060] As the computation unit performs a round of computation, the number of times the input words are used changes accordingly. The embedding vector stored in the first storage unit, which has limited storage resources, also changes with the number of times the words are used. This change in the embedding vector stored in the first storage unit can be controlled by the cache management unit, and this change is recorded through dynamic address mapping information.

[0061] For example, in a certain round of computation, the input word is "I like to eat apples because apples are rich in vitamin C, and also, apples". Each time "apple" is used in multiple rounds of computation, a counter counts the number of times the word is used, resulting in 3 uses. If the embedding vector of "apple" was originally in the target storage unit, then when the word is used 3 times, the embedding vector of "apple" in the target storage unit can be read out and stored in the first storage word, thus updating the embedding vectors of multiple words stored in the first storage unit. Simultaneously, since the storage unit and storage address of the embedding vector have changed, the address mapping information matching the token_id can be determined from the address mapping table based on the token_id of "apple". The first and second address identifier bits in the address mapping information are then modified to 01 and token_id, respectively. The storage address of the word in the first storage unit is the same as the token_id.

[0062] In this embodiment, by maintaining a counter for the token in the task processing device and dynamically adjusting the embedding vector and address mapping table stored in the first storage unit according to the number of times the token's tokens are used, the first storage unit always retains the embedding vectors of tokens with high usage or access frequency. Thus, when performing multi-round computation tasks, frequently accessed tokens can be directly read from the first storage unit without accessing the target storage unit with lower access speed. This maximizes the reduction of access latency within the limited high-speed storage capacity.

[0063] According to an embodiment of this application, the cache management unit is further configured to update the embedding vectors of multiple words stored in the first storage unit based on the word usage count of each input word word using the embedding vectors of multiple words stored in the target storage unit: reading the embedding vectors of input words whose word usage count exceeds a threshold from the target storage unit and writing them to the first storage unit; reading the embedding vectors of input words whose word usage count does not exceed the threshold from the first storage unit and writing them to the target storage unit.

[0064] The usage threshold can be set according to actual needs, such as 5 times, 10 times, etc. The comparison between the word usage count and the threshold can be implemented using a comparator. For example, the cache management unit may include this comparator, with one end connected to the output of a counter and the other end connected to the output of a usage threshold register. The word usage count obtained from the counter is compared with the preset usage threshold stored in the register, and the comparison result is output. Based on the comparison result, the cache management unit writes the corresponding word embedding vector into the first storage unit or the target storage unit, completing the dynamic adjustment of the cached content. Alternatively, the above comparison operation can also be implemented using other hardware devices.

[0065] For example, if the number of times an input token is used exceeds (e.g., greater than) a threshold, the input token is reclassified from a "non-hotspot token" to a new "hotspot token"; conversely, if the number of times an input token is used does not exceed (e.g., less than or equal to) the threshold, the input token is reclassified from a "hotspot token" to a "non-hotspot token". For input tokens reclassified as new "hotspot tokens", the cache management unit migrates their embedding vectors from the target storage unit to the faster first storage unit. For input tokens reclassified as "non-hotspot tokens", to make room for new hotspot data, the old data that has become "non-hotspot" in the first storage unit is written back to the target storage unit.

[0066] In the embodiments of this application, the capacity of the first storage unit is limited due to the constraints of static storage resources within the task processing device. Therefore, the embedding vectors in the first storage unit are usually subject to dynamic write and delete operations in order to balance the access speed of the embedding vectors to the internal storage under the capacity constraints.

[0067] In some embodiments, if the first storage unit includes multiple input tokens whose usage count does not exceed a threshold, the embedding vectors of all or part of the input tokens whose usage count does not exceed the threshold can be written back to the target storage unit from the first storage unit. For example, the embedding vector of the input token with the lowest usage count among the input tokens whose usage count does not exceed the threshold can be written back to the target storage unit. Alternatively, the target number of input tokens whose usage count exceeds the threshold can be determined in the target storage unit, and the target number of input tokens with the lowest usage count among the input tokens whose usage count does not exceed the threshold can be determined. The embedding vectors of these input tokens can be written back to the target storage unit, thereby achieving the replacement of the embedding vectors stored in the target embedding unit and the first storage unit based on the usage count of the tokens. Each data migration requires synchronous updating of the address mapping table to ensure that the next lookup of the token_id can directly locate the new and faster storage address.

[0068] In other embodiments, when the embedded vectors are read from the target storage unit and written to the first storage unit, the embedded vectors that have undergone data migration can be deleted from the target storage unit, or they can remain in place. In one specific embodiment, it can be determined whether to delete the embedded vectors that have undergone data migration based on the relative position of the target storage unit and the task processing device and / or the capacity of the target storage unit.

[0069] For example, when the target storage unit is located within the task processing device, embedding vectors that have undergone data migration can be deleted. This allows for the progressive retrieval of embedding vectors through multi-level caching within the task processing device without requiring an increase in static storage resources. Conversely, when the target storage unit is located outside the task processing device, embedding vectors that have undergone data migration may not be deleted, or may be partially deleted. Alternatively, when the capacity of the target storage unit is less than a capacity threshold, embedding vectors that have undergone data migration can be deleted to balance efficient retrieval and access speed within the constraints of limited storage resources on computing devices (such as devices including task processing devices). Conversely, when the capacity of the target storage unit is greater than or equal to the capacity threshold, embedding vectors that have undergone data migration may not be deleted, or may be partially deleted, reducing the overall read / write resource consumption and further reducing access latency.

[0070] According to an embodiment of this application, the cache management unit is further configured to: read the target embedding vector from the target storage unit and write it to the first storage unit; and when the computing unit performs the next round of computing tasks, read the target embedding vector from the first storage unit and send the target embedding vector to the computing unit.

[0071] When a target embedding vector stored in the target storage unit is read out, it means that the target embedding vector is either used as part of the input words in the user's input text or as a predicted word in the current round. In both cases, it will be used as an input word in subsequent rounds of computation. Therefore, when reading the target embedding vector from the target storage unit and transmitting it to the computation unit (which uses the target embedding vector for this round of computation), it can also be written to the first storage unit simultaneously. It can be understood that when the computation unit reads the target embedding vector from the target storage unit and performs the computation task, the word usage count is incremented by one. Therefore, the operation of the cache management unit writing the target embedding vector to the first storage unit is also an operation of adjusting the embedding vector in the first storage unit based on the word usage count.

[0072] For example, if the target embedding vector does not exist in the first storage unit (cold data or has been swapped out), the address mapping table provides the computing unit with the storage address of the target embedding vector in the target storage unit, and the computing unit retrieves the target embedding vector from the storage address of the target storage unit. At the same time, the cache management unit also triggers a fill request from the target storage unit to the first storage unit to read the target embedding vector from the target storage unit and write it to the first storage unit.

[0073] In this embodiment, the cache management unit reads the target embedding vector from the target storage unit according to the logic of each round of calculation unit, and at the same time realizes the migration of the embedding vector from the target storage unit to the first storage unit, thereby improving the subsequent data reading speed.

[0074] In one specific embodiment, the cache management unit can dynamically adjust the embedding vector in the first storage unit by combining the two strategies mentioned above.

[0075] For example, if "banana" is a predicted word in the i-th round and is stored in the target storage unit, where i is a positive integer greater than 0, when the computation unit reads the target embedding vector corresponding to "banana" from the target storage unit in the (i+1)-th round and performs computation, the cache management unit stores the target embedding vector corresponding to "banana" into the first storage unit. At the same time, the cache management unit reads the embedding vectors of input words whose usage frequency has not exceeded the threshold from the first storage unit and writes them to the target storage unit, thus realizing the substitution of "banana" with other non-hotspot tokens in the first storage unit.

[0076] According to embodiments of this application, the task processing device further includes a second storage unit, and the computing device where the task processing device is located further includes a third storage unit and a fourth storage unit disposed outside the task processing device and inside the computing device, wherein the capacities of the second storage unit, the third storage unit, and the fourth storage unit increase sequentially; the second storage unit and the third storage unit are configured to store embedding vectors of multiple words determined based on the number of times the words are used, and the fourth storage unit is configured to store a fixed number of embedding vectors of multiple words. The target storage unit is at least one of the second storage unit, the third storage unit, and the fourth storage unit.

[0077] For example, still using Figure 2 Taking the address mapping information in the memory as an example, if the second address flag is 02, the target memory unit is the second memory unit; if the second address flag is 03, the target memory unit is the third memory unit; if the second address flag is 04, the target memory unit is the fourth memory unit.

[0078] In this embodiment, the capacity of the first to fourth storage units increases sequentially, and the number of word embedding vectors that can be stored in each unit also gradually increases. In one specific embodiment, the fourth storage unit can store all word embedding vectors supported by the LLM implemented by the computing unit of the task processing device, and it stores all word embedding vectors at a fixed value during the execution of multiple rounds of computing tasks, supporting only reading embedding vectors and not writing to them. The embedding vectors stored in the first to third storage units can all be dynamically changed.

[0079] In one specific embodiment, the following will illustrate the data interaction diagram of the four-level storage structure using the implementable devices of the first to fourth storage units and their interaction with the cache management unit as examples. As stated above, the first storage unit can be BRAM, the second storage unit can be Ultra Random Access Memory (URAM), the third storage unit can be Double Data Rate Synchronous Dynamic Random Access Memory (DDR RAM), and the fourth storage unit can be a Secure Digital Card (SD card) or other high-capacity storage module. If there are sufficient DDR resources in the embedded computing device, similar to an SD card, the third storage unit can also be a fixed-stored embedding vector of multiple tokens.

[0080] SD cards or other high-capacity storage modules, belonging to the four-level storage architecture, are slow but large-capacity components. During the initialization phase of the task processing device or LLM, the embedding vectors of all tokens can be written to the SD card at once. Different models deployed on the task processing device have different sizes and quantized bit widths, resulting in inconsistent sizes on the SD card, typically ranging from tens to hundreds of megabytes (MB). For computing devices with SD cards or other high-capacity storage modules, no optimization is needed; all token embedding vectors can be directly converted to in8 / int16 format and written using quantization tools. In a specific implementation, after selecting the LLM to be deployed to the computing unit, tools can be used to quantize and compress pre-trained embeddings and weights to obtain the selected low-precision format embeddings and weights, such as half-precision floating-point (fp16) or 8-bit integer (int8). Before starting multi-round computation tasks, the entire embedding and other parameters are preloaded into the FPGA's DDR or SD card at once via PCIe or other external interfaces.

[0081] The DDR portion needs to be partitioned for use. To reduce access latency caused by loading from the SD card each time, this application allocates a fixed space in the DDR for storing the embedding vectors of tokens required for computation by the task processing device (such as an FPGA). This portion is fixed for FPGA computation and will not be swapped out; it is always stored in the DDR. The specific access mode of the DDR uses the AXI interface for burst access to maximize bandwidth utilization.

[0082] The URAM stores a subset of the token embedding vectors, which is smaller and more "hot" than DDR. In one embodiment, depending on the FPGA's resources, approximately 30% of the URAM resources are used to store the token embedding vectors because URAM is faster to access than DDR, and if the required embedding vector is in this cache, it can be hit more quickly during computation.

[0083] BRAM is the fastest cache, with the smallest capacity but the fastest access speed. It does not need to be connected to the AXI interface and can be accessed directly by the cache management unit using the token_id to achieve the lowest latency. Data read from URAM can be directly written to BRAM, and related operations can be performed by the cache management unit.

[0084] In this embodiment, the multi-level on-chip cache architecture, which implements a four-level storage architecture for progressively reading token embeddings, effectively supports the independent deployment of LLM on embedded FPGAs, reduces dependence on high-performance external processors, and promotes true edge autonomy. Furthermore, high-hit-rate on-chip data access effectively reduces latency in the autoregressive decoding process within the computing unit, improving response speed. Through the resource isolation and dynamic scheduling design of the four-level storage architecture, high utilization efficiency is achieved with limited storage resources, avoiding resource conflicts with other computing modules within the FPGA.

[0085] Figure 3 This illustration schematically depicts an embedded vector read process under a four-level storage architecture according to an embodiment of this application. Figure 3 As shown, the second to fourth storage units can each have a corresponding controller, such as an SD controller, a DDR controller, and a URAM controller, to enable data reading and writing within the corresponding SD card, DDR, and URAM. The SD controller and DDR controller can be located outside the task processing device, while the URAM controller and URAM can both be located within the task processing device.

[0086] The AXI interface connector serves as a unified bus, connecting all storage and compute units. For example, all lower-level SD controllers, DDR controllers, and URAM controllers, as well as upper-level compute units and cache management units, are connected to this bus, enabling data transfer between any other devices. Data transfer between BRAM and cache management units, and between cache management units and compute units, can be performed without using the AXI interface connector.

[0087] For computing devices with SD cards or other high-capacity storage modules and limited DDR and on-chip RAM resources (URAM and BRAM), if the target storage unit is an SD card or other high-capacity storage module, the cache management unit can send the storage address of the target embedding vector in that target storage unit to the computing unit. The computing unit can then obtain the target embedding vector through path ① in the diagram. If the target storage unit is DDR, the computing unit can obtain the target embedding vector through path ② in the diagram; if the target storage unit is URAM, the computing unit can obtain the target embedding vector through path ③ in the diagram.

[0088] If the target storage unit is an SD card, the computing unit can obtain the target storage unit from the SD card in the manner of path ①. At the same time, the cache management unit can also write the target embedding vector stored in the target storage unit into DDR, URAM and BRAM step by step through the AXI interface connector. Then, when the target embedding vector needs to be read in the next round, it can be read directly from BRAM. The whole process can be referred to in the manner of path ④.

[0089] In one embodiment, when the target storage unit is the second storage unit within the task processing device, the four-layer storage architecture can be simplified to a two-layer storage architecture. The processing operations of the two-layer storage architecture are similar to the interaction between the target storage unit and the first storage unit described above, and will not be repeated here.

[0090] In this embodiment, the on-chip two-layer storage architecture and its scheduling enable most of the lookup and retrieval of embedded vectors to be completed on-chip, reducing access to external storage.

[0091] When the target storage unit is the third or fourth storage unit, unlike the two-tier storage architecture, there will also be hierarchical data read and write between multiple storage units.

[0092] In another embodiment, when the target storage unit is a third or fourth storage unit, the computing unit reads the target embedding vector from the target storage unit and performs computing tasks using the target embedding vector; at the same time, the cache management unit can synchronously adjust the embedding vectors stored in the first to third storage units.

[0093] For example, the cache management unit can dynamically adjust the embedding vectors stored in the first to third storage units based on the number of times the token is used.

[0094] For example, the frequency threshold may include a first sub-frequency threshold, a second sub-frequency threshold, and a third sub-frequency threshold, with values ​​decreasing sequentially. When the target storage unit is the second storage unit, the embedding vectors of input words whose frequency exceeds the first sub-frequency threshold are read from the second storage unit and written to the parent storage unit (such as the first storage unit); when the target storage unit is the third storage unit, the embedding vectors of input words whose frequency exceeds the second sub-frequency threshold are read from the third storage unit and written to the parent storage unit (such as the second storage unit and / or the first storage unit); when the target storage unit is the fourth storage unit, the embedding vectors of input words whose frequency exceeds the third sub-frequency threshold are read from the fourth storage unit and written to the parent storage unit (such as at least one of the third storage unit, the second storage unit, and the first storage unit). Correspondingly, the embedding vectors of input words whose usage count does not exceed the first sub-threshold can also be read from the first storage unit and written to the second storage unit; the embedding vectors of input words whose usage count does not exceed the second sub-threshold can be read from the first storage unit and written to the second storage unit and / or the third storage unit; the embedding vectors of input words whose usage count does not exceed the third sub-threshold can be read from the first storage unit and written to the second storage unit and / or the third storage unit, or directly deleted (no data is written to the fourth storage unit during the calculation process).

[0095] The cache management unit can also dynamically adjust the embedding vectors stored in the first to third storage units from the read operations of the target storage unit.

[0096] For example, the cache management unit is further configured to: when the target storage unit is a third storage unit or a fourth storage unit, read the target embedding vector from the target storage unit according to the storage address, and write the target embedding vector to the parent storage unit of the target storage unit, wherein when the target storage unit is a third storage unit, the parent storage unit includes a second storage unit and a first storage unit; and when the target storage unit is a fourth storage unit, the parent storage unit includes a first storage unit, a second storage unit, and a third storage unit.

[0097] Figure 4 This illustration schematically depicts a data flow diagram of the embedding vectors stored in multiple storage units under a dynamically adjusted four-level storage architecture according to an embodiment of this application. For example... Figure 4 As shown, when the target storage unit is a DDR or SD card, the computing unit reads the target embedding vector from the target storage unit and uses the target embedding vector to perform computing tasks; at the same time, the cache management unit dynamically adjusts the storage unit used to store the target embedding vector.

[0098] like Figure 4As shown, when the target embedding vector of the input token in round i is stored on the SD card, the target embedding vector is not directly migrated to BRAM in round i. Instead, it is migrated from the SD card to DDR (i.e., the storage unit above the SD card). In round i+1, the target embedding vector is migrated from DDR to URAM (the storage unit above DDR). In round i+2, the target embedding vector is migrated from URAM to BRAM (the storage unit above URAM). Thus, hot data loading is achieved through step-by-step data migration.

[0099] Alternatively, if the target embedding vector of the input word in the i-th round is stored on the SD card, the target embedding vector can be directly read from the SD card and written to the BRAM. This operation is similar to the direct interaction between the target storage unit and the first storage unit described above, and will not be repeated here. If the target embedding vector of the input word in the i-th round is stored in DDR, the operation can be performed using the step-by-step data migration method described above.

[0100] like Figure 4 As shown, while loading the hot data as described above, the cache management unit can also read the embedding vectors stored in the BRAM that have not exceeded the usage threshold from the BRAM based on the number of times each word is used in the BRAM, and write them to the URAM or DDR.

[0101] For the multi-level storage architecture described above, when migrating an embedding vector from the first storage unit to another storage unit, the corresponding embedding vector can be deleted from the first storage unit.

[0102] In this embodiment, by designing data interaction rules and migration triggering conditions for different levels of storage units, the embedding vectors of frequently accessed hot tokens are always stored in storage units with faster access speeds, while the embedding vectors of infrequently accessed non-hot tokens are automatically migrated to storage units with larger capacity and slightly lower access speeds. This balances the dual requirements of storage capacity and access latency for computing devices to perform LLM computing tasks.

[0103] According to an embodiment of this application, the task processing device includes a plurality of first storage units, and the target embedding vector is divided into a plurality of sub-embedded vectors, which are respectively stored in the plurality of first storage units. The cache management unit is configured to read the target embedding vector from the first storage units and send the target embedding vector to the computing unit in the following manner: reading the corresponding sub-embedded vector from the plurality of first storage units according to the lexical identifier to obtain a plurality of sub-embedded vectors; and combining the plurality of sub-embedded vectors into a target embedding vector and then sending the target embedding vector to the computing unit.

[0104] For example, the target embedding vector is divided into multiple sub-embedding vectors, and correspondingly, these sub-embedding vectors can be concatenated to obtain the target embedding vector. The task processing device, such as an FPGA, may include multiple BRAMs. To further utilize the limited on-chip BRAM resources and improve access speed, the corresponding sub-embedding vectors can be read in parallel from multiple BRAMs based on the token_id, and then concatenated to obtain the target embedding vector. The target embedding vector is then sent to the computing unit.

[0105] Therefore, by interleaving different parts of a target embedding vector into multiple BRAMs, when a read request for the target embedding vector is received, multiple BRAMs can be accessed in parallel, and the data of a target embedding vector can be assembled within one cycle, achieving higher throughput.

[0106] According to an embodiment of this application, multiple sub-embedded vectors have the same storage address in their corresponding first storage units; the cache management unit is configured to read the corresponding sub-embedded vectors from the multiple first storage units respectively according to the lexical identifier in the following manner: using the lexical identifier as the storage address of each sub-embedded vector relative to the first storage unit, and reading the corresponding sub-embedded vectors from the multiple first storage units in parallel according to the storage address to obtain multiple sub-embedded vectors.

[0107] Figure 5 This schematically illustrates a structure for parallel reading of multiple sub-embedded vectors according to an embodiment of this application. For example... Figure 5 As shown, taking token_id1 as an example, each target embedding vector requires 768 bytes, such as... Figure 5 The bytes involved are [0, 767], and the target embedding vector is quantized in int8 form. In the 12 BRAMs, the size of a single BRAM is 8K, the bit width is 512 bits, and each BRAM can store 0~511 bits of data as shown in the figure. Its depth is 128, and its capacity can store the embedding vectors of 10 tokens, such as token_id1~token_id10.

[0108] Taking 12 BRAMs as an example, 12 8K-sized BRAMs can be instantiated, storing different parts of a vector at the same memory address in different BRAMs. Addr_1_1, Addr_1_2, Addr_2_1...Addr_10_1 and Addr_x_12 all represent address lines. Taking Addr_10_1 as an example, the last bit "1" indicates that it belongs to BRAM1, and the prefix number "10" is the row address. For example, there are a total of 10 valid addresses (addresses 0 to 9), corresponding to 10 rows.

[0109] In this design, for token_id1, within one FPGA cycle, the same memory address of 12 BRAMs can be accessed simultaneously via address line Addr_x_12, where x=1, to obtain 12 sub-embedding vectors in parallel. These 12 sub-embedding vectors are then concatenated to obtain the complete target embedding vector for token_id1, achieving a read speed of 768 bytes within one read cycle. This ensures low latency, efficient use of on-chip BRAM resources, and timely timing convergence. In reality, accessing such a large bit width within a single BRAM would put pressure on FPGA placement and routing, potentially leading to timing issues. This embodiment addresses this by using multiple BRAMs to store multiple sub-embedding vectors, enabling single-cycle multi-data reads under a wide-bit data bus to meet the high throughput requirements of the computing unit.

[0110] Understandable Figure 5 This explanation uses a 12-BRAM with int8 quantization as an example. If the quantization format is int4, two token_ids can be obtained per FPGA cycle. In this case, the cache management unit can directly concatenate multiple embedding sub-vectors from BRAM0~5 or BRAM6~12 to obtain the final target embedding vector. This BRAM design also accommodates DDR prefetching modes. For example, based on a 512-bit AXI4 interface, by setting the burst length, 4K or 8K data can be prefetched at a time, facilitating access to and reading of the embedding vector from DDR.

[0111] In the embodiments of this application, by designing that multiple sub-embedded vectors have the same storage address in the corresponding first storage unit, and that the storage address is a token identifier, the cache management unit can obtain multiple sub-embedded vectors in parallel through a single signal based on the token identifier, and acquire all target embedded vectors within 1-2 FPGA cycles, thereby reducing access latency.

[0112] According to an embodiment of this application, the task processing apparatus further includes a first read / write control unit, wherein the cache management unit is configured to send a lexical identifier as a storage address to the first read / write control unit; the first read / write control unit is configured to, in response to receiving the storage address, read the corresponding sub-embedded vectors from a plurality of first storage units in parallel according to the storage address, and send the obtained plurality of sub-embedded vectors to the cache management unit.

[0113] The first read / write control unit can be a BRAM controller. For example, the cache management unit can communicate with the BRAM controller through an interface (such as the token_vector interface) to directly obtain the target embedding unit stored in the BRAM from the BRAM controller. The BRAM controller can flexibly support embedding vectors of various bit widths, which can be customized according to the deployed LLM design. For example, the BRAM controller and the corresponding BRAM instantiation can be customized according to the physical characteristics of the FPGA on-chip BRAM and the number of bytes required to represent each token, such as the instantiation bit width and depth.

[0114] Still with Figure 5 For example, if it is necessary to retrieve the target embedding vector of token_id1, the cache management unit can transmit token_id1 to the BRAM controller. The BRAM controller generates the Addr_x_12 signal and directly retrieves multiple sub-embedding vectors in parallel.

[0115] In this embodiment, by setting a first read / write control unit, it is possible to flexibly adapt to embedding vectors of various bit widths in accordance with the design requirements of LLM.

[0116] According to an embodiment of this application, the cache management unit includes a cache subunit configured to cache at least one of a target embedding vector and a storage address read from a first storage unit, and to transmit at least one of the target embedding vector and the storage address to the computing unit in a first-in-first-out order.

[0117] The cache subunit can be implemented as a queue to buffer the transmission of the target embedding vector and its storage address.

[0118] In actual operation, the computing unit may be temporarily unable to receive new data due to internal congestion. The FIFO queue in the cache management unit can temporarily store the target embedding vector and / or storage address to avoid congestion of the upstream path of the computing unit or blockage of the BRAM controller.

[0119] For example, the cache subunit can be a First In First Out (FIFO) queue. Data read from the BRAM is not directly and immediately sent to the computing unit, but is temporarily cached through the FIFO queue of the cache management unit. The target embedding vector and storage address remain unchanged in the FIFO queue, and then are passed to the computing unit in an orderly manner according to the order in which they entered the FIFO queue, realizing "pass-through".

[0120] In one specific embodiment, the operating clock of the BRAM controller and the operating clock of the computing unit may be at different frequencies. In this case, the FIFO queue can be used as an asynchronous FIFO queue to enable secure cross-clock domain data transmission through the cache management unit, thereby achieving a higher clock frequency for the computing module and improving computing performance.

[0121] In this embodiment, the cache sub-unit in the cache management unit can not only avoid data access congestion on the FPGA chip, but also achieve cross-clock domain data transmission through simple design when the BRAM controller and the computing unit are in different clock domains.

[0122] According to an embodiment of this application, the task processing apparatus further includes: an entry module configured to: acquire user input text during a pre-filling stage and perform lexical transformation on the input text; and determine the lexical identifier of the lexical obtained after performing lexical transformation on the user input text according to a vocabulary.

[0123] Unlike heterogeneous computing models where the prefilling stage is implemented by an external processor, this embodiment of the application implements the prefilling stage independently by a task processing device. In the prefilling stage: the entry module performs text tokenization on the user input text to obtain a token sequence that can be processed by the LLM deployed in the computing unit; then, the token sequence is converted into integer indices in the vocabulary; by querying the tokenizer's vocabulary, each token is mapped to a unique token_id; and the token_id is sent all at once to the FPGA's cache management unit to enable subsequent target embedding vector acquisition and computation.

[0124] In this embodiment, the task processing device performs word conversion through its entry module and quickly locates and obtains the target embedding vector through its cache management unit, enabling the task processing device to independently and quickly execute LLM computation tasks.

[0125] Figure 6 A schematic diagram of the framework of a task processing apparatus according to an embodiment of this application is shown. Figure 6As shown, the task processing device may include an entry module, a cache management unit, a first storage unit, a second storage unit, a selection switch, and a calculation unit. Users can input text through a user port. The entry module acquires the user input text through the user port and performs word-based transformation on the input text to obtain word identifiers for the input words. Subsequently, the word identifiers are transmitted to the cache management unit, which performs embedding vector lookup based on the word identifiers and an address mapping table. The task processing device may also include a selection switch, which can be a multiplexer connected to the first, second, third, and fourth storage units. This selection switch is used to select the storage unit containing the target embedding vector from the four storage units based on a read signal from the cache management unit or the calculation unit, thus activating the path so that the cache management unit can read the target embedding vector stored in the four storage units. Furthermore, the calculation unit may also be connected to the selection switch to read the target embedding vector from the second to fourth storage units.

[0126] In this embodiment, the third and fourth storage units within the dashed box can be located outside the task processing device or inside the computing device where the task processing device is located.

[0127] In this embodiment, the computing unit can have multiple input words, and there is a sequential order among the multiple target embedding vectors of the multiple input words. Therefore, the task processing device may further include a position encoding generator, which determines the order among the multiple input words. Then, the position encoding can be added to the target embedding vectors by the arithmetic circuit in the task processing device or a module built into the computing unit, so that the computing unit knows the order among the multiple input words when performing the computing task.

[0128] After the computing unit performs one round of computation, it obtains the token identifier of the predicted token, which serves as the input token for the next round of computation. The token identifier of the predicted token can then be processed by the cache management unit, which reads the target embedding vector of the predicted token from four storage units and performs the next round of computation. This process is repeated until multiple rounds of computation are completed.

[0129] Figure 7 A flowchart illustrating a task processing method according to an embodiment of this application is shown, such as... Figure 7 As shown, the task processing method is applied to the task processing device described above, and the task processing method includes operations S710 to S720.

[0130] During operation of S710, the computing unit of the task processing device performs multiple rounds of computation tasks based on the target embedding vector of the input lexical and outputs the predicted lexical.

[0131] The input lexical units for the computation task include at least one of the lexical units obtained by performing lexical transformation on the user input text and the predicted lexical units output from the previous round of computation task.

[0132] During operation S720, the cache management unit of the task processing device reads the target embedding vector from the first storage unit when the target embedding vector is in the first storage unit and sends the target embedding vector to the computing unit; when the target embedding vector is not stored in the first storage unit, the target storage unit and storage address where the target embedding vector is stored are determined and the storage address is sent to the computing unit. The computing unit reads the target embedding vector from the target storage unit according to the storage address. The capacity of the target storage unit is greater than the capacity of the first storage unit.

[0133] The operations implemented by the task processing device using the above task processing method can be found in the description of the task processing device section above, and will not be repeated here.

[0134] This application also provides a computing device, including: the task processing apparatus as described above; at least one of a third storage unit and a fourth storage unit, configured to store embedding vectors of a plurality of lexical units, such that the task processing apparatus, when the first storage unit in the task processing apparatus does not include a target embedding vector, obtains the target embedding vector from at least one of the third storage unit and the fourth storage unit and performs a computing task using the target embedding vector.

[0135] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0136] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described task processing method embodiments at runtime.

[0137] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0138] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described task processing method embodiments.

[0139] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described task processing method embodiments.

[0140] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0141] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0142] The foregoing has provided a detailed description of the task processing apparatus, method, and computing device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A task processing device, characterized in that, The task processing device includes: The computation unit is configured to perform multiple rounds of computation tasks based on the target embedding vector of the input lexical units and output predicted lexical units, wherein the input lexical units of the computation tasks include at least one of the lexical units obtained after lexical transformation of the user input text and the predicted lexical units output by the previous round of computation tasks. At least one counter is configured to: count the number of times each input word is used in the multi-round computation task performed by the computing unit; The first storage unit is used to store the embedding vectors of multiple lexical units determined based on the number of times the lexical unit is used; A cache management unit is configured to: when the target embedding vector is stored in the first storage unit, read the target embedding vector from the first storage unit and send the target embedding vector to the computing unit; when the target embedding vector is not stored in the first storage unit, determine the target storage unit and storage address where the target embedding vector is stored, and send the storage address to the computing unit, wherein the computing unit reads the target embedding vector from the target storage unit according to the storage address, and the capacity of the target storage unit is greater than the capacity of the first storage unit; obtain the usage count of each input word from the at least one counter; update the embedding vector of multiple words stored in the first storage unit using the embedding vector of multiple words stored in the target storage unit according to the usage count of each input word, so that the first storage unit always retains the embedding vector of input words whose usage count exceeds the threshold; and update the address mapping table according to the updated embedding vector in the first storage unit.

2. The task processing device according to claim 1, characterized in that, The cache management unit is configured to determine whether the target embedding vector is stored in the first storage unit in the following manner: Based on the lexical identifier of the input lexical, determine the address mapping information that matches the input lexical from the address mapping table; Based on the first address range of the address mapping information, the first address identifier bit is read from the address mapping information; If the storage unit indicated by the first address identifier is the first storage unit, then the target embedding vector is determined to be stored in the first storage unit.

3. The task processing device according to claim 2, characterized in that, The cache management unit is configured to determine the target storage unit and storage address where the target embedding vector is stored in the following manner: If the storage unit indicated by the first address identifier is not the first storage unit, the storage unit indicated by the first address identifier shall be used as the target storage unit. Based on the second address range of the address mapping information, the second address identifier bit is read from the address mapping information, and the address indicated by the second address identifier bit is determined as the storage address.

4. The task processing device according to claim 1, characterized in that, The cache management unit is further configured to update the embedding vectors of multiple words stored in the first storage unit based on the word usage frequency of each input word, using the embedding vectors of multiple words stored in the target storage unit: The embedding vectors of input words whose usage frequency exceeds a threshold are read from the target storage unit and written to the first storage unit; The embedding vectors of input words whose usage frequency does not exceed the threshold are read from the first storage unit and written to the target storage unit.

5. The task processing apparatus according to claim 1, characterized in that, The cache management unit is also configured to: The target embedding vector is read from the target storage unit and written to the first storage unit; and when the computing unit executes the next round of computing tasks, the target embedding vector is read from the first storage unit and sent to the computing unit.

6. The task processing apparatus according to any one of claims 1 to 5, characterized in that, The task processing device further includes a second storage unit, and the computing device in which the task processing device is located further includes a third storage unit and a fourth storage unit disposed outside the task processing device and inside the computing device, wherein the capacities of the second storage unit, the third storage unit and the fourth storage unit increase sequentially; the second storage unit and the third storage unit are configured to store the embedding vectors of multiple words determined based on the word usage frequency, and the fourth storage unit is configured to store the embedding vectors of multiple words in a fixed manner; The target storage unit is at least one of the second storage unit, the third storage unit, and the fourth storage unit.

7. The task processing apparatus according to claim 6, characterized in that, The cache management unit is also configured to: When the target storage unit is the third storage unit or the fourth storage unit, the target embedding vector is read from the target storage unit according to the storage address, and the target embedding vector is written to the parent storage unit of the target storage unit. Wherein, when the target storage unit is the third storage unit, the parent storage unit includes the second storage unit and the first storage unit; when the target storage unit is the fourth storage unit, the parent storage unit includes the first storage unit, the second storage unit and the third storage unit.

8. The task processing apparatus according to any one of claims 1 to 5, characterized in that, The task processing device includes a plurality of first storage units, and the target embedding vector is divided into a plurality of sub-embedding vectors, which are respectively stored in the plurality of first storage units; The cache management unit is configured to read the target embedding vector from the first storage unit and send the target embedding vector to the computing unit in the following manner: Based on the lexical identifier, the corresponding sub-embedding vectors are read from the multiple first storage units respectively to obtain the multiple sub-embedding vectors; and After combining the multiple sub-embedded vectors into the target embedded vector, the target embedded vector is sent to the computing unit.

9. The task processing apparatus according to claim 8, characterized in that, The multiple sub-embedded vectors have the same storage address in their corresponding first storage units; the cache management unit is configured to read the corresponding sub-embedded vectors from the multiple first storage units respectively according to the lexical identifier in the following manner: The lexical identifier is used as the storage address of each sub-embedding vector relative to the first storage unit, and the corresponding sub-embedding vector is read from multiple first storage units in parallel according to the storage address to obtain multiple sub-embedding vectors.

10. The task processing apparatus according to claim 8, characterized in that, The task processing device further includes a first read / write control unit, wherein... The cache management unit is configured to send the lexical identifier as the storage address to the first read / write control unit; The first read / write control unit is configured to, in response to receiving the storage address, read the corresponding sub-embedding vector from a plurality of first storage units in parallel according to the storage address, and send the obtained plurality of sub-embedding vectors to the cache management unit.

11. The task processing apparatus according to any one of claims 1 to 5, characterized in that, The cache management unit includes: The cache subunit is configured to cache at least one of the target embedding vector and the storage address read from the first storage unit, and to transmit at least one of the target embedding vector and the storage address to the computing unit in a first-in-first-out order.

12. The task processing apparatus according to any one of claims 1 to 5, characterized in that, The task processing device further includes: The entry module is configured to: acquire the user input text during the pre-filling stage, and perform lexical transformation on the input text; and determine the lexical identifier of the lexical obtained after lexical transformation of the user input text according to the vocabulary.

13. A task processing method, applied to the task processing apparatus according to any one of claims 1 to 12, characterized in that, The method includes: Using the computing unit of the task processing device, multiple rounds of computing tasks are performed based on the target embedding vector of the input lexical units, and predicted lexical units are output. The input lexical units of the computing tasks include at least one of the lexical units obtained after lexical transformation of the user input text and the predicted lexical units output by the previous round of computing tasks. Using the cache management unit of the task processing device, when the target embedding vector is in the first storage unit, the target embedding vector is read from the first storage unit and sent to the computing unit; when the target embedding vector is not stored in the first storage unit, the target storage unit and storage address where the target embedding vector is stored are determined and the storage address is sent to the computing unit, wherein the computing unit reads the target embedding vector from the target storage unit according to the storage address, and the capacity of the target storage unit is greater than the capacity of the first storage unit; The method further includes: using the cache management unit to obtain the usage count of each input word from the at least one counter; updating the embedding vector of multiple words stored in the first storage unit according to the usage count of each input word using the embedding vector of multiple words stored in the target storage unit, so that the first storage unit always retains the embedding vector of input words whose usage count exceeds the threshold; and updating the address mapping table according to the updated embedding vector in the first storage unit.

14. A computing device, characterized in that, The computing device includes: The task processing apparatus as described in any one of claims 1 to 12; At least one of the third and fourth storage units is configured to store embedding vectors of multiple lexical units, such that, if the first storage unit in the task processing device does not include the target embedding vector, the task processing device obtains the target embedding vector from at least one of the third and fourth storage units and performs a computational task using the target embedding vector.

Images