Method and electronic device for generative model processing

Abstract

A method and an electronic device for generative model processing are provided. The method for generative model processing includes determining a task ratio between a first processing group and a second processing group for an inference task of a generative model, generating sub-tensors by dividing tensors for the inference task based on communication dependencies of a computation stage of the inference task, storing the generated sub-tensors in a first memory pool of the first processing group and a second memory pool of the second processing group based on the task ratio, and performing the inference task by the first processing group and the second processing group using the stored sub-tensors.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2025-0049433, filed on April 16, 2025, with the Korean Intellectual Property Office, and Korean Patent Application No. 10-2025-0077369, filed on June 12, 2025, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to methods and apparatus for generative model processing. Background Technology

[0003] Generative models can be machine learning models trained to generate new data similar to a given data distribution. For example, deep learning-based generative models (such as Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and Transformer-based language models) can demonstrate superior performance in generating images, speech, and text. Deep learning-based generative models are typically trained on large datasets and can effectively learn complex patterns and structures present in the data. Transformer-based generative models can be used in natural language processing and visual language domains to achieve high-quality text generation and multimodal inference. Summary of the Invention

[0004] The present invention is provided in a simplified form to introduce the choice of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0005] In one or more general aspects, a processor-implemented method includes: determining a task ratio between a first processing group and a second processing group for an inference task of a generative model; generating a sub-tensor by partitioning tensors used for the inference task based on communication dependencies of computational phases of the inference task; storing the generated sub-tensor in a first memory pool of the first processing group and a second memory pool of the second processing group based on the task ratio; and performing the inference task by the first processing group and the second processing group using the stored sub-tensor.

[0006] The step of determining the task ratio between the first processing group and the second processing group may include: determining the task ratio based on the processing performance of the first processing group and the second processing group.

[0007] Generative models may include attention sub-layers and multilayer perceptron (MLP) sub-layers, and the task ratio may include a first task ratio between a first processing group and a second processing group for the task corresponding to the attention sub-layer, and a second task ratio between a first processing group and a second processing group for the task corresponding to the MLP sub-layer.

[0008] The steps of generating a subtensor may include: generating a first subtensor within a subtensor by partitioning a first tensor within a tensor according to the attention head dimension, where the attention head dimension represents the dimension of the head direction of the multi-head attention of the generative model; and generating a second subtensor within a subtensor by partitioning a second tensor according to the column direction dimension among the row direction dimension and column direction dimension of a second tensor within a tensor.

[0009] The first tensor may include any one or any combination of any two or more of the query weights, key weights, value weights, key tensor sequences, and value tensor sequences of the attention sublayer of the generative model, and the second tensor includes one or both of the weights of the multilayer perceptron (MLP) sublayer and the output weights of the attention sublayer of the generative model.

[0010] The task ratio may include a first task ratio between a first processing group and a second processing group for tasks corresponding to the attention sublayer of the generative model, and a second task ratio between a first processing group and a second processing group for tasks corresponding to the multilayer perceptron (MLP) sublayer of the generative model. The step of storing the subtensor in the first memory pool and the second memory pool may include: dividing the first subtensor according to the first task ratio and storing the first subtensor in the first memory pool and the second memory pool; and dividing the second subtensor according to the second task ratio and storing the second subtensor in the first memory pool and the second memory pool.

[0011] The step of storing a subtensor in a first memory pool and a second memory pool may include storing the subtensor to be processed by a first processor among a plurality of first processors in a first processing group in a first memory node located in the local region of the first processor among a plurality of first memory nodes in the first memory pool.

[0012] The first memory pool may include memory nodes with a first bandwidth and memory nodes with a second bandwidth, and the step of storing subtensors in the first memory pool and the second memory pool may include: storing subtensors in memory nodes with the first bandwidth and memory nodes with the second bandwidth based on the ratio between the first bandwidth and the second bandwidth.

[0013] The step of storing subtensors in a first memory pool and a second memory pool may include storing the subtensors in the first memory pool in blocks smaller than pages.

[0014] The steps for performing the inference task may include: generating the attention output tensor of the current attention sublayer of the generative model by processing the current attention sublayer; and generating the MLP output tensor of the current MLP sublayer by processing the current multilayer perceptron (MLP) sublayer of the generative model based on the attention output tensor of the current attention sublayer.

[0015] The step of generating the attention output tensor of the current attention sublayer may include: generating, independently by the first and second processing groups, a first part MLP output tensor and a second part MLP output tensor of the previous MLP sublayer of the generative model, in the absence of communication between the first and second processing groups; generating the attention input tensor of the current attention sublayer of the generative model by merging the first part MLP output tensor and the second part MLP output tensor, based on communication between the first and second processing groups; and generating, independently by the first and second processing groups, a first part attention fraction tensor and a second part attention fraction tensor of the current attention sublayer, based on the attention input tensor and the first subtensor, in the absence of communication between the first and second processing groups.

[0016] The first subtensor can be generated by partitioning the tensor according to the attention head dimension, which represents the dimension of the head direction of the multi-head attention in the generative model.

[0017] The method may include: generating an attention fraction tensor of the current attention sub-layer by combining a first partial attention fraction tensor and a second partial attention fraction tensor based on communication between a first processing group and a second processing group; and generating a first partial attention output tensor and a second partial attention output tensor of the current attention sub-layer independently by the first processing group and the second processing group based on the attention fraction tensor and the second sub-tensor, in the absence of communication between the first processing group and the second processing group.

[0018] The second subtensor can be generated by partitioning the second tensor according to the column direction dimension of the row direction and column direction dimension of the second tensor within the tensor.

[0019] In one or more general aspects, an electronic device includes: one or more processors, including processing circuitry; and a memory, including one or more storage media storing instructions, which, when executed individually or jointly by the one or more processors, cause the electronic device to: determine a task ratio between a first processing group and a second processing group for an inference task of a generative model; generate a sub-tensor by partitioning tensors used for the inference task based on communication dependencies of computational phases of the inference task; store the generated sub-tensor in a first memory pool of the first processing group and a second memory pool of the second processing group based on the task ratio; and perform the inference task by the first processing group and the second processing group using the sub-tensor.

[0020] In order to generate a subtensor, the execution of the instruction enables the electronic device to: generate a first subtensor of subtensors by partitioning a first tensor of a tensor according to the attention head dimension, where the attention head dimension represents the dimension of the head direction of the multi-head attention of the generative model; and generate a second subtensor of subtensors by partitioning a second tensor according to the column direction dimension of the row direction dimension and the column direction dimension of the second tensor of the tensor.

[0021] The task ratio may include a first task ratio between a first processing group and a second processing group for tasks related to the attention sublayer of the generative model, and a second task ratio between a first processing group and a second processing group for tasks related to the multilayer perceptron (MLP) sublayer of the generative model. Furthermore, in order to store the subtensor in a first memory pool and a second memory pool, the execution of the instructions causes the electronic device to: partition the first subtensor according to the first task ratio and store the first subtensor in the first memory pool and the second memory pool; and partition the second subtensor according to the second task ratio and store the second subtensor in the first memory pool and the second memory pool.

[0022] In order to store the subtensor in the first memory pool and the second memory pool, the execution of the instruction causes the electronic device to store the subtensor to be processed by the first processor among the plurality of first processors of the first processing group in the first memory node located in the local region of the first processor among the plurality of first memory nodes of the first memory pool.

[0023] The first memory pool may include memory nodes with a first bandwidth and memory nodes with a second bandwidth, and in order to store the subtensor in the first memory pool and the second memory pool, the execution of the instruction causes the electronic device to: store the subtensor in the memory nodes with the first bandwidth and the memory nodes with the second bandwidth based on the ratio between the first bandwidth and the second bandwidth.

[0024] In one or more general aspects, a processor-implemented method includes: generating a first sub-tensor by partitioning a first tensor dimensionally based on the head direction of a multi-head attention model, and storing the first sub-tensor in a first memory pool and a second memory pool based on a first task ratio; generating a second sub-tensor by partitioning a second tensor dimensionally based on the column direction of a second tensor, and storing the second sub-tensor in the first memory pool and the second memory pool based on a second task ratio; and performing an inference task by processing the first and second sub-tensors stored in the first memory pool using a first processing group and processing the first and second sub-tensors stored in the second memory pool using a second processing group.

[0025] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0026] Figure 1 An example of a processing configuration for an electronic device for performing a generative model, according to one or more embodiments, is shown.

[0027] Figure 2 Examples of inference operations for generative models according to one or more embodiments are shown.

[0028] Figure 3 An example of the configuration of a transformer layer according to one or more embodiments is shown.

[0029] Figure 4 Examples of operations for performing inference tasks of generative models using a multiprocessor group are shown according to one or more embodiments.

[0030] Figure 5 An example of the task ratio between a first processing group and a second processing group according to one or more embodiments is shown.

[0031] Figure 6 Examples of subtensors partitioned according to the attention head dimension are shown according to one or more embodiments.

[0032] Figure 7 Examples of subtensors partitioned according to the column direction dimension are shown according to one or more embodiments.

[0033] Figure 8 An example of the processing structure of a first processing group according to one or more embodiments is shown.

[0034] Figure 9 Examples of operations for placing sub-tensors in a heterogeneous memory pool according to one or more embodiments are shown.

[0035] Figure 10AAn example is shown of the processing of the attention sublayer by a first processing group according to one or more embodiments.

[0036] Figure 10B An example of the processing of the attention sublayer by a second processing group according to one or more embodiments is shown.

[0037] Figure 10C An example is shown of the processing of a multilayer perceptron (MLP) sublayer by a first processing group according to one or more embodiments.

[0038] Figure 10D An example of processing of the MLP sublayer by a second processing group is shown according to one or more embodiments.

[0039] Figure 11 Examples of configurations of electronic devices according to one or more embodiments are shown.

[0040] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be exaggerated. Detailed Implementation

[0041] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations within and / or the sequence of operations described herein are merely examples and are not limited to the order set forth herein, but may be changed as will become clear upon understanding this disclosure, except for the order of operations within and / or the sequence of operations that must occur in a specific order. As another example, the order of operations and / or the sequence of operations within and / or the sequence of operations may be performed in parallel, except for at least a portion of the order of operations within and / or the sequence of operations that must occur in a specific order (e.g., a specific order). Furthermore, for greater clarity and conciseness, descriptions of features known upon understanding this disclosure may be omitted.

[0042] Although terms such as “first,” “second,” and “third,” or A, B, (a), (b), etc., may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. For example, each of these terms is not used to define the nature, order, or sequence of the corresponding component, assembly, region, layer, or part, but only to distinguish the corresponding component, assembly, region, layer, or part from other components, assemblies, regions, layers, or parts. Therefore, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0043] Throughout this specification, when a component or element is described as being "on," "connected to," "bonded to," or "joined to" another component, element, or layer, it may be directly (e.g., in contact with) "on," "connected to," "bonded to," or "joined to" that other component, element, or layer, or one or more other components, elements, or layers may reasonably be present therein. When a component, element, or layer is described as being "directly on," "directly connected to," "directly bonded to," or "directly joined to" another component, element, or layer, no other components, elements, or layers may be present therein. Similarly, expressions such as "between" and "immediately between," and "adjacent to" and "closely adjacent to" may also be interpreted as described above.

[0044] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. As a non-limiting example, the terms “comprising,” “including,” and “having” indicate the presence of the stated features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof, or optionally, alternative features, quantities, operations, components, elements, and / or combinations thereof. Furthermore, while one embodiment may illustrate the use of the terms “comprising,” “including,” and “having” to indicate the presence of the stated features, quantities, operations, components, elements, and / or combinations thereof, other embodiments may exist in which one or more of the stated features, quantities, operations, components, elements, and / or combinations thereof are not present.

[0045] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more. Phrases such as “at least one of A, B, and C”, “at least one of A, B, or C” are intended to have a disjunctive meaning, and unless the corresponding description and embodiments require such a list (e.g., “at least one of A, B, and C”) to be interpreted as having a conjunctive meaning, these phrases “at least one of A, B, and C”, “at least one of A, B, or C” (e.g., each phrase may include any one of the individual corresponding items, all items listed together, and all possible combinations thereof) also include examples of one or more of each of A, B, and / or C (e.g., any combination of one or more of each of A, B, and C).

[0046] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains, particularly in the context of understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field, particularly in the context of this disclosure, and shall not be interpreted as having an idealized or overly formal meaning.

[0047] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided only to illustrate some of the many feasible ways in which the methods, apparatus, and / or systems described herein will be clear upon understanding the disclosure of this application. The term “may” (e.g., regarding what an example or embodiment may include or implement) is used herein with respect to examples or embodiments to indicate the existence of at least one example or embodiment that includes or implements such a feature, while not all examples are limited thereto. The terms “example,” “embodiment,” and “example embodiment” are used herein with the same meaning (e.g., the phrase “in an example or in one example” has the same meaning as “in an embodiment or in one embodiment” and “in an example embodiment or in an example embodiment”), and “one or more examples” has the same meaning as “one or more embodiments” and “one or more example embodiments.” Furthermore, each of the multiple or all individually described “example,” “embodiment,” “example embodiment,” and “multiple examples,” “multiple embodiments,” “multiple example embodiments” may be combined into the same embodiment in any combination.

[0048] In the following description, examples will be described in detail with reference to the accompanying drawings. When describing examples with reference to the accompanying drawings, the same reference numerals refer to the same elements, and repeated descriptions related to them will be omitted.

[0049] Figure 1 An example of a processing configuration for an electronic device for performing a generative model, according to one or more embodiments, is shown. (Refer to...) Figure 1 The electronic device 100 may include a first processing group 111, a first memory pool 112, a second processing group 121, and a second memory pool 122.

[0050] The first processing group 111 may include one or more first processors. The first processing group 111 may be the main processing group of the electronic device 100, and the first processor may be the main processor of the electronic device 100. For example, the first processor may be, but is not limited to, a central processing unit (CPU).

[0051] The first memory pool 112 may include one or more memories. The first memory pool 112 may include heterogeneous memories. Heterogeneous memories may be memories at different tiers. Heterogeneous memories may be connected to the first processor with different interfaces and / or may have different bandwidths. For example, the first memory pool 112 may include one or more main memories and one or more extended memories. Extended memories may be external memories. According to an example, the first memory pool 112 (e.g., extended memories) may include near-memory processing units (NMPUs) that provide operational functions.

[0052] For example, main memory may include dynamic random access memory (DRAM), and extended memory may include compute fast link memory (CXL memory), but the examples are not limited thereto. Main memory (e.g., DRAM) may have a wide bandwidth to extended memory (e.g., CXL memory). Main memory (e.g., DRAM) may be connected to a first processor (e.g., CPU) via a double data rate (DDR) interface, and extended memory (e.g., CXL memory) may be connected to the first processor (e.g., CPU) via peripheral component interconnect fast (PCI-e), but the examples are not limited thereto.

[0053] The second processing group 121 may include one or more second processors. The second processing group 121 may be a coprocessor group of the electronic device 100, and the second processor may be a coprocessor of the electronic device 100. For example, the second processor may include one or more of a graphics processing unit (GPU), a neural processing unit (NPU), and a tensor processing unit (TPU), but is not limited thereto.

[0054] The second memory pool 122 may include one or more memories. For example, the second memory pool 122 may include, but is not limited to, high-bandwidth memory (HBM). The second processing group 121 may have a structure efficient for parallel processing of neural network models (e.g., generative model 101). The second memory pool 122 may maximize the parallel processing performance of the second processing group 121 with wide bandwidth.

[0055] The memory size of the second memory pool 122 may be insufficient to store the generative model 101. According to an example, the electronic device 100 of one or more embodiments can efficiently perform the inference task of the generative model 101 by using collaborative processing of the first processing group 111 and the second processing group 121. According to an example, the first processing group 111 and the second processing group 121 can perform collaborative processing based on an optimization objective for the workload used for generative inference.

[0056] According to an example, electronic device 100 may determine an optimal task ratio between the first processing group 111 and the second processing group 121 for the inference task of generative model 101 based on the processing performance of the first processing group 111 and the second processing group 121 and / or the memory size of the second memory pool 122. Electronic device 100 of one or more embodiments may partition tensors used for the inference task (e.g., tensor sequences and weights associated with key-value (KV) caches) into subtensors, thereby minimizing communication overhead between the first processing group 111 and the second processing group 121 when processing subtensors, and placing the subtensors in the first memory pool 112 and the second memory pool 122 based on the task ratio. Here, "placing" tensors or subtensors in storage media (e.g., memory, memory pools, and / or memory nodes) or processing groups may mean storing tensors or subtensors in storage media or storage media corresponding to processing groups. The electronic device may perform fine-grained placement of subtensors based on locality and / or heterogeneity. The electronic device can use the first processing group 111 and the second processing group 121 to process the inference tasks assigned to the first processing group 111 and the second processing group 121 in parallel.

[0057] According to the example, electronic device 100 can perform generative inference when interacting with a user. The workload of generative inference can satisfy, for example, Quality of Service (QoS) objectives to improve the user experience. According to the example, electronic device 100 can perform generative inference to maximize throughput while satisfying QoS objectives.

[0058] For example, QoS can be determined based on latency. According to the example, the average token generation time (ATGT), representing the average execution time of each decoding stage of generative model 101, can be used as a latency metric. For example, ATGT can be represented by Equation 1 below.

[0059] Equation 1:

[0060] In equation 1, t decoding This indicates the execution time of the decoding phase, and 'o' can represent the number of output tokens. A single output token can be generated during the pre-padding phase.

[0061] To maximize throughput for generative inference workloads while meeting QoS objectives, increasing block and batch sizes can be limited. The size of the tensor sequence associated with the KV cache can be proportional to the block and batch sizes, thus the communication overhead between the first processing group 111 and the second processing group 121 for each tensor sequence can be constant. For example, the size of the weights can be constant. In this case, due to constraints on block and batch sizes, the communication overhead of the weights per tensor sequence may increase as the number of tensor sequences decreases. Therefore, the communication overhead of the weights may become a performance bottleneck. According to an example, the electronic device 100 of one or more embodiments can reduce the communication overhead of the weights by offloading operations performed using the weights to the first processing group.

[0062] Figure 2 Examples of inference operations for generative models according to one or more embodiments are shown. (Refer to...) Figure 2 The generative model 200 may include an input embedding layer 201, a transformer layer 202, and an output embedding layer 203. The generative model 200 may receive input prompts 211 and generate lexical units 213, 223, 233, 241, and 243 through a pre-filling stage 210, a first decoding stage 220, a second decoding stage 230, and a final decoding stage 240. In the final decoding stage 240, an inference result corresponding to the input prompt 211 may be generated based on lexical unit 243.

[0063] Input prompt 211 can be data in various formats (such as text, voice, images, video, etc.). For example, input prompt 211 can be a user's query or request, and the inferred result can be a response to the query or request.

[0064] Input embedding layer 201 can perform embedding and / or positional encoding on the input data. The input data can be input cue 211 or tokens 213, 223, 233, and 241. The input data in pre-filling stage 210 can be input cue 211, and the input data in the first decoding stage 220 to the final decoding stage 240 can be tokens 213, 223, 233, and 241. Based on the embedding of the input data, the format of the input data can be converted to a format that can be processed by transformer layer 202 (e.g., an input tensor). By positionally encoding the input data, the temporal and / or spatial relationships between elements of the input data (e.g., words in text, tiles in an image, etc.) can be defined. For example, input embedding layer 201 can add positional information to the input data.

[0065] Transformer layer 202 may execute attention mechanisms. For example, attention mechanisms may include, but are not limited to, self-attention. For example, input tensors may be input from input embedding layer 201 to each of transformer layers 202. Each of transformer layers 202 may execute an attention mechanism on each input tensor to generate an output tensor. When transformer layer 202 includes a first transformer layer and a second transformer layer, and the second transformer layer is the layer below the first transformer layer, the output tensor of the first transformer layer may be an input tensor of the second transformer layer.

[0066] Each of the transformer layers 202 may include query weights, key weights, and value weights. Each transformer layer 202 performs an attention mechanism by applying the query weights, key weights, and value weights to the input tensor. An attention result can be generated based on the attention mechanism, and an output tensor can be generated based on the attention result. The number of transformer layers 202 can be N, but is not limited to this.

[0067] The output embedding layer 203 can transform the output tensor of the transformer layer 202 into tokens 213, 223, 233, and 243. The output embedding layer 203 can operate in the reverse manner to the input embedding layer 201. For example, the output embedding layer 203 can perform an inverse transformation of the format of the input embedding layer 201 and / or removal of positional information.

[0068] Inference operations can be performed through the pre-filling stage 210, the first decoding stage 220, the second decoding stage 230, and the final decoding stage 240 of the generative model 200. The input cue 211 may include multiple lexical units. In the pre-filling stage 210, the generative model 200 can analyze the correlations between the lexical units of the input cue 211 based on an attention mechanism. In the first decoding stage 220, the second decoding stage 230, and the final decoding stage 240, the generative model 200 can generate lexical units 223, 233, and 243 based on the attention mechanism. The number of decoding stages can be M, but is not limited to this. An index i can be used to identify decoding stages 220 to 240. For example, the index of the final decoding stage 240 could be i=M.

[0069] According to the example, a KV cache can be executed when transformer layer 202 is performed. As decoding stages 220 through 240 proceed, redundant operations may occur when key tensors and value tensors are obtained (e.g., generated). A KV cache is a technique used to cache sequences of key tensors and value tensors to prevent redundant operations. A sequence can represent a data format in which tensors are concatenated (e.g., concatenated). For example, a sequence can be formed by adding a second tensor to a first tensor. Key tensors can form a sequence of key tensors, and value tensors can form a sequence of value tensors. A cache can represent the operation of storing sequences.

[0070] Figure 3 Examples of transformer layer configurations according to one or more embodiments are shown. Transformer layer of a generative model (e.g., Figure 2 Each of the converter layers 202 can be configured as follows: Figure 3 Transformer layer 300. (Refer to...) Figure 3 Transformer layer 300 may include, but is not limited to, multilayer perceptron (MLP) sublayer 310 and attention sublayer 320. The attention mechanism of transformer layer 300 may be executed in attention sublayer 320, but is not limited thereto.

[0071] Figure 4 Examples of operations for performing inference tasks of generative models using multiprocessor groups according to one or more embodiments are shown. Although Figure 4 Operations 410 to 440 are shown to be performed sequentially, but one or more operations may be performed in a different order, one or more operations may be omitted, two or more operations may be performed in parallel or simultaneously, and / or additional operations may be performed without departing from the spirit and scope of the described embodiments.

[0072] Reference Figure 4 In operation 410, the electronic device may determine a task ratio between a first processing group and a second processing group for an inference task of a generative model. For example, the electronic device may determine the task ratio based on, but is not limited to, the processing performance of the first and second processing groups. According to an example, the electronic device may experimentally determine the processing performance. In this case, the electronic device may measure the processing performance of the first and second processing groups based on microbenchmarks of one or more layers of the generative model (e.g., a transformer layer), but is not limited to, this. For example, the electronic device may measure the processing performance assuming maximum utilization of a second memory pool for the second processing group, but is not limited to, this. The electronic device may determine the task ratio to obtain maximum performance (e.g., maximum throughput).

[0073] As an example, an electronic device may receive and use data with pre-measured processing performance. For instance, performance data for obtaining maximum performance (e.g., maximum throughput) may be provided to the electronic device.

[0074] As an example, a generative model may include an attention sublayer and an MLP sublayer. For instance, a generative model may include a transformer layer, and the transformer layer may include an attention sublayer and an MLP sublayer. As an example, the task ratio may include a first task ratio (e.g., r) between a first processing group and a second processing group for tasks associated with the attention sublayer. SA,C r SA,G ) and the second task ratio (e.g., r) between the first and second processing groups for tasks related to the MLP sublayer.MLP,C r MLP,G For example, the ratio of the first task to the second processing group can be expressed as r. SA,C :r SA,G Furthermore, the ratio of the second task between the first processing group and the second processing group can be expressed as r. MLP,C :r MLP,G .

[0075] In operation 420, the electronic device may partition tensors used in the inference task into sub-tensors based on communication dependencies of computational phases of the inference task. Communication dependencies may indicate whether communication between a first processing group and a second processing group will be performed (e.g., required) to obtain the processing results of previous computational phases for processing in each computational phase.

[0076] For example, an inference task may include computational stages associated with an attention sublayer and computational stages associated with an MLP sublayer. The computational stages may be executed sequentially. A communication dependency of the current computational stage can be determined when communication between a first processing group and a second processing group is performed to obtain the processing results of a previous computational stage used for processing in the current computational stage. According to an example, when a communication dependency exists in the current computational stage, the first and second processing groups may process the current stage without tensor partitioning. When a communication dependency exists in the current computational stage, the tensor may be partitioned into subtensors, and the first and second processing groups may use the subtensors assigned to them to independently process the current stage.

[0077] For example, a tensor may include weights and / or a sequence of tensors. Weights may include query weights, key weights, value weights, and output weights of the attention sublayer, as well as weights of the MLP sublayer. A sequence of tensors may include a sequence of key tensors and a sequence of value tensors of the attention sublayer. The device may determine a first sub-tensor within a sub-tensor by partitioning a first tensor within a tensor according to "the attention head dimension, representing the dimension of the head direction of the multi-head attention of the generative model," and a second sub-tensor within a sub-tensor by partitioning a second tensor within a tensor according to "the column direction dimension, which is one of the row direction dimension and column direction dimension of the second tensor." The first tensor may include one or more of the query weights, key weights, value weights, key tensor sequences, and value tensor sequences of the attention sublayer, and the second tensor may include one or more of the output weights of the attention sublayer and weights of the MLP sublayer of the generative model.

[0078] In operation 430, the electronic device may place (e.g., store) sub-tensors in a first memory pool of a first processing group and a second memory pool of a second processing group based on a task ratio. For example, the electronic device may place a first sub-tensor in the first memory pool and the second memory pool by partitioning a first sub-tensor according to a first task ratio, and place a second sub-tensor in the first memory pool and the second memory pool by partitioning a second sub-tensor according to a second task ratio.

[0079] According to the example, the first memory pool may include memory nodes with a first bandwidth and memory nodes with a second bandwidth. For example, the memory node with the first bandwidth may be a main memory node (e.g., a DRAM memory node), and the memory node with the second bandwidth may be an extended memory node (e.g., a CXL memory node), but the memory nodes are not limited thereto. The electronic device may place subtensors in the memory nodes with the first bandwidth and the memory nodes with the second bandwidth based on the ratio between the first bandwidth and the second bandwidth. For example, when the ratio between the first bandwidth and the second bandwidth is β... SNC :β CXL At that time, electronic devices can use β SNC :β CXL The ratio places the sub-tensor in a memory node with a first bandwidth and a memory node with a second bandwidth. The electronic device can place the sub-tensor in a first memory pool in units smaller than a page (tile). For example, the page size could be 4 kilobytes (KB), and the tile unit could be 1 KB, but this is not an isolated example.

[0080] In operation 440, the electronic device can perform an inference task using subtensors by a first processing group and a second processing group. The inference task can be processed in parallel by the first processing group and the second processing group.

[0081] For example, the electronic device can process the current attention sublayer to generate the attention output tensor of the current attention sublayer, and process the current MLP sublayer of the generative model based on the attention output tensor of the current attention sublayer to generate the MLP output tensor of the current MLP sublayer. The generative model may include multiple transformer layers, and each transformer layer may include an attention sublayer and an MLP sublayer. The electronic device may process multiple transformer layers sequentially. For example, the electronic device may process the attention sublayer and MLP sublayer of each transformer layer sequentially. For example, the electronic device may process the previous attention sublayer and previous MLP sublayer of a previous transformer layer, and in response to processing the previous attention sublayer and previous MLP sublayer, process the current attention sublayer and current MLP sublayer of the current transformer layer.

[0082] The electronic device can, without communication between the first and second processing groups, independently generate a first part of the MLP output tensor and a second part of the MLP output tensor of the previous MLP sublayer of the generative model; based on communication between the first and second processing groups, generate the attention input tensor of the current attention sublayer of the generative model by merging the first part of the MLP output tensor and the second part of the MLP output tensor; and, without communication between the first and second processing groups, independently generate a first part of the attention fraction tensor and a second part of the attention fraction tensor of the current attention sublayer based on the attention input tensor and the first subtensor. The first subtensor can be obtained by partitioning the tensor according to the attention head dimension, which represents the head direction of the multi-head attention of the generative model.

[0083] The electronic device can generate the attention fraction tensor of the current attention sublayer by merging a first partial attention fraction tensor and a second partial attention fraction tensor based on communication between a first processing group and a second processing group. Furthermore, in the absence of communication between the first and second processing groups, the first and second processing groups can independently generate the first partial attention output tensor and the second partial attention output tensor of the current attention sublayer based on the attention fraction tensor and the second subtensor. The second subtensor can be obtained by partitioning the tensor according to the column direction dimension of the second tensor's row and column directions.

[0084] An example of the processing procedures for the attention sublayer and the MLP sublayer will be described in more detail later.

[0085] Figure 5 An example of the task ratio between a first processing group and a second processing group according to one or more embodiments is shown. (Refer to...) Figure 5 The electronic device (e.g., electronic device 100) may include a first processing group 511, a first memory pool 512, a second processing group 521, and a second memory pool 522. See reference... Figure 5 The electronic device may include a first task ratio (e.g., r) between a first processing group 511 and a second processing group 521 for tasks related to the attention sublayer 520. SA,C r SA,G ), and a second task ratio (e.g., r) between the first processing group 511 and the second processing group 521 for tasks related to the MLP sublayer 510. MLP,C r MLP,G For example, the first task ratio between the first processing group 511 and the second processing group 521 can be expressed as r. SA,C :r SA,GFurthermore, the second task ratio between the first processing group 511 and the second processing group 521 can be expressed as r. MLP,C :r MLP,G .

[0086] The electronic device can determine the task ratio for the inference task based on the processing performance of the first processing group 511 and the second processing group 521 and / or the memory size of the second memory pool 522. For example, the electronic device can determine the task ratio based on, but is not limited to, the processing performance of the first processing group 511 and the second processing group 521. For example, the processing performance can be determined experimentally. The electronic device can measure the processing performance of the first processing group 511 and the second processing group 521 based on micro-benchmarks of one or more layers of a generative model (e.g., a converter layer). For example, the electronic device can measure the processing performance assuming maximum usage of the second memory pool 522 of the second processing group 521, but is not limited to. In another example, the electronic device can use data with pre-measured processing performance.

[0087] Regarding the task ratio, for example, the relationship between Equations 2 and 3 below can be satisfied.

[0088] Equation 2:

[0089] Equation 3:

[0090] Generative inference can exhibit characteristics highly dependent on memory bandwidth. According to the example, the second processor of the second processing group 521 may include a GPU, and the second memory pool 522 may include HBM, but the example is not limited to this. In this case, when the second processing group 521 has higher memory bandwidth than the first processing group 511, the electronic device can maximize the use of the second memory pool 522 to meet the QoS objectives of the generative inference workload and maximize throughput. Regarding the use of the second memory pool 522, conditions can be defined, for example, as shown in Equation 4 below.

[0091] Equation 4:

[0092] In equation 4, s SA It can represent the size of the tensor sequence associated with the weights of all attention sublayers and the KV cache of the generative model, s MLP The value m can represent the magnitude of the weights of all MLP sublayers in a generative model. G This can represent the size of the second memory pool 522.

[0093] r SA,C r SA,G rMLP,C and r MLP,G It can be a variable, and s SA s MLP and m G It can be a constant. The three linear equations in Equations 2 through 4 can be expressed based on four variables (e.g., r). SA,C r SA,G r MLP,C and r MLP,G The constraints of equations 2 through 4. Once the values ​​of a single variable in equations 2 through 4 are determined, the values ​​of the remaining variables are obtained. The electronic device may experimentally obtain the values ​​of a single variable or may use a single predetermined value of a single variable, but the examples are not limited thereto. For example, the electronic device may measure the processing performance of a first processing group and a second processing group based on micro-benchmarking of one or more layers (e.g., a converter layer) of a generative model. For example, the electronic device may perform micro-benchmarking based on optimization techniques (e.g., binary search), but is not limited thereto.

[0094] Electronic devices can determine the batch size and a variable value that maximize throughput while meeting QoS objectives. For example, electronic devices can experimentally determine r. SA,G And in response to experimentally determining r SA,G Use equations 2 to 4 to determine r SA,C r MLP,C and r MLP,G .

[0095] The electronic device can obtain the execution time of the decoding phase of a generative inference workload (e.g., t) based on the number of layers, sequence length, and layer execution time of the decoding phase as measured by micro-benchmarks. decoding The electronic device can search for the maximum batch size that satisfies QoS objectives based on micro-benchmarking. For example, the electronic device can perform micro-benchmarking based on optimization techniques (e.g., binary search), but is not limited to this. The electronic device can set a variable value (e.g., r) for the maximum batch size. SA,G The ratio of tasks that maximize throughput while satisfying QoS is obtained from this (e.g., r). SA,C r SA,G r MLP,C and r MLP,G ).

[0096] Figure 6 Examples of subtensors partitioned according to the attention head dimension are shown according to one or more embodiments. Figure 7 Examples of subtensors partitioned according to the column direction dimension are shown according to one or more embodiments.

[0097] One or more embodiments of the electronic device may partition a tensor into sub-tensors, thereby minimizing communication between a first processing group and a second processing group while processing the sub-tensors, and allocating the sub-tensors to a first memory pool of the first processing group and a second memory pool of the second processing group according to a obtained task ratio. For example, a tensor may include weights and / or tensor sequences. For example, weights may include query weights, key weights, value weights, and output weights of an attention sublayer, and weights of an MLP sublayer. Tensor sequences may include key tensor sequences and value tensor sequences of an attention sublayer.

[0098] The electronic device can determine a first sub-tensor within a sub-tensor by partitioning a tensor according to the attention head dimension, which represents the head direction of the multi-head attention of the generative model, and a second sub-tensor within a sub-tensor by partitioning a tensor according to the column direction dimension, which represents the row direction dimension and column direction dimension of the second tensor. The first tensor may include one or more of the query weights, key weights, value weights, key tensor sequences, and value tensor sequences of the attention sub-layer, and the second tensor may include one or more of the output weights of the attention sub-layer and the weights of the MLP sub-layer of the generative model.

[0099] According to the example, the electronic device can partition tensors used in an inference task into sub-tensors based on communication dependencies between computational phases of the inference task. Communication dependencies can indicate whether communication between a first processing group and a second processing group will be performed to obtain the processing results of previous computational phases for each computational phase. For example, the inference task may include computational phases associated with an attention sublayer and computational phases associated with an MLP sublayer. The computational phases may be executed sequentially.

[0100] When communication between a first processing group and a second processing group is about to be performed to obtain the processing results of a previous computational stage for the current computational stage, it can be determined that the current computational stage has a communication dependency. According to the example, when the current computational stage has a communication dependency, the first and second processing groups can process the current stage without tensor partitioning. When the current computational stage has a communication dependency, the tensor can be partitioned into subtensors, and the first and second processing groups can use the subtensors to process the current stage.

[0101] For example, Table 1 below may show the computational stages associated with the attention sublayer, and Table 2 below may show the computational stages associated with the MLP sublayer.

[0102] Table 1:

[0103] Table 2:

[0104] Generative models may include multiple transformer layers, and each transformer layer may include an attention sublayer and an MLP sublayer. Electronic devices may process multiple transformer layers sequentially. For example, an electronic device may process the attention sublayer and MLP sublayer of each transformer layer sequentially. For example, an electronic device may execute the previous attention sublayer and previous MLP sublayer of a previous transformer layer, and in response to executing the previous attention sublayer and previous MLP sublayer, process the current attention sublayer and current MLP sublayer of the current transformer layer. In this disclosure, `gather()` may correspond to a function performing an aggregation operation, and `normalize()` may correspond to a function performing a normalization operation. f relu () or f_relu() can correspond to the modified linear unit (relu) function, and f softmax () or f_softmax() can correspond to the softmax function.

[0105] Tables 1 and 2 illustrate the processing flow of the current attention sublayer and the current MLP sublayer. The electronic device can generate the attention output tensor T of the current attention sublayer by processing it. SA,output And through the attention output tensor T based on the current attention sublayer SA,output Process the current MLP sublayer to generate the MLP output tensor T of the current MLP sublayer. MLP,output .

[0106] For example, in calculation phases 5f and 5g of Table 2, the electronic device can generate, independently by the first and second processing groups, the first part of the MLP output tensor of the previous MLP sublayer (e.g., the MLP output tensor T of the previous MLP sublayer) from the first and second processing groups without communication between them. MLP,output (a portion) and the second portion of the MLP output tensor (e.g., the MLP output tensor T of the previous MLP sublayer) MLP,output (The remaining part). In calculation phase 4a of Table 1, the electronic device can generate the attention input tensor of the current attention sublayer of the generative model (e.g., the attention input tensor T of the current attention sublayer) based on the communication between the first processing group and the second processing group by merging (e.g., performing an aggregation operation) the first part of the MLP output tensor and the second part of the MLP output tensor. SA,input ).

[0107] In calculation phases 4b to 4k of Table 1, the electronic device can generate, without communication between the first and second processing groups, a first partial attention fraction tensor of the current attention sub-layer (e.g., the attention fraction tensor T of the current attention sub-layer) independently by the first and second processing groups based on the attention input tensor and the first sub-tensor. SA,5 (a portion) and the second part of the attention fraction tensor (e.g., the attention fraction tensor T of the current attention sublayer) SA,5 (The remaining part). The first processing group can generate a first part of the MLP output tensor and a first part of the attention fraction tensor, and the second processing group can generate a second part of the MLP output tensor and a second part of the attention fraction tensor.

[0108] The attention sublayer may have a structure for multi-head attention. The electronic device can determine the first sub-tensor within the sub-tensor by partitioning the tensor according to the attention head dimension, which represents the head direction of the multi-head attention in the generative model. The first tensor may include the attention score tensor T used to obtain the attention score tensor. SA,5 Tensors (e.g., query weight W) Q Key weight W K Value weight W V Key tensor sequence X K Sum tensor sequence X V Independent parallel processing between the first and second processing groups can be optimized in computation phases 4b to 4k through "tensor partitioning based on attention head dimension".

[0109] In calculation phase 4l of Table 1, the electronic device can, based on communication between the first processing group and the second processing group, combine (e.g., aggregate operation) the first part of the attention fraction tensor (e.g., the attention fraction tensor T of the current attention sublayer) by the first and second processing groups. SA,5 (a portion) and the second part of the attention fraction tensor (e.g., the attention fraction tensor T of the current attention sublayer) SA,5 The remaining part is used to generate the attention score tensor T of the current attention sublayer. SA,5 Furthermore, in the absence of communication between the first and second processing groups, the first and second processing groups base their decisions on the attention fraction tensor T. SA，5 The second sub-tensor independently generates the first part of the attention output tensor (e.g., the attention output tensor T of the current attention sub-layer). SA,output (a portion) and the second portion of the attention output tensor (e.g., the attention score tensor T of the current attention sublayer) SA,output (The remaining part). The first processing group can generate the first part of the attention output tensor, and the second processing group can generate the second part of the attention output tensor.

[0110] The electronic device can determine the second sub-tensor within the sub-tensor by partitioning the second tensor within the tensor according to the column direction dimension, which is one of the row-direction and column-direction dimensions of the second tensor. The second tensor may include the attention output weights W of the attention sublayer. o The MLP weights W1 and W2 of the MLP sublayers can be optimized for independent parallel processing between the first and second processing groups in computation phases 4m, 4n, 5b to 5d, 5f, and 5g via "column-direction-based tensor partitioning".

[0111] The calculation phases 5a to 5g in Table 2 can also be performed in a similar manner to the calculation phases 4a to 4n in Table 1.

[0112] Computation stages 4a, 4l, 5a, and 5e can be determined to be performed using communication between the first and second processing groups. In other words, computation stages 4a, 4l, 5a, and 5e can have communication dependencies. The communication volume can be 2 × b × h1, where b represents the batch size and h1 represents the tensor (e.g., the MLP output tensor T). MLP,output Attention fractional tensor T SA,5 The dimension of the head direction. The first and second processing groups can process computation stages 4a, 4l, 5a, and 5e based on communication between them. Computation stages 4b to 4k, 4m, 4n, 5b to 5d, 5f, and 5g can be determined to be executed without communication between them. In the absence of communication, the first and second processing groups can use a first subtensor to process computation stages 4b to 4k and a second subtensor to process computation stages 4m, 4n, 5b to 5d, 5f, and 5g.

[0113] According to the example, the electronic device can place sub-tensors in a first memory pool of a first processing group and a second memory pool of a second processing group based on a task ratio. For example, the electronic device can place sub-tensors in a first memory pool of a first processing group and a second memory pool of a second processing group based on a first task ratio (e.g., r). SA,C :r SA,G The first sub-tensor is divided into a first memory pool and a second memory pool, and then placed in the first memory pool according to the second task ratio (e.g., r). MLP,C :r MLP,G The second subtensor is partitioned and placed in the first and second memory pools. For example, when the number of heads in a generative model's multi-attention head is n... h At that time, the electronic device can connect the first subtensor to r. SA,C ×n h The sub-tensors corresponding to each size are placed in the first memory pool, and will be combined with the remaining r SA,G ×n hThe sub-tensors corresponding to each head are placed in the second memory pool.

[0114] Reference Figure 6 The electronic device can divide tensor 610 into subtensors 620, place one or more subtensors 620 (e.g., subtensors from head 1 to head m-1) in a first processing group 650, and place the remaining subtensors 620 (e.g., subtensors from head m to the last head) in a second processing group 660. For example, the electronic device can divide query weight 611 into subtensors (e.g., W). Q,1 W Q,2 ... W Q,m W Q,m+1 ...etc.), dividing the key weights 612 into subtensors (e.g., W... K,1 W K,2 ... W K,m W K,m+1 ...etc.), and divide the value weight 613 into subtensors (e.g., W). V,1 W V,2 ... W V,m W V,m+1 (e.g., ...). Tensor 610 can be the first tensor, and subtensor 620 can be the first subtensor. The electronic device can obtain the first subtensor by partitioning the first tensor according to the attention head dimension, where the attention head dimension represents the dimension of the head direction of the multi-head attention in the generative model.

[0115] Reference Figure 7 The electronic device can divide tensor 710 into subtensors 720, and divide one or more subtensors 720 (e.g., weight W) into subtensors 720. C The remaining subtensors 720 (e.g., weights W) are placed in the first processing group 750, and the remaining subtensors 720 (e.g., weights W) are placed in the first processing group 750. G The tensor 710 can be a second tensor, and the subtensor 720 can be a second subtensor. The electronic device can obtain the second subtensor by partitioning the second tensor according to the column direction dimension, which is one of the row direction dimension and the column direction dimension of the second tensor. The partitioning direction 701 can represent the column direction.

[0116] Figure 8 An example of the processing structure of a first processing group according to one or more embodiments is shown. (Refer to...) Figure 8 The electronic device 800 may include first processors 810 and 840, main memory groups 820 and 850, and extended memory groups 830 and 860. The first processors 810 and 840 may form a first processing group, and the main memory groups 820 and 850 and the extended memory groups 830 and 860 may form a first memory pool.

[0117] The first processor 810 may include a core block 811 and one or more memory controllers (MCs) 812, the core block 811 including multiple cores, and the first processor 840 may include a core block 841 and one or more MCs 842. MCs 812 may control main memory group 820, and MCs 842 may control main memory group 850. Main memory group 820 may include memories 821 and 822, and main memory group 850 may include memories 851 and 852. The first processor 810 may be connected to an extended memory group 830, and the first processor 840 may be connected to an extended memory group 860.

[0118] Main memory group 820 and / or extended memory group 830 may be located in a local area of ​​the first processor 810 and may correspond to the local memory of the first processor 810. Main memory group 850 and / or extended memory group 860 may be located in a local area of ​​the first processor 840 and may correspond to the local memory of the first processor 840.

[0119] First processors 810 and 840 may each correspond to a first processing node. Main memory groups 820 and 850 and extended memory groups 830 and 860 may each correspond to a first memory node. Memory groups 821, 822, 851 and 852 may include DRAM, and extended memory groups 830 and 860 may include CXL memory, but the examples are not limited thereto.

[0120] According to the example, each of the first processors 810 and 840 may include multiple memory modules (MCs). Each MC may have a sub-non-uniform memory access (sub-NUMA) cluster (SNC) node. For example, when the first processors 810 and 840 each include p MCs, the main memory group connected to each MC of the first processors 810 and 840 may form an SNC node. In this case, each of the first processors 810 and 840 may have p SNC nodes. Each SNC node may be matched with a core that is physically located near the memory connected via the MC of the corresponding SNC node.

[0121] According to the example, the number of first processors in electronic device 800 can be n. C The number of SNC nodes for each first processor can be n. SNC And the number of attention heads can be n h Electronic device 800 can connect the first subtensor to r. SA,C ×n h The sub-tensors corresponding to each size are placed in the first memory pool, and will be combined with the remaining r SA,G ×n h The sub-tensor corresponding to each size is placed in the second memory pool. When the first sub-tensor is compared with r... SA,C ×nh When the subtensor corresponding to the size is placed in the first memory pool, the electronic device 800 can connect it with (r) SA,C ×n h ) / (n C ×n SNC The sub-tensors corresponding to each attention head are placed in the corresponding SNC node.

[0122] Figure 9 Examples of operations for placing sub-tensors in a heterogeneous memory pool according to one or more embodiments are shown. (Refer to...) Figure 9 The electronic device may place sub-tensor 910 in main memory pool 921 and extended memory pool 922. For example, sub-tensor 910 may be a sub-tensor among sub-tensors placed in a first memory pool. Main memory pool 921 may be a memory pool of a main memory group, and extended memory pool 922 may be a memory pool of an extended memory group. For example, main memory pool 921 may include DRAM, and extended memory pool 922 may include CXL memory, but the examples are not limited thereto.

[0123] According to the example, the first processor can provide extended capabilities to improve the performance of machine learning workloads. For example, the first processor can support operations and memory access in blocks. The electronic device can place the sub-tensor 910 in blocks smaller than pages in the main memory pool 921 and / or the extended memory pool 922. For example, the page size can be 4 kilobytes (KB), and the block unit can be 1 KB, but the example is not limited to this.

[0124] According to the example, the electronic device can perform fine-grained placement of subtensors based on locality and / or heterogeneity. According to the example, the electronic device can place subtensor 910 in main memory pool 921 and / or extended memory pool 922 by taking locality into account. For example, the electronic device can place subtensor 910 in a memory node in a local region of a first processor.

[0125] According to the example, the electronic device can place the sub-tensor 910 in a main memory pool 921 and an extended memory pool 922 based on heterogeneity. For example, the main memory pool 921 may have a first bandwidth, and the extended memory pool 922 may have a second bandwidth. The electronic device can place the sub-tensor 910 in the main memory pool 921 and the extended memory pool 922 based on the ratio between the first bandwidth and the second bandwidth. For example, when the ratio between the first bandwidth and the second bandwidth is β... SNC :β CXL At that time, electronic devices can use β SNC :β CXLThe ratio of the number of subtensors placed in memory nodes with a first bandwidth and memory nodes with a second bandwidth is used. For example, the electronic device can place the subtensor 910 in main memory pool 921 and extended memory pool 922 in blocks. For example, the ratio between the number of blocks placed in main memory pool 921 and the number of blocks placed in extended memory pool 922 can be β. SNC :β CXL .

[0126] Figure 10A This illustrates an example of processing of the attention sublayer by a first processing group according to one or more embodiments. Figure 10B An example of the processing of the attention sublayer by a second processing group according to one or more embodiments is shown. Figure 10A and Figure 10B The processing procedure can be identified as computational stages 4a to 4n related to the attention sublayer in Table 1.

[0127] Reference Figure 10A In computation phase 4a, the first processing group can merge (e.g., aggregation operation) the first part of the MLP output tensor T_MLP,output (i.e., T_MLP,output) of the previous MLP sublayer generated by the first processing group. MLP,output ) and the second part of the MLP output tensor T_MLP,output (i.e., T) generated by the second processing group from the previous MLP sublayer. MLP,output ), to generate the attention input tensor T_SA,input of the current attention sublayer (i.e., T SA,input In computation phase 4b, the first processing group can generate the first attention intermediate tensor T_SA,1 (i.e., T_SA,1) by normalizing the attention input tensor T_SA,input. SA,1 ).

[0128] In computation phase 4c, the first processing group can use the first attention intermediate tensor T_SA,1 and the query weight W_Q (i.e., W) Q ) perform matrix multiplication operations between them to generate the query tensor T_Q (i.e., T Q The first attention intermediate tensor T_SA,1 can have dimensions of h1×1×1, the query weight W_Q can have dimensions of hd×h1×3, and the query tensor T_Q can have dimensions of hd×1×3. hd can represent the attention head dimension, and h1 can represent the attention hidden dimension. When the number of attention heads is nh, h1 can be hd×nh.

[0129] exist Figure 10A and Figure 10B In the example, the query weight W_Q can be the first subtensor. Figure 10A and Figure 10BIn the example, the dimensions of all tensors for querying weight W_Q can be hd×h1×4, and the first task ratio (e.g., r_SA,C:r_SA,G) (i.e., r SA,C :r SA,G The ratio can be 3:1. Based on the first task ratio, the first sub-tensor of hd×h1×3 can be assigned to the first processing group, and the first sub-tensor of hd×h1×1 can be assigned to the second processing group.

[0130] During computation phase 4d, the first processing group can use the first attention intermediate tensor T_SA,1 and the key weights W_K (i.e., W) to... K ) perform matrix multiplication operations between them to generate the key tensor T_K (i.e., T K The dimension of the first attention intermediate tensor can be h1×1×1, the dimension of the key weight W_K can be hd×h1×3, and the dimension of the key tensor T_K can be hd×1×3. Similar to the query weight W_Q, the key weight W_K can also be the first subtensor.

[0131] In computation phase 4e, the first processing group can process the previous key tensor sequence X_K (i.e., X... K The key tensor sequence X_K is generated by concatenating the key tensor T_K with the key tensor T_K (i.e., X_K). K The dimension of the previous key tensor sequence X_K can be hd×L-1×3, and the dimension of the current key tensor sequence X_K can be hd×L×3.

[0132] In computation phase 4f, the first processing group generates the second attention intermediate tensor T_SA,2 (i.e., T_Q) by performing matrix multiplication between the query tensor T_Q and the transpose of the current key tensor sequence X_K (i.e., X_K^T). SA,2 In computation phase 4g, the first processing group can generate the third attention intermediate tensor T_SA,3 (i.e., T_SA,2) by inputting the second attention intermediate tensor T_SA,2 into the softmax function. SA,3 ).

[0133] During the computation phase 4h, the first processing group can use the first attention intermediate tensor T_SA,1 and the value weight W_V (i.e., W) to perform the computation. V ) perform matrix multiplication operations between them to generate the value tensor T_V (i.e., T V The dimension of the first attention intermediate tensor can be h1×1×1, the dimension of the value weight W_V can be hd×h1×3, and the dimension of the value tensor T_V can be hd×1×3. Similar to the query weight W_Q and the key weight W_K, the value weight W_V can also be the first subtensor.

[0134] In computation phase 4i, the first processing group can process the previous value tensor sequence X_V (i.e., X... V The current value tensor sequence X_V is generated by concatenating the tensor T_V with the value tensor T_V (i.e., X). V The dimension of the previous value tensor sequence X_V can be hd×L-1×3, and the dimension of the current value tensor sequence X_V can be hd×L×3.

[0135] In computation phase 4j, the first processing group can generate the fourth attention intermediate tensor T_SA,4 (i.e., T_SA,4) by performing a matrix multiplication operation between the third attention intermediate tensor T_SA,3 and the current value tensor sequence X_V. SA,4 In computation phase 4k, the first processing group can generate the fifth attention intermediate tensor T_SA,5 (i.e., T_SA,4) by concatenating the fourth attention intermediate tensor T_SA,4 (e.g., using the multi-head attention concatHeads() function). SA,5 The fifth attention intermediate tensor T_SA,5 can be the first part of the attention fraction tensor.

[0136] In computation phase 41, the first processing group can generate a sixth attention intermediate tensor T_SA,6 by merging (e.g., aggregation operation) the fifth attention intermediate tensor T_SA,5 generated by the first processing group and the fifth attention intermediate tensor T_SA,5 generated by the second processing group (i.e., T_SA,6). SA,6 ).

[0137] During computation phase 4m, the first processing group can be connected to the attention output weights W_o (i.e., W) via the intermediate tensor T_SA,6 of the sixth attention (i.e., W). o ) perform matrix multiplication operations between to generate the seventh attention intermediate tensor T_SA,7 (i.e., T SA,7 The dimension of the attention output weight W_o can be (h1×r_SA,C)×h1×1.

[0138] exist Figure 10A and Figure 10B In the example, the attention output weight W_o can be a second subtensor. Figure 10A and Figure 10B In the example, the dimension of all tensors of the attention output weights W_o can be h1×h1×1, and the first task ratio (e.g., r_SA,C:r_SA,G) can be 3:1. Based on the first task ratio, a second sub-tensor of h1×(h1×r_SA,C)×1 can be assigned to the first processing group, and a second sub-tensor of h1×(h1×r_SA,G)×1 can be assigned to the second processing group.

[0139] In computation phase 4n, the first processing group can generate the attention output tensor T_SA,output of the current attention sub-layer by merging the seventh attention intermediate tensor T_SA,7 and the attention input tensor T_SA,input of the current attention sub-layer. SA,output Here, the attention output tensor T_SA, output can be the first part of the attention output tensor.

[0140] Reference Figure 10B In computation phase 4a, the second processing group can generate the attention input tensor T_SA,input of the current attention sublayer by merging (e.g., by aggregation) the first part of the MLP output tensor T_MLP,output of the previous MLP sublayer generated by the first processing group and the second part of the MLP output tensor T_MLP,output of the previous MLP sublayer generated by the second processing group. In computation phase 4b, the second processing group can generate the first attention intermediate tensor T_SA,1 by normalizing the attention input tensor T_SA,input.

[0141] In computation phase 4c, the second processing group generates the query tensor T_Q by performing matrix multiplication between the first attention intermediate tensor T_SA,1 and the query weight W_Q. The first attention intermediate tensor T_SA,1 can have dimensions h1×1×1, the query weight W_Q can have dimensions hd×h1×1, and the query tensor T_Q can have dimensions hd×1×1. hd can represent the attention head dimension, and h1 can represent the attention hidden dimension. When the number of attention heads is nh, h1 can be hd×nh.

[0142] exist Figure 10A and Figure 10B In the example, the query weight W_Q can be the first subtensor. Figure 10A and Figure 10B In the example, the dimensions of all tensors for query weight W_Q can be hd×h1×4, and the first task ratio (e.g., r_SA,C:r_SA,G) can be 3:1. Based on the first task ratio, the first sub-tensor of hd×h1×3 can be assigned to the first processing group, and the first sub-tensor of hd×h1×1 can be assigned to the second processing group.

[0143] In computation phase 4d, the second processing group generates the key tensor T_K by performing matrix multiplication between the first attention intermediate tensor T_SA,1 and the key weight W_K. The first attention intermediate tensor can have a dimension of h1×1×1, the key weight W_K can have a dimension of hd×h1×1, and the key tensor T_K can have a dimension of hd×1×1. Similar to the query weight W_Q, the key weight W_K can also be the first subtensor.

[0144] In computation phase 4e, the second processing group can generate the current key tensor sequence X_K by concatenating the previous key tensor sequence X_K and the key tensor T_K. The dimension of the previous key tensor sequence X_K can be hd×L-1×1, and the dimension of the current key tensor sequence X_K can be hd×L×1.

[0145] In computation phase 4f, the second processing group generates a second attention intermediate tensor T_SA,2 by performing matrix multiplication between the query tensor T_Q and the transpose of the current key tensor sequence X_K. In computation phase 4g, the second processing group generates a third attention intermediate tensor T_SA,3 by inputting the second attention intermediate tensor T_SA,2 into the softmax function.

[0146] During computation phase 4h, the second processing group generates the value tensor T_V by performing matrix multiplication between the first attention intermediate tensor T_SA,1 and the value weight W_V. The dimension of the first attention intermediate tensor can be h1×1×1, the dimension of the value weight W_V can be hd×h1×1, and the dimension of the value tensor T_V can be hd×1×1. Similar to the query weight W_Q and the key weight W_K, the value weight W_V can also be the first subtensor.

[0147] In computation phase 4i, the second processing group can generate the current value tensor sequence X_V by concatenating the previous value tensor sequence X_V and the value tensor T_V. The dimension of the previous value tensor sequence X_V can be hd×L-1×1, and the dimension of the current value tensor sequence X_V can be hd×L×1.

[0148] In computation phase 4j, the second processing group generates a fourth attention intermediate tensor T_SA,4 by performing matrix multiplication between the third attention intermediate tensor T_SA,3 and the current value tensor sequence X_V. In computation phase 4k, the second processing group generates a fifth attention intermediate tensor T_SA,5 by concatenating the fourth attention intermediate tensor T_SA,4. The fifth attention intermediate tensor T_SA,5 can be a second part of the attention fraction tensor.

[0149] In computation phase 4l, the second processing group can generate the sixth attention intermediate tensor T_SA,6 by merging (e.g., by aggregation operation) the fifth attention intermediate tensor T_SA,5 generated by the second processing group and the fifth attention intermediate tensor T_SA,5 generated by the first processing group.

[0150] In computation phase 4m, the second processing group can generate the seventh attention intermediate tensor T_SA,7 by performing matrix multiplication between the sixth attention intermediate tensor T_SA,6 and the attention output weights W_o. The dimension of the attention output weights W_o can be (h1×r_SA,G)×h1×1.

[0151] exist Figure 10A and Figure 10B In the example, the attention output weight W_o can be a second subtensor. Figure 10A and Figure 10B In the example, the dimension of all tensors of the attention output weights W_o can be h1×h1×1, and the first task ratio (e.g., r_SA,C:r_SA,G) can be 3:1. Based on the first task ratio, a second sub-tensor of h1×(h1×r_SA,G)×1 can be assigned to the second processing group, and a second sub-tensor of h1×(h1×r_SA,C)×1 can be assigned to the first processing group.

[0152] In computation phase 4n, the second processing group can generate the attention output tensor T_SA,output of the current attention sub-layer by merging the seventh attention intermediate tensor T_SA,7 and the attention input tensor T_SA,input of the current attention sub-layer. Here, the attention output tensor T_SA,output can be the second part of the attention output tensor.

[0153] Figure 10C This illustrates an example of processing of the MLP sublayer by a first processing group according to one or more embodiments. Figure 10D An example of processing of the MLP sublayer by a second processing group is shown according to one or more embodiments. Figure 10C and Figure 10D The processing procedure can be identified as the computational stages 5a to 5g related to the MLP sublayer in Table 2.

[0154] Reference Figure 10C In computation phase 5a, the first processing group can generate the MLP input tensor T_MLP,input of the current MLP sublayer by merging (e.g., aggregation operation) the first part of the attention output tensor T_SA,output of the current attention sublayer generated by the first processing group and the second part of the attention output tensor T_SA,output of the current attention sublayer generated by the second processing group. MLP,input In computation phase 5b, the first processing group can generate the first MLP intermediate tensor T_MLP,1 (i.e., T_MLP,1) by normalizing the MLP input tensor T_MLP,input. MLP,1 ).

[0155] In computation phase 5c, the first processing group can generate the second MLP intermediate tensor T_MLP,2 (i.e., T) by performing matrix multiplication between the first MLP intermediate tensor T_MLP,1 and the MLP weights W_1 (i.e., W1). MLP,2 The first MLP intermediate tensor T_MLP,1 can have a dimension of h1×1×1, the MLP weights W_1 can have a dimension of h2×r_MLP,C×h1×1, and the second MLP intermediate tensor T_MLP,2 can have a dimension of (h2×r_MLP,C)×1×1. h2 can represent the hidden dimension of the MLP.

[0156] exist Figure 10C and Figure 10D In the example, the MLP weight W_1 can be the second subtensor. Figure 10C and Figure 10D In the example, the dimensions of all tensors of the MLP weight W_1 can be h2×h1×1. Based on the second task ratio (e.g., r_MLP,C:r_MLP,G (i.e., r...)... MLP,C :r MLP,G The second subtensor of (h2×r_MLP,C)×h1×1 can be assigned to the first processing group, and the second subtensor of (h2×r_MLP,G)×h1×1 can be assigned to the second processing group.

[0157] During computation phase 5d, the first processing group can generate the third MLP intermediate tensor T_MLP,3 (i.e., T_MLP,2) by inputting the second MLP intermediate tensor T_MLP,2 into the ReLU function. MLP,3 ).

[0158] In computation phase 5e, the first processing group can generate a fourth MLP intermediate tensor T_MLP,4 (i.e., T_MLP,4) by merging (e.g., aggregation operation) the third MLP intermediate tensor T_MLP,3 generated by the first processing group and the third MLP intermediate tensor T_MLP,3 generated by the second processing group. MLP,4 ).

[0159] In computation phase 5f, the first processing group can generate the fifth MLP intermediate tensor T_MLP,5 (i.e., T_MLP,4) by performing matrix multiplication between the fourth MLP intermediate tensor T_MLP,4 and the MLP weights W_2 (i.e., W2). MLP,5 The dimension of MLP weight W_2 can be (h1×r_MLP,C)×h2×1. Like MLP weight W_1, MLP weight W_2 can also be a second subtensor.

[0160] In computation phase 5g, the first processing group can generate the MLP output tensor T_MLP,output of the current MLP sub-layer by merging the fifth MLP intermediate tensor T_MLP,5 and the MLP input tensor T_MLP,input of the current MLP sub-layer. MLP,output Here, the MLP output tensor T_MLP,output can be the first part of the MLP output tensor.

[0161] Reference Figure 10D In computation phase 5a, the second processing group can generate the MLP input tensor T_MLP,input of the current MLP sublayer by merging (e.g., by aggregation) the first part of the attention output tensor T_SA,output of the current attention sublayer generated by the first processing group and the second part of the attention output tensor T_SA,output of the current attention sublayer generated by the second processing group. In computation phase 5b, the second processing group can generate the first MLP intermediate tensor T_MLP,1 by normalizing the MLP input tensor T_MLP,input.

[0162] In computation phase 5c, the second processing group can generate a second MLP intermediate tensor T_MLP,2 by performing matrix multiplication between the first MLP intermediate tensor T_MLP,1 and the MLP weights W_1. The first MLP intermediate tensor T_MLP,1 can have a dimension of h1×1×1, the MLP weights W_1 can have a dimension of h2×r_MLP,G×h1×1, and the second MLP intermediate tensor T_MLP,2 can have a dimension of (h2×r_MLP,G)×1×1.

[0163] exist Figure 10C and Figure 10D In the example, the MLP weight W_1 can be the second subtensor. Figure 10C and Figure 10D In the example, the dimension of all tensors of the MLP weight W_1 can be h2×h1×1. Based on the second task ratio (e.g., r_MLP,C:r_MLP,G), the second sub-tensor (h2×r_MLP,C)×h1×1 can be assigned to the first processing group, and the second sub-tensor (h2×r_MLP,G)×h1×1 can be assigned to the second processing group.

[0164] In computation phase 5d, the second processing group can generate the third MLP intermediate tensor T_MLP,3 by inputting the second MLP intermediate tensor T_MLP,2 into the ReLU function.

[0165] In computation phase 5e, the second processing group can generate the fourth MLP intermediate tensor T_MLP,4 by merging (e.g., by aggregation operation) the third MLP intermediate tensor T_MLP,3 generated by the first processing group and the third MLP intermediate tensor T_MLP,3 generated by the second processing group.

[0166] In computation phase 5f, the second processing group generates the fifth MLP intermediate tensor T_MLP,5 by performing matrix multiplication between the fourth MLP intermediate tensor T_MLP,4 and the MLP weights W_2. The dimension of the MLP weights W_2 can be (h1×r_MLP,G)×h2×1. Like the MLP weights W_1, the MLP weights W_2 can also be the second subtensor.

[0167] In computation phase 5g, the second processing group can generate the MLP output tensor T_MLP,output of the current MLP sub-layer by merging the fifth MLP intermediate tensor T_MLP,5 and the MLP input tensor T_MLP,input of the current MLP sub-layer. Here, the MLP output tensor T_MLP,output can be the second part of the MLP output tensor.

[0168] Figure 11 Examples of configurations of electronic devices according to one or more embodiments are shown. (Refer to...) Figure 11 Electronic device 1100 may include one or more processors 1110, memory 1120, storage device 1130, input / output (I / O) devices 1140, and network interface 1150. These components may communicate with each other via communication bus 1160. For example, electronic device 1200 may be implemented as at least a part of, for example, mobile devices (such as mobile phones, smartphones, personal digital assistants (PDAs), netbooks, tablet computers, and laptop computers), wearable devices (such as smartwatches, smart bracelets, and smart glasses), and / or computing devices (such as desktop computers and servers).

[0169] One or more processors 1110 can execute instructions stored in memory 1120 or storage device 1130. When executed by one or more processors 1110, the instructions can cause electronic device 1100 to perform referenced functions. Figures 1 to 10D The described operation. For example, one or more processors 1110 may include a first processing group and a second processing group, and instructions may be executed by one or more first processors of the first processing group.

[0170] Memory 1120 may include a computer-readable storage medium or a computer-readable storage device. Memory 1120 may store instructions to be executed by one or more processors 1110, and may store related information while electronic device 1100 is executing software and / or applications. For example, memory 1120 may be or include a non-transitory computer-readable storage medium that stores instructions that, when executed by one or more processors 1110, configure one or more processors 1110 to perform [the instructions described herein]. Figures 1 to 11 Any, any combination, or all of the described operations and / or methods. For example, memory 1120 may correspond to a first memory pool. A second memory pool may include the memory of a second processing group of one or more processors 1110.

[0171] Storage device 1130 may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. Storage device 1130 may store a larger amount of information than memory 1120 for a longer period of time. For example, storage device 1130 may include magnetic hard disks, optical disks, flash memory, floppy disks, and / or any other form of non-volatile memory known in the art.

[0172] I / O device 1140 can receive input from the user via a keyboard and mouse in conventional input methods, and can also receive input from the user in newer input methods (such as touch input, voice input, and image input). For example, I / O device 1140 may include a keyboard, mouse, touchscreen, microphone, and / or any other device that detects input from the user and sends the detected input to electronic device 1100. I / O device 1140 can provide the output of electronic device 1100 to the user via visual, auditory, and / or tactile channels. I / O device 1140 may include, for example, a display, touchscreen, speaker, vibration generator, and / or any other device that provides output to the user. Network interface 1150 can communicate with external devices via wired or wireless networks.

[0173] Including about Figures 1 to 11The description includes an electronic device, a first processing group, a first memory pool, a second processing group, a second memory pool, a first processor, a main memory group, an extended memory group, a core block, a memory (MC), a memory, a main memory pool, an extended memory pool, a processor, a storage device, an I / O device, a network interface, a communication bus, an electronic device 100, a first processing group 111, a first memory pool 112, a second processing group 121, a second memory pool 122, a first processing group 511, a first memory pool 512, a second processing group 521, a second memory pool 522, a first processing group 650, a second processing group 660, a first processing group 750, a second processing group 760, an electronic device 800, a first processor 810 and 840, a main memory group 820 and 850, an extended memory group 830 and 860, a core block 811, an MC 812, memories 821 and 822, a core block 841, and an MC. 842, memories 851 and 852, main memory pool 921, extended memory pool 922, electronic device 1100, one or more processors 1110, memory 1120, storage device 1130, I / O device 1140, network interface 1150, and communication bus 1160 are implemented by or represent hardware components. As described above, or in addition to the above description, examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components performing the operations described in this application are implemented by computing hardware (e.g., one or more processors or a computer). A processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units (ALUs), digital signal processors (DSPs), microcomputers, programmable logic controllers, field-programmable gate arrays (FPGAs), programmable logic arrays (PLUs), microprocessors, or any other means or combination of means configured to respond to and execute instructions (e.g., code or encoding) in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by the processor or computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to the execution of instructions or software.For the sake of brevity, the singular terms “processor” or “computer” may be used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both, and thus while some references may be made to a single processor or computer, such references are also intended to refer to multiple processors or computers. For example, a single hardware component, or two or more hardware components, may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or processors and controllers, may implement a single hardware component, or two or more hardware components. As described above, or in addition to the foregoing description, the example hardware components may have any one or more different processing configurations, examples of which include: a single processor, a standalone processor, a parallel processor, a single-instruction single-data (SISD) multiprocessing, a single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing. Therefore, reference to a processor herein refers to a processing circuit system (e.g., a circuit system including one or more processing element circuits). One or more processors including a processing circuit system also refers to each processor including a processing circuit system, and some or all of one or more processors including the same processing circuit system. Furthermore, one or more processors and one or more controllers, as non-limiting examples, do not represent human processing or human control, but rather represent hardware components as described herein as non-limiting examples.

[0174] Performing the operations described in this application Figures 1 to 11 Shown and about Figures 1 to 11The methods discussed are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented as described above to implement instructions (e.g., computer or processor / processing device readable instructions) or software to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may perform a single operation, or two or more operations. References to processors, or one or more processors, configured to perform two or more operations as non-limiting examples indicate that processors, or two or more processors, are configured to jointly perform all of the two or more operations, and that two or more processors respectively perform any corresponding operation of the two or more operations (e.g., corresponding one or more processors are configured to perform each of the two or more operations, or any corresponding combination of one or more processors is configured to perform any corresponding combination of the two or more operations). Similarly, a reference to a processor-implemented method is a reference to a method executed by one or more processors or other processing or computing hardware of a device or system.

[0175] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above may be written as computer programs, code segments, or other executable instructions or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software may be written in any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding description herein, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0176] Instructions or software used to control computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, may be recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media, and are therefore not the signal itself. Thus, references to storage media herein refer to storage media hardware and not to transient media, not the signal itself. As described above, or in addition to the above description, examples of non-transitory computer-readable storage media include any one or more of the following: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drives (HDDs), solid-state drives (SSDs), card storage devices (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state drives, and / or any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner, and to provide instructions or software and any associated data, data files, and data structures to one or more processors or computers, such that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system, such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0177] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered descriptively only and not for limiting purposes. The description of features or aspects in each example should be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

[0178] Therefore, in addition to the above and all the accompanying drawings, the scope of the disclosure also includes the claims and their equivalents (i.e., all variations within the scope of the claims and their equivalents should be interpreted as included in the disclosure).

Claims

1. A method for generative model processing, comprising: Determine the task ratio between the first and second processing groups for the inference task of the generative model; Subtensors are generated by partitioning the tensors used for the inference task based on the communication dependencies of the computation phase of the inference task. The generated sub-tensors are stored in the first memory pool of the first processing group and the second memory pool of the second processing group based on the task ratio. as well as The inference task is performed by the first and second processing groups using the stored sub-tensors.

2. The method according to claim 1, wherein, The steps for determining the task ratio between the first processing group and the second processing group include: determining the task ratio based on the processing performance of the first processing group and the second processing group.

3. The method according to claim 1, wherein, Generative models include attention sublayers and multilayer perceptron sublayers, and The task ratio includes a first task ratio between the first processing group and the second processing group for tasks corresponding to the attention sublayer, and a second task ratio between the first processing group and the second processing group for tasks corresponding to the multilayer perceptron sublayer.

4. The method according to claim 1, wherein, The steps to generate a subtensor include: The first sub-tensor is generated by partitioning the first tensor within the tensor according to the attention head dimension, where the attention head dimension represents the dimension of the head direction of the multi-head attention in the generative model; and The second tensor is divided according to the column direction dimension of the row direction and column direction dimension of the second tensor in the tensor, and a second subtensor of the subtensor is generated.

5. The method according to claim 4, wherein, The first tensor includes any one or any combination of any two or more of the query weights, key weights, value weights, key tensor sequences, and value tensor sequences of the attention sublayer of the generative model, and The second tensor includes one or both of the weights of the multilayer perceptron sublayer and the output weights of the attention sublayer in the generative model.

6. The method according to claim 4, wherein, The task ratio includes a first task ratio between the first and second processing groups for tasks corresponding to the attention sublayer of the generative model, and a second task ratio between the first and second processing groups for tasks corresponding to the multilayer perceptron sublayer of the generative model. The steps of storing the subtensor in the first memory pool and the second memory pool include: The first sub-tensor is divided according to the first task ratio and stored in the first memory pool and the second memory pool; and The second sub-tensor is divided according to the second task ratio and stored in the first memory pool and the second memory pool.

7. The method according to claim 1, wherein, The step of storing a subtensor in a first memory pool and a second memory pool includes: storing the subtensor to be processed by a first processor among a plurality of first processors in a first processing group in a first memory node located in the local region of the first processor among a plurality of first memory nodes in the first memory pool.

8. The method according to claim 1, wherein, The first memory pool includes memory nodes with a first bandwidth and memory nodes with a second bandwidth, and The step of storing the subtensor in the first memory pool and the second memory pool includes: storing the subtensor in a memory node with the first bandwidth and a memory node with the second bandwidth based on the ratio between the first bandwidth and the second bandwidth.

9. The method according to claim 1, wherein, The steps of storing the subtensor in the first memory pool and the second memory pool include: storing the subtensor in the first memory pool in blocks smaller than pages.

10. The method according to any one of claims 1 to 9, wherein, The steps involved in performing an inference task include: The attention output tensor of the current attention sublayer is generated by processing the current attention sublayer of the generative model; and The multilayer perceptron output tensor of the current multilayer perceptron sublayer is generated by processing the attention output tensor of the current attention sublayer in the generative model.

11. The method according to claim 10, wherein, The steps for generating the attention output tensor of the current attention sublayer include: Without communication between the first processing group and the second processing group, the first processing group and the second processing group independently generate the first part of the multilayer perceptron output tensor and the second part of the multilayer perceptron output tensor of the previous multilayer perceptron sublayer of the generative model. Based on the communication between the first processing group and the second processing group, the attention input tensor of the current attention sublayer of the generative model is generated by combining the output tensors of the first and second parts of the multilayer perceptron; and Without communication between the first processing group and the second processing group, the first processing group and the second processing group independently generate the first part attention fraction tensor and the second part attention fraction tensor of the current attention sublayer based on the attention input tensor and the first subtensor.

12. The method according to claim 11, wherein, The first subtensor is generated by partitioning the tensor according to the attention head dimension, which represents the dimension of the head direction of the multi-head attention in the generative model.

13. The method according to claim 11, wherein the step of generating the attention output tensor of the current attention sublayer further comprises: Based on the communication between the first processing group and the second processing group, the attention fraction tensor of the current attention sublayer is generated by combining the first part of the attention fraction tensor and the second part of the attention fraction tensor. as well as Without communication between the first and second processing groups, the first and second processing groups independently generate the first and second partial attention output tensors of the current attention sublayer based on the attention fraction tensor and the second sub-tensor.

14. The method according to claim 13, wherein, The second subtensor is generated by partitioning the second tensor according to the column direction dimension of the row direction and column direction dimension of the second tensor within the tensor.

15. An electronic device comprising: One or more processors, including processing circuitry; as well as The memory includes one or more storage media storing instructions that, when executed individually or jointly by the one or more processors, cause the one or more processors to: Determine the task ratio between the first and second processing groups for the inference task of the generative model; Subtensors are generated by partitioning the tensors used for the inference task based on the communication dependencies of the computation phase of the inference task. The generated sub-tensors are stored in the first memory pool of the first processing group and the second memory pool of the second processing group based on the task ratio. as well as The inference task is performed by the first and second processing groups using sub-tensors.

16. The electronic device according to claim 15, wherein, In order to generate the subtensor, the execution of the instructions causes the one or more processors to: The first sub-tensor is generated by dividing the first tensor in the tensor according to the attention head dimension, where the attention head dimension represents the dimension of the head direction of the multi-head attention in the generative model. as well as The second tensor is divided according to the column direction dimension of the row direction and column direction dimension of the second tensor in the tensor, and a second subtensor of the subtensor is generated.

17. The electronic device according to claim 16, wherein, The task ratios include a first task ratio between a first processing group and a second processing group for tasks related to the attention sublayer of the generative model, and a second task ratio between a first processing group and a second processing group for tasks related to the multilayer perceptron sublayer of the generative model. In order to store the subtensor in the first memory pool and the second memory pool, the execution of the instruction causes the one or more processors to: The first sub-tensor is divided according to the first task ratio and stored in the first memory pool and the second memory pool; and The second sub-tensor is divided according to the second task ratio and stored in the first memory pool and the second memory pool.

18. The electronic device according to claim 15, wherein, In order to store the subtensor in the first memory pool and the second memory pool, the execution of the instruction causes the one or more processors to store the subtensor to be processed by the first processor among the plurality of first processors of the first processing group in the first memory node located in the local region of the first processor among the plurality of first memory nodes of the first memory pool.

19. The electronic device according to claim 15, wherein, The first memory pool includes memory nodes with a first bandwidth and memory nodes with a second bandwidth, and In order to store the subtensor in a first memory pool and a second memory pool, the execution of the instruction causes the one or more processors to: store the subtensor in a memory node with a first bandwidth and a memory node with a second bandwidth based on the ratio between the first bandwidth and the second bandwidth.

20. A method for generative model processing, comprising: The first sub-tensor is generated by partitioning the first tensor by the dimension of the head direction of the multi-head attention based on the generative model, and the first sub-tensor is stored in the first memory pool and the second memory pool based on the first task ratio. The second sub-tensor is generated by partitioning the second tensor according to the column direction dimension of the second tensor, and the second sub-tensor is stored in the first memory pool and the second memory pool according to the second task ratio. as well as An inference task is performed by using a first processing group to process the first and second subtensors stored in a first memory pool, and by using a second processing group to process the first and second subtensors stored in a second memory pool.

Images