Data processing apparatus, data processing method and related product

Abstract

The present disclosure relates to data processing apparatus, data processing methods and related products, wherein the data processing apparatus of the present disclosure is included in an integrated circuit apparatus which integrates one or more combined processing apparatuses, the combined processing apparatus being an artificial intelligence operation unit to support various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in the fields of computer vision, speech, natural language processing, data mining, etc.

Description

Technical Field

[0001] This disclosure generally relates to the field of data processing technology. More specifically, this disclosure relates to a data processing apparatus, a data processing method, and related products. Background Technology

[0002] In the field of modern deep learning, especially in tasks such as natural language processing and computer vision, the Transformer architecture and its variants have been widely used. The core of the Transformer architecture is the attention mechanism, which captures long-range dependencies between elements in a sequence, thereby significantly improving the model's ability to understand and generate complex data. Within the attention mechanism, the softmax operation plays a crucial role, converting attention scores into a probability distribution used to determine the relative importance weights between elements in different sequences.

[0003] Traditional softmax implementations have several shortcomings. Directly calculating softmax for large-scale sequences leads to enormous computational and storage overhead. Since attention mechanisms involve computation between elements at different positions in the sequence, the computational complexity and storage requirements of attention mechanisms increase quadratically with sequence length, causing a sharp rise in memory usage. Hardware storage and computing capabilities often struggle to meet these demands, especially when processing long sequences, where computational latency increases significantly, severely limiting model performance and efficiency.

[0004] In view of this, there is an urgent need to provide a data processing solution to improve the processing speed of softmax operation in the attention mechanism, while optimizing the utilization efficiency of hardware resources to meet the growing computing demands of deep learning. Summary of the Invention

[0005] In order to at least solve one or more of the technical problems mentioned above, this disclosure provides solutions for data processing apparatus, data processing method, integrated circuit apparatus, circuit board, computing device and computer-readable storage medium in several aspects.

[0006] In a first aspect, this disclosure provides a data processing apparatus, including a control circuit and a first processing circuit, wherein: the control circuit is configured to control the first processing circuit to cyclically execute at least a portion of the operations in an attention mechanism according to quantization groups, wherein the quantization group is a group divided into computational units according to sequence length in the attention mechanism; the first processing circuit includes: a maximum value calculation circuit, configured to perform row maximum value calculation of the attention matrix divided by quantization groups in the attention mechanism to obtain a local maximum value for each quantization group; an exponentiation operation circuit, configured to perform an exponentiation operation on a first difference between the attention matrix and the local maximum value to obtain a first intermediate result; and a quantization circuit, configured to perform a quantization operation on the first intermediate result based on quantization coefficients to obtain a probability matrix for each quantization group, wherein the quantization coefficients are the reciprocals of the maximum value that the quantized target data type can represent, and the probability matrix is ​​used for softmax operation in the attention mechanism.

[0007] In a second aspect, this disclosure provides a data processing method, comprising cyclically performing the following operations in an attention mechanism task by quantization groups, wherein the quantization groups are groups divided into computational units in the attention mechanism according to sequence length: calculating the row maximum value of the attention matrix divided by quantization groups in the attention mechanism task to obtain the local maximum value of each quantization group; performing an exponential operation on a first difference between the attention matrix and the local maximum value to obtain a first intermediate result; and performing a quantization operation on the first intermediate result based on quantization coefficients to obtain a probability matrix for each quantization group, wherein the quantization coefficients are the reciprocal of the maximum value that the quantized target data type can represent, and the probability matrix is ​​used for the softmax operation in the attention mechanism task.

[0008] In a third aspect, this disclosure provides an integrated circuit apparatus including the data processing apparatus described in the first aspect of this disclosure.

[0009] In a fourth aspect, this disclosure provides a board including the integrated circuit device described in the third aspect of this disclosure.

[0010] In a fifth aspect, this disclosure provides a computing device comprising: at least one processor; and at least one memory in communication with the at least one processor, having stored computer-readable instructions that, when loaded and executed by the at least one processor, cause the at least one processor to perform the data processing method described in the second aspect of this disclosure.

[0011] In a sixth aspect, this disclosure provides a computer-readable storage medium storing program instructions that, when loaded and executed by a processor, cause the processor to perform the data processing method according to the second aspect of this disclosure.

[0012] Through the data processing scheme provided above, this embodiment controls the first processing circuit to cyclically execute at least some operations in the attention mechanism according to quantization groups. The attention matrix is ​​divided according to quantization groups, and the maximum value calculation circuit, exponentiation circuit, and quantization circuit in the first processing circuit are set to perform pipelined operations according to quantization groups, achieving deep hardware integration of softmax operation and MX quantization operation in the attention mechanism. This integration eliminates the cumbersome data transmission and multiple storage operations at the software level in the traditional softmax operation method, enabling the entire calculation process to be pipelined at the hardware level. Intermediate results are passed in the pipelined operation circuit, reducing the storage requirements of intermediate results, thereby significantly improving computational efficiency and reducing hardware storage and computational overhead. Simultaneously, the quantization operation is based on the quantization coefficient, which is the reciprocal of the maximum value that the target data type can represent after quantization. This ensures a balance between the numerical range and precision of the quantized probability matrix, improving computational accuracy and effectively reducing the complexity of the quantization operation. Attached Figure Description

[0013] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0014] Figure 1a A schematic diagram of the MX data format is shown;

[0015] Figure 1b A schematic block diagram of a data processing apparatus according to some embodiments of this disclosure is shown;

[0016] Figure 2 A schematic diagram illustrating the partitioning of the attention matrix by quantization group is shown for some embodiments of this disclosure;

[0017] Figure 3 A schematic diagram illustrating the calculation of probability matrices by quantization group in some embodiments of this disclosure is shown;

[0018] Figure 4 Schematic block diagrams of a data processing apparatus illustrating other embodiments of this disclosure;

[0019] Figure 5 A schematic block diagram illustrating a data processing apparatus according to further embodiments of this disclosure;

[0020] Figure 6 An exemplary flowchart illustrating a data processing method according to some embodiments of this disclosure is shown;

[0021] Figure 7 This diagram illustrates the structure of a circuit board according to an embodiment of the present disclosure.

[0022] Figure 8 It is shown Figure 7 A structural diagram of the combined processing device in chip 701 of the illustrated embodiment. Detailed Implementation

[0023] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0024] It should be understood that the terms “comprising” and “including” used in this disclosure and claims indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0025] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0026] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0027] First, we provide explanations of the technical terms that may be used in this disclosure.

[0028] Floating-point numbers: The IEEE floating-point standard uses V = (-1). sign*mantissa*2 E A number is represented in the form of . Here, `sign` is the sign bit, where 0 represents a positive number and 1 represents a negative number; `E` represents the exponent (or index), which is a weighted sum of 2 raised to the power of `E` (which may be a negative power); `mantissa` represents the mantissa, a binary fraction. Floating-point numbers are represented in computers using three fields, each encoded separately:

[0029] (1) A single sign bit directly encodes the symbol s;

[0030] (2) The exponent field of the k-bit is encoded as exp=e(k-1)......e(1)e(0);

[0031] (3) The n-digit mantissa is used to encode the mantissa. However, the encoding result depends on whether the exponent field is all 0.

[0032] Fixed-point numbers are a way of representing numbers in computers, characterized by a fixed decimal point. Fixed-point numbers can represent integers or decimals, depending on the implicit position of the decimal point. A fixed-point number consists of three parts: a shared exponent, a sign bit, and a mantissa. The shared exponent means that the exponent is shared within a set of real numbers to be quantized; the sign bit indicates whether the fixed-point number is positive or negative; and the mantissa determines the number of significant digits, i.e., the precision.

[0033] Bit width: How many bits are used to represent data.

[0034] Quantization: The process of converting high-precision numbers into lower-precision data that takes up less memory.

[0035] Dequantization: The process of converting low-precision numbers into higher-precision data.

[0036] FP*: Represents a floating-point data type, where * can be an integer such as 2, 4, 6, 8, 16, 32, 64, etc. The value of * indicates the bit width of the data represented by this data type.

[0037] BF16 and TF32: Data types suitable for deep learning, which improve computational performance by sacrificing precision and range.

[0038] int*: Represents an integer data type, where * can be an integer such as 2, 4, 6, 8, 16, 32, 64, etc. The value of * indicates the bit width of the data represented by this data type.

[0039] ExMy: A data format where E represents the exponent, M represents the mantissa, x represents the exponent width, and y represents the mantissa width.

[0040] MX: refers to the Microscaling specification (see “OCP Microscaling Formats (MX) Specification Version 1.0”).

[0041] It should be noted that in this disclosed embodiment, the data to be processed or already processed can be various types of data in a neural network, including but not limited to input neurons, weights, output neurons, gradients, etc. The data processing apparatus and data processing method provided in this disclosed embodiment can be applied to various fields and can process various types of data. For example, in image recognition and processing, facial recognition, object detection, image classification, and medical image analysis can include data types such as pixel data and image features; in natural language processing (NLP), language translation, sentiment analysis, text summarization, and speech recognition can include data types such as text data, word embeddings, and sentence structure; in speech recognition and processing, intelligent assistants, automatic caption generation, and speech-to-text conversion can include data types such as audio signals and spectrograms; in recommendation systems, personalized content recommendation, product recommendation, and advertising can include data types such as user behavior data, item features, and rating data; in healthcare, disease diagnosis, drug discovery, and gene sequence analysis can include data types such as medical records, biomarker data, and genomic data; in the financial field, risk assessment, fraud detection, and stock market prediction can include data types such as transaction data, user credit scores, and market data; and in autonomous driving, vehicle perception, decision-making, and path planning can include data types such as sensor data, environmental features, and traffic signals. In gaming and entertainment, AI applications such as game AI, virtual reality, and animation generation can include data types such as game state data and user interaction data. In scientific research, applications such as physics simulation, chemical compound prediction, and astrophysical analysis can include data types such as experimental data, simulation results, and observational data. Generative AI applications, on the other hand, utilize complex algorithms, models, and rules to learn from large-scale datasets to create new original content. This includes, but is not limited to, creating various types of content such as text, images, audio, video, and code. Correspondingly, the processed data can be image data, audio data, video data, speech data, text data, document data, etc., and the corresponding output data can also be image data, audio data, video data, speech data, text data, document data, etc. The output of the data processing apparatus or data processing method in this disclosure embodiment can include a probability score of an image belonging to a specific object category, a probability score of a document relating to a specific topic, a probability score of the correct translation of a text fragment in the target language being a text fragment in the source language, or a probability score of the correct transcription of a text fragment being spoken discourse, etc. By quantizing / dequantizing the data, the processing effect of these tasks can be guaranteed while reducing the computational load and saving computational resources.The data processing scheme of this disclosure embodiment can also be used in the inference and training of neural networks. By performing the data processing process described in this disclosure embodiment on the data of the neural network, the accuracy of the task can be guaranteed when the neural network performs one or more of the aforementioned processing tasks, while improving the speed and performance of the neural network model's inference and training.

[0042] Exemplary MX Quantization Scheme

[0043] Figure 1a A schematic diagram of the MX data format is shown. (For example...) Figure 1a As shown, the MX data format mainly consists of three parts: the scaling factor (X, shared scale), the scaling block size N (N, Scaling Block size), and the private element data within the block (Pi).

[0044] The scaling factor X relates to the data type or encoding of the scaling factor. The private element (Pi) also relates to its data type or encoding. The scaling block size (N) represents the number of private elements Pi within the block. All N elements Pi within the block have the same data type and therefore the same bit width. The scaling factor X is shared among these N elements. The data type of the elements and the data type of the scaling factor can be chosen independently. In a sense, MX can be viewed as a mechanism for constructing vector data types based on scalar data types. The values ​​V1, ..., V in an MX block represent... K It can be determined as follows: V i =XPi.

[0045] like Figure 1a As further illustrated, W represents the number of bits used to encode the shared scaling factor X, i.e., the bit width of scaling factor X; d represents the number of bits used to represent each element Pi, i.e., the bit width of element Pi. Therefore, an MX block containing N elements can be encoded using (W+Nd) bits. Table 1 shows some common formats and related parameters of MX.

[0046]

[0047]

[0048] Table 1

[0049] In Table 1 above, ExMy represents a scalar format notation with x exponent bits and y mantissa bits. For example, FP8(E4M3) represents an FP8 format with 1 sign bit, 4 exponent bits, and 3 mantissa bits. When y is 0 (e.g., E8M0), the format does not include a sign bit.

[0050] Furthermore, in MX, the scaling factor X is calculated as follows:

[0051] (1) Determine the largest power of the absolute value of the N elements Pi that are less than or equal to N elements;

[0052] (2) Divide the maximum power of 2 by the power of 2 of the maximum value that the data type of element Pi can represent, and set the result as the scaling factor ScaleX.

[0053] The specific embodiments disclosed herein will now be described in detail with reference to the accompanying drawings.

[0054] Exemplary data processing device

[0055] Figure 1b A schematic block diagram of a data processing apparatus according to some embodiments of this disclosure is shown. Figure 1b As shown, the data processing device 100 may include a control circuit 110 and a first processing circuit 120. The control circuit 110 is configured to control the first processing circuit 120 to cyclically execute at least a portion of the operations in the attention mechanism according to quantization groups. A quantization group is a group of computational units divided according to sequence length in the attention mechanism. The first processing circuit 120 includes: a maximum value calculation circuit 123, configured to perform row maximum value calculation of the attention matrix divided according to quantization groups in the attention mechanism to obtain the local maximum value of each quantization group; an exponentiation operation circuit 125, configured to perform an exponentiation operation on the first difference between the attention matrix and the local maximum value to obtain a first intermediate result; and a quantization circuit 126, configured to perform a quantization operation on the first intermediate result based on the quantization coefficient to obtain a probability matrix of each quantization group, wherein the quantization coefficient is the reciprocal of the maximum value that the quantized target data type can represent, and the probability matrix is ​​used for the softmax operation in the attention mechanism. For ease of understanding, the following will be combined with... Figure 2 An illustrative example is provided.

[0056] Figure 2 A schematic diagram illustrating the partitioning of the attention matrix by quantization group in some embodiments of this disclosure is shown. For example... Figure 2 As shown, in the attention mechanism, matrix multiplication (i.e., Q × K) is typically performed based on the query matrix Q and the key matrix K. T This yields the attention matrix S. For attention mechanism operations with a long sequence length (seq_len), to reduce the number of read / write operations to memory (especially high-bandwidth memory, HBM), the matrix involved in the computation can be divided into blocks. The sequence length mentioned in this paper can refer to the number of elements in the input sequence, that is, the number of elements in the matrix involved in the attention mechanism operation.

[0057] Specifically, for example Figure 2As shown, a long sequence of attention matrices S can be split into multiple computational units (or blocks) as indicated by the gray areas. Correspondingly, the query matrix Q and key matrix K can also be split into multiple computational units to perform corresponding operations. It is understood that, depending on the method of partitioning the computational units, at least some elements in the query matrix Q and key matrix K will be reused in different computational units. For example, when the first row of elements in the attention matrix S is partitioned into different computational units, the first row of elements in the query matrix Q will be reused in different computational units.

[0058] like Figure 2 As further illustrated in the embodiments disclosed herein, the computing unit can be further divided into multiple quantization groups (as shown in the thick outline in the figure), so that the control circuit 110 can be configured to control the first processing circuit 120 to cyclically execute the relevant operations in the attention mechanism, group by group. In some embodiments, the control circuit 110 can be configured to determine the number of quantization groups in a computing unit based on the size of the computing unit and the size of the quantization group, so as to divide the computing unit into multiple quantization groups. For example, assuming that the size of a computing unit (i.e., the number of elements) is 128, and the size of a quantization group is 32, then the computing unit can be divided into 4 quantization groups. It is understood that the size of the computing unit is not limited to 128, but can also be 256, or 512, etc.; the size of the quantization group is not limited to 32, but can also be 8, 16, 64, etc.

[0059] Compared to performing block-based computation on the matrices involved in the calculation (i.e., performing computation according to computational units), the operation according to quantization groups in the embodiments of this disclosure can further reduce the amount of data in a single operation, thereby improving the hardware's computing speed and reducing the output and storage of intermediate computation results, and thus enabling the following: Figure 1b The pipelined circuit architecture shown further reduces memory space requirements and read / write overhead.

[0060] In combination Figure 2 Having understood the basic principle of performing operations by quantization group as described in the embodiments of this disclosure, we will now return to... Figure 1b Continue the description.

[0061] like Figure 1b As shown, the maximum value calculation circuit 123 is configured to perform row maximum value calculations for the attention matrix divided by quantization groups in the attention mechanism, obtaining the local maximum value for each quantization group. That is, the maximum value calculation circuit 123 can calculate the row maximum value of the attention matrix for a corresponding quantization group in each round, for example, using the following formula:

[0062] m r=redmax(S r (Formula 1);

[0063] In the formula, m r Represents a local maximum, redmax represents the calculation of the row maximum, S r This represents the attention matrix divided by quantization groups.

[0064] For example, suppose that each row of the attention matrix divided by quantization groups contains 4 elements, then the maximum value of each row is the maximum value among the 4 elements, and the local maximum value of each quantization group is a vector that includes the maximum value of each row.

[0065] Furthermore, the exponentiation circuit 125 is configured to perform an exponential operation on the first difference between the attention matrix and the local maximum to obtain a first intermediate result, which can be calculated, for example, by the following formula:

[0066]

[0067] In the formula, P r S represents the first intermediate result calculated by quantization group. r This represents the attention matrix divided by quantization groups, m r This represents the local maximum value calculated by quantization group.

[0068] In softmax operations, the element values ​​of the attention matrix can vary significantly. Directly performing exponential operations on the original values ​​may lead to numerical overflow or underflow. By performing exponential operations on the first difference between the attention matrix and the local maximum, the input values ​​in subsequent operations can be limited to a smaller range. Since the local maximum is the maximum value within each quantization group, the range of the first difference is typically between negative and 0. The exponential operation circuit performs exponential operations on the first difference (negative or zero), mapping them to a range between 0 and 1. This effectively avoids the numerical overflow or underflow problem in exponential operations, improving the stability of numerical computation.

[0069] Further, the quantization circuit 126 is configured to perform a quantization operation on the first intermediate result based on the quantization coefficients to obtain a probability matrix for each quantization group. In some embodiments, the quantization operation includes a division operation between the first intermediate result and the quantization coefficients, or a multiplication operation between the first intermediate result and the reciprocal of the quantization coefficients. In other embodiments, the quantization operation may also include a rounding operation. For example, it can be calculated using the following formula:

[0070]

[0071] In the formula, P r ' represents the probability matrix calculated by quantization group, round represents the rounding operation, P rThis represents the first intermediate result calculated by quantization group, where scale represents the quantization coefficient, and S r This represents the attention matrix divided by quantization groups, m r This indicates the local maximum value calculated by quantization group, where u represents the maximum value that the target data type after quantization can represent.

[0072] The target data type is the data type of the quantized data obtained after quantizing the data to be quantized. Before the quantization operation, the target data type can be set as needed, and its precision is generally lower than that of the original data type of the data to be quantized (such as the first intermediate result). In some embodiments, the original data type of the data to be quantized may include at least one of FP16, BF16, FP32, FP64, TF32, etc.; the target data type may include at least one of FP8, FP6, FP4, FP2, int8, int4, int2, int6, FP12, etc. Among them, floating-point data types such as FP8, FP6, and FP4 can support multiple data formats. Taking FP4 as an example, it can be divided in several ways according to the rule of "sign bit + exponent bit + mantissa bit". Usually, the sign bit occupies 1 bit, and the common combinations of exponent and mantissa bits are E3M0 and E2M1. Similarly, FP8 can support data formats such as E4M3 and E5M2, and FP6 can support data formats such as E3M2 and E2M3.

[0073] Due to the bit width limitations of each data type, each data type has a maximum representable value. For example, the maximum representable value when the target data type is FP4 is 6; the maximum representable value when the target data type is int8 is 127. Typically, there is a one-to-one correspondence between the maximum value and the target data type, which can be obtained through methods such as table lookup.

[0074] Unlike the traditional MX quantization method that calculates scaling factors, this disclosed embodiment directly uses the reciprocal of the maximum value that the quantized target data type can represent as the quantization factor. This eliminates the need for complex dynamic scaling calculations, simplifies the quantization process, makes hardware implementation simpler, and ensures that the quantized data can fully utilize the representation range of the target data type, thus guaranteeing that the quantization accuracy is within a controllable range.

[0075] For ease of understanding Figure 1b The operation process of the maximum value calculation circuit, exponentiation circuit, and quantization circuit shown below will be explained in conjunction with... Figure 3 An illustrative example is provided.

[0076] Figure 3 A schematic diagram illustrating the calculation of probability matrices by quantized groups in some embodiments of this disclosure is shown. For example... Figure 3As shown in the figure, the attention matrix S, which is divided according to the computing unit, is divided into four quantization groups, namely the first quantization group S r 1 Second quantization group S r 2 Third quantization group S r 3 and the fourth quantization group S r 4 The maximum value calculation circuit 123 performs row maximum value calculation according to the quantization group in a loop, and obtains the first local maximum value m respectively. r 1 The second local maximum value m r 2 The third local maximum value m r 3 The fourth local maximum value m r 4 Then, the exponentiation circuit 125 performs exponentiation operations cyclically according to the quantization group, and the quantization circuit 126 performs quantization operations cyclically according to the quantization group to obtain the first probability matrix P′. r1 The second probability matrix P′ r2 The third probability matrix P′ r3 The fourth probability matrix P′ r4 The first probability matrix P′ r1 The second probability matrix P′ r2 The third probability matrix P′ r3 The fourth probability matrix P′ r4 The probabilities can be combined to obtain a probability matrix P' divided according to the computational units.

[0077] Understandable Figure 2 and Figure 3 The matrices shown are exemplary and not restrictive. For example, the number of elements in the matrix (each small square can represent one element) may not be limited to the number shown in the diagram, and may be more or less depending on the application scenario. Also, the computational units split according to sequence length may not be limited to multiple units. When the sequence length is short or according to the application task requirements, the sequence participating in the computation may be performed as a single computational unit, dividing the computation by quantization group.

[0078] Through the above combination Figures 1b to 3As described above, dividing the computational units in the attention mechanism into quantization groups and independently performing local maximum calculations and subsequent operations within each quantization group effectively reduces computational complexity. The division into quantization groups makes computational tasks easier to process in parallel. Furthermore, by sequentially performing row maximum calculations, exponential operations, and quantization operations on the attention matrix divided according to quantization groups, the amount of data per operation is reduced. At the software level, the quantized probability matrix can be obtained solely through the aforementioned formulas one through three, simplifying the computation process and reducing the storage requirements for intermediate results. The computation can be achieved through precise scheduling of the control circuit and a pipelined first processing circuit, fully leveraging the hardware's parallel computing capabilities and significantly improving processing speed. Simultaneously, using the reciprocal of the maximum value that the target data type can represent as the quantization coefficient for each quantization group reduces the complexity of quantization operations, enabling deep integration of group quantization and softmax operations, which further enhances the computational speed of the data processing device.

[0079] It is also understood that the first processing circuit according to the embodiments of this disclosure may not be limited to only including Figure 1b The maximum value calculation circuit 123, exponentiation circuit 125, and quantization circuit 126 shown can be supplemented with other circuits as needed. The following will combine... Figure 4 An exemplary description is provided.

[0080] Figure 4 Schematic block diagrams of a data processing apparatus according to other embodiments of this disclosure are shown. Figure 4 As shown, the data processing device 400 may include a control circuit 110 and a first processing circuit 120, wherein the first processing circuit 120 may include a matrix multiplication circuit 121, a multiplication circuit 122, a maximum value calculation circuit 123, an addition circuit 124, an exponentiation operation circuit 125, and a quantization circuit 126.

[0081] In some embodiments, the matrix multiplication circuit 121 can be configured to perform matrix multiplication operations on the query matrix Q and key matrix K partitioned by quantization group to obtain a second intermediate result. The query matrix Q and key matrix K partitioned by quantization group refer to the query matrix Q and key matrix K corresponding to the attention matrix partitioned by quantization group; that is, the query matrix Q and key matrix K involved in the operation required to compute the attention matrix for each quantization group. Figure 2 Taking the matrix to be calculated as an example, in a certain round of calculation, the data in the gray area of ​​the query matrix Q and the data of group K1 in the key matrix K are input into the matrix multiplication circuit 121. The matrix multiplication circuit 121 can perform matrix multiplication on the two to obtain a second intermediate result of a quantized group.

[0082] In other embodiments, the multiplication circuit 122 can be configured for multiplying the second intermediate result with the adjustment factor to obtain the attention matrix. Different attention mechanism tasks and datasets may have different statistical characteristics and numerical ranges. The adjustment factor can be dynamically adjusted according to the characteristics of specific tasks and data, enabling the computation process to better adapt to different scenarios, improving the flexibility and adaptability of computation, and thus enhancing the overall performance of the attention mechanism. In the hardware implementation of this embodiment, the introduction of the adjustment factor can make the computation process more compact and efficient. By combining the adjustment factor with matrix multiplication, more efficient pipelined operations can be implemented in the hardware circuit, reducing the storage and transmission overhead of intermediate results, thereby improving the utilization of hardware resources and computation speed.

[0083] In some embodiments, the adjustment factor may include an attention scaling factor and a logarithmic transformation factor. In the computation of a traditional attention mechanism, the matrix multiplication result of the query matrix Q and the key matrix K generates an attention matrix. The numerical range of this attention matrix may be large, and directly performing subsequent operations may lead to numerical instability, such as overflow problems during exponential operations. The attention scaling factor (e.g., fa_scale) in the adjustment factor can scale the attention matrix, adjusting its numerical range to a more suitable interval, thereby improving the stability of numerical computation.

[0084] Logarithmic transformation factors (e.g., log₂e) can convert operations on the attention matrix into operations in the logarithmic field. This helps reduce computational complexity and resource consumption during numerical calculations, while also improving accuracy to some extent. By using logarithmic transformation factors, certain nonlinear operations on the attention matrix in the attention mechanism can be converted into linear operations or simpler forms, thereby optimizing the computational process.

[0085] In other embodiments, the adder circuit 124 may be configured to perform subtraction on the attention matrix and local maxima by quantization group to obtain the difference between them (i.e., a first difference). In softmax operations, directly performing exponential operations on the original attention matrix can lead to numerical overflow or underflow problems, especially when the attention matrix has a large numerical range. By first subtracting the local maxima, the input values ​​for the exponential operation can be effectively limited to a relatively small range, thereby avoiding numerical overflow or underflow and improving the stability of numerical computation.

[0086] By pre-calculating the difference between the attention matrix and the local maxima using the addition circuit 124, direct input data can be provided to the subsequent exponential operation circuit 125. This optimization of the calculation process makes the entire attention mechanism calculation process more efficient, reduces unnecessary data transmission and intermediate result storage, and improves the operating efficiency of the hardware.

[0087] In some other embodiments, the data processing apparatus 400 may further include a storage circuit 410, which may be configured to: cyclically store the query matrix Q and the key matrix K in the attention mechanism per computing unit; cyclically store the key matrix K in the computing unit per quantization group; and cyclically store the probability matrix per computing unit.

[0088] For example, in some embodiments, the storage circuit 410 may be configured to cyclically store the query matrix Q and the key matrix K by computation unit, still using Figure 2 Taking the matrix shown as an example, in the first round of storage, the storage circuit 410 can store the gray areas in the query matrix Q and the gray areas in the key matrix K; in the second round of storage, the storage circuit 410 can store the gray areas in the query matrix Q and the white areas in the key matrix K to replace the data stored in the first round; in the third round of storage, the storage circuit 410 can store the white areas in the query matrix Q and the gray areas in the key matrix K to replace the data stored in the second round; in the fourth round of storage, the storage circuit 410 can store the white areas in the query matrix Q and the white areas in the key matrix K to replace the data stored in the third round. This achieves cyclical storage of the query matrix Q and the key matrix K according to the computation unit.

[0089] In some embodiments, the data processing apparatus 400 may further include a readout control unit (not shown) configured to cyclically read the key matrix K from the storage circuit 410 by quantization group. In some embodiments, the readout control unit may be connected between the storage circuit 410 and the matrix multiplication circuit 121. When the storage circuit 410 cyclically stores the query matrix Q and the key matrix K by calculation unit, the readout control unit can cyclically read the corresponding data of the key matrix K from the key matrix K stored by calculation unit in the storage circuit 410 by quantization group and transmit it to the matrix multiplication circuit 121 so that the matrix multiplication circuit 121 can cyclically perform matrix multiplication operations by quantization group.

[0090] Still with Figure 2 As shown in the example, when the storage circuit 410 stores the data of the gray area of ​​the key matrix K, the reading control unit can read the K1 group of data of the key matrix K from the storage circuit 410 and transmit the K1 group of data to the matrix multiplication circuit 121, and then continue to read other groups of data of the key matrix K stored in the storage circuit 410, and so on.

[0091] In some other embodiments, when the storage circuit 410 stores the key matrix K in the calculation unit according to the quantization group, the readout control unit can cyclically read the key matrix K from the storage circuit 410 according to the quantization group. Still using... Figure 2Taking the matrix shown as an example, in one cycle of storage, the storage circuit 410 can store K1 group data of the key matrix K; after the reading control unit reads the K1 group data, the storage circuit 410 can store the next group data of the key matrix K to replace the K1 group data, and so on, storing and reading in a loop.

[0092] In other embodiments, the readout control unit may also be configured to cyclically read the query matrix Q from the storage circuit 410 by calculation unit. When the readout control unit cyclically reads the query matrix Q by calculation unit and cyclically reads the key matrix K by quantization group, the data read each time can be used for the calculation of one quantization group in the attention matrix S, and the read query matrix Q can also continue to be used for the calculation of other quantization groups in the same calculation unit of the attention matrix S.

[0093] In some embodiments, the storage circuit 410 may also be configured to cyclically store the probability matrix per computational unit. Figure 3 Taking the matrix calculation process shown in the figure as an example, the first probability matrix P′ calculated by the quantization circuit 126 according to the quantization group r1 The second probability matrix P′ r2 The third probability matrix P′ r3 The fourth probability matrix P′ r4 They can be stored sequentially into the storage circuit 410 so that they can be merged in the storage circuit 410 to obtain the probability matrix P' divided by the calculation unit.

[0094] In other embodiments, the storage circuitry includes SRAM circuitry. SRAM (Static Random Access Memory) is characterized by fast read / write speeds but small storage space. Therefore, the storage circuitry configuration according to the embodiments of this disclosure for storing corresponding data by computing unit or quantization group can reduce the space occupied by data storage, fully utilize the characteristics of SRAM, and improve the data processing efficiency of the data processing device.

[0095] In some embodiments, the data processing apparatus may further include a high-bandwidth memory (HBM), which may be configured to store all data of the query matrix Q and key matrix K involved in the computation of the attention mechanism, and may be configured to store the data processing results output by the SRAM circuitry per computation unit.

[0096] SRAM offers significantly faster read / write speeds than HBM, but its storage space is much smaller. The control circuitry can be further configured to split the query matrix Q and key matrix K stored in HBM into computational units and then load them into SRAM unit by unit. This improves overall read / write speed by reducing read / write operations on HBM. In other embodiments, the size of the computational unit can be determined based on the storage space available in the storage circuitry. Combining HBM and SRAM improves the overall read / write speed of the data processing device, thereby increasing its data processing speed and reducing computational latency.

[0097] Furthermore, in addition to being configured to cyclically control the first processing circuit 120 to perform the aforementioned operations according to quantization groups, in some other embodiments, the control circuit 110 may also be configured to cyclically control the first processing circuit 120 to perform other operations in the attention mechanism according to computing units.

[0098] For example, in some embodiments, the first processing circuit 120 may also be configured to perform matrix multiplication of the probability matrix with the value matrix V and / or normalization operations in the softmax operation to obtain an output result. In some embodiments, the matrix multiplication circuit 121 in the first processing circuit 120 may also be configured to perform matrix multiplication of the probability matrix with the value matrix V; and / or the exponentiation circuit 125 in the first processing circuit 120 may also be configured to perform normalization operations in the softmax operation.

[0099] In some other embodiments, the first processing circuit 120 may be further configured to: cyclically update the global maximum value by computation unit, wherein the global maximum value is the maximum of the local maximum values ​​of all quantization groups in a computation unit; and update the summation term in the normalization operation based on the updated global maximum value.

[0100] by Figure 3 Taking the matrix shown as an example, assume that the local maximum values ​​of all quantization groups within the current computational unit include the first local maximum value m. r 1 The second local maximum value m r 2 The third local maximum value m r 3 The fourth local maximum value m r 4 Then the global maximum value of the current computational unit is the first local maximum value m. r 1 The second local maximum value m r 2 The third local maximum value m r 3 and the fourth local maximum m r4 The maximum value of a row.

[0101] In some embodiments, the storage circuit 410 may be further configured to cyclically store local maximum values ​​and / or global maximum values ​​by computation unit; the maximum value calculation circuit 123 may be further configured to cyclically update the global maximum value by computation unit. Specifically, the local maximum values ​​calculated by the maximum value calculation circuit 123 by quantization group can be stored in the storage circuit 410, the storage circuit 410 stores the local maximum values ​​of each quantization group by computation unit, the maximum value calculation circuit 123 performs row maximum value calculation on the local maximum values ​​of all quantization groups within a computation unit to obtain the global maximum value of a computation unit, and sends the global maximum value to the storage circuit 410. Further, the maximum value calculation circuit 123 is also configured to perform maximum value calculation on the global maximum value of the current computation unit and the historical global maximum value (such as the global maximum value of the previous round of computation unit) to obtain the updated global maximum value. For example, the maximum value calculation circuit 123 may perform the following calculation:

[0102] m2 = max(m1, m r_tile (Formula 4);

[0103] In the formula, m2 represents the updated global maximum value, m1 represents the historical global maximum value, and m r_tile This represents the global maximum value of the current calculation unit, where max indicates the calculation of the maximum value.

[0104] In other embodiments, the exponentiation circuit 125 in the first processing circuit 120 may also be configured to perform exponential operations on the second difference between the local maximum value and the updated global maximum value of each quantization group in a cyclic manner, to obtain a group correction factor for each quantization group. The group correction factor can be used to normalize the local maximum value to the global maximum value scale. In still other embodiments, the addition circuit 124 may be further configured to perform a subtraction operation on the local maximum value and the updated global maximum value to obtain the second difference value.

[0105] For example, the exponentiation circuit 125 can perform the following operations:

[0106]

[0107] In the formula, group_fix represents the group correction factor, m r m1 represents the local maximum value, and m2 represents the updated global maximum value.

[0108] In some embodiments, the exponentiation circuit 125 may also be configured to cyclically perform an exponentiation operation on the third difference between the historical global maximum value and the updated global maximum value per computation unit to obtain a difference correction factor. In other words, the historical global maximum value may be the global maximum value in a previous computation unit, and the updated global maximum value may be the global maximum value updated based on the current computation unit. This difference correction factor can be used to correct differences in computation results between multiple computation units to ensure scale consistency and accuracy across computation units. In some embodiments, the addition circuit 124 may be further configured to perform a subtraction operation on the historical global maximum value and the updated global maximum value to obtain a third difference.

[0109] For example, the exponentiation circuit 125 can perform the following operations:

[0110]

[0111] In the formula, tile_fix represents the difference correction factor, m1 represents the historical global maximum value, and m2 represents the updated global maximum value.

[0112] In some other embodiments, the adder circuit 124 may also be configured to: perform row-wise summation on each element of the probability matrix of each quantization group to obtain a local sum for each quantization group. In some embodiments, the multiplier circuit 122 may also be configured to perform multiplication on the local sum of each quantization group with the corresponding group correction factor to obtain a fourth intermediate result; the adder circuit 124 may also be configured to accumulate the fourth intermediate results of each quantization group within each computing unit to obtain a local sum for each computing unit. For example, the first processing circuit 120 may be configured to perform the following operation:

[0113] sum_reduce = reduce_sum(sum_group × group_fix) (Protocol 7);

[0114] In the formula, sum_reduce represents the local sum, reduce_sum represents the cumulative sum operation, sum_group represents the local sum, and group_fix represents the group correction factor.

[0115] Furthermore, in other embodiments, the multiplication circuit 122 may also be configured to perform a multiplication operation on the historical summation term and the difference correction factor to obtain a corrected summation term; the addition circuit 124 may also be configured to perform an addition operation on the corrected summation term and the local sum of the current calculation unit to obtain a summation term updated based on the current calculation unit. The historical summation term may be a summation term updated by a previous calculation unit. For example, the first processing circuit 120 may be configured to perform the following operations:

[0116] sum_new = sum_old × tile_fix + sum_reduce (Protocol 8);

[0117] In the formula, sum_new represents the updated summation term, sum_old represents the historical summation term, tile_fix represents the difference correction factor, and sum_reduce represents the local summation of the current calculation unit.

[0118] Thus, the first processing circuit 120 can update the summation term (or global sum) in the normalization operation based on the updated global maximum value.

[0119] Furthermore, in some embodiments, the first processing circuit 120 may also be configured to: correct the quantization coefficients of each quantization group based on the updated global maximum value to obtain the corrected quantization coefficients of each quantization group; perform matrix multiplication of the probability matrix and the value matrix V based on the corrected quantization coefficients to obtain a third intermediate result; and perform normalization operation on the third intermediate result and the updated summation term to obtain an output result.

[0120] In some implementations, the quantization circuit 126 in the first processing circuit 120 may also be configured to correct the quantization coefficients of each quantization group based on the updated global maximum value, to obtain the corrected quantization coefficients of each quantization group; the matrix multiplication circuit 121 may also be configured to perform matrix multiplication of the probability matrix and the value matrix V based on the corrected quantization coefficients, to obtain a third intermediate result; and the exponentiation circuit 125 may also be configured to perform normalization operation on the third intermediate result and the updated summation term, to obtain the output result.

[0121] For example, quantization circuit 126 can be configured to perform the following operations:

[0122]

[0123] In the formula, scale fix This indicates the correction of the quantization coefficients, round indicates the rounding operation, and m r m1 represents the local maximum value, m2 represents the updated global maximum value, and u represents the maximum value that the quantized target data type can represent.

[0124] In other embodiments, matrix multiplication circuit 121 is further configured to perform matrix multiplication of the probability matrix and value matrix V based on the corrected quantization coefficients to obtain a third intermediate result; multiplication circuit 122 can also be configured to perform multiplication of the historical third intermediate result and the difference correction factor to obtain a corrected third intermediate result; addition circuit 124 can also be configured to perform addition of the third intermediate result of the current computing unit and the corrected third intermediate result to obtain a third intermediate result updated based on the current computing unit. In the process of performing matrix multiplication of the probability matrix and value matrix V, matrix multiplication circuit 121 incorporates the corrected quantization coefficients, which can be considered, from a certain perspective, as an inverse quantization process.

[0125] For example, the first processing circuit 120 can be configured to perform the following operations:

[0126] O2 = matmul(P′, V, scale) fix )+O1×tile_fix (Formula 10);

[0127] In the formula, O2 represents the updated third intermediate result, matmul represents matrix multiplication, P′ represents the quantized probability matrix of the current computational unit, V represents the value matrix, and scale... fix O1 represents the corrected quantization coefficient, O1 represents the third intermediate result in history (such as the third intermediate result updated by the preceding calculation unit), and tile_fix represents the difference correction factor.

[0128] Furthermore, based on the calculated O2 and sum_new, the exponentiation circuit 125 can also be configured to perform a normalization operation on the updated third intermediate result and the updated summation term, that is, to perform a division operation on the updated third intermediate result and the updated summation term to obtain the output result of the softmax operation in the attention mechanism.

[0129] The above description provides an exemplary view of the data processing apparatus including a control circuit and a first processing circuit according to the embodiments disclosed herein. It is understood that the first processing circuit 120 is also configured to perform normalization operations in matrix multiplication and / or softmax operations on the probability matrix and the value matrix V. This achieves reuse of the first processing circuit 120, allowing all operations in the attention mechanism to be implemented using only the first processing circuit 120. Furthermore, the pipeline of the first processing circuit only needs to be controlled in the hierarchical order of quantization group → computation unit, simplifying instruction distribution logic. Intermediate result data such as the global maximum value and summation terms can be directly stored in the internal registers or local SRAM (storage circuit 410) of the first processing circuit, without needing to be written to the external HBM, significantly reducing dependence on HBM. In addition, by independently executing all operations through the first processing circuit, the overall circuit architecture of the data processing apparatus can be simplified, resulting in a more compact and efficient hardware implementation and reduced power consumption. This is particularly suitable for scenarios sensitive to area, power consumption, and latency, meeting the energy efficiency requirements of edge computing devices (such as mobile phones and IoT terminals).

[0130] It is also understood that the above description is exemplary and not limiting; for example, the data processing apparatus may not be limited to including only control circuitry and the first processing circuitry, but may also include a second processing circuitry. The following will be combined with... Figure 5 An illustrative example is provided.

[0131] Figure 5 A schematic block diagram of a data processing apparatus according to further embodiments of this disclosure is shown. Figure 5 As shown, the data processing device 500 may include a control circuit 110, a first processing circuit 120, a second processing circuit 510, and a storage circuit 410.

[0132] In some embodiments, the control circuit 110 may also be configured to control the second processing circuit 510 to perform attention mechanism operations cyclically according to the computing unit; the second processing circuit 510 is configured to perform normalization operations in matrix multiplication and / or softmax operations on the probability matrix and the value matrix V to obtain the output result.

[0133] In some other embodiments, the second processing circuit 510 may be further configured to: cyclically update the global maximum value by computation unit, wherein the global maximum value is the maximum of the local maximum values ​​of all quantization groups in a computation unit; and update the summation term in the normalization operation based on the updated global maximum value.

[0134] In some embodiments, the second processing circuit 510 may also be configured to: correct the quantization coefficients of each quantization group based on the updated global maximum value to obtain the corrected quantization coefficients of each quantization group; perform matrix multiplication of the probability matrix and the value matrix V based on the corrected quantization coefficients to obtain a third intermediate result; and perform normalization operation on the third intermediate result and the summation term to obtain an output result.

[0135] The specific operation process performed by the second processing circuit 510 has been described in the preceding text. Figure 4 The implementation of the first processing circuit 120 shown has been described in detail and will not be repeated here. It is understood that the second processing circuit 510 may include one or more of the following: matrix multiplication circuit, multiplication circuit, accumulation circuit, maximum value calculation circuit, addition circuit, exponentiation circuit, division circuit, quantization circuit, etc., to implement the operation process of the aforementioned embodiment.

[0136] In some implementations, the control circuit can cyclically schedule the first processing circuit to perform a portion of the attention mechanism operations according to the quantization group, or cyclically schedule the first or second processing circuit to perform another portion of the attention mechanism operations according to the computing unit, thereby achieving dynamic merging of the global maximum value and the summation term, realizing hardware-level fusion of Softmax and MX quantization.

[0137] When the first processing circuit 120 and the second processing circuit 510 are combined to implement the attention mechanism operation, different granularities of division of labor and cooperation can be realized. For example, the first processing circuit 120 is responsible for the local calculation of fine-grained quantization groups, and the second processing circuit is responsible for the global integration of coarse-grained calculation units. The hardware design is simple and efficient.

[0138] Especially for long sequences of data to be computed, the first processing circuit achieves fine-grained computation with low latency and low storage overhead through deep hardware fusion, which is suitable for optimizing efficiency within a single quantization group. The second processing circuit supports block processing of long sequences through division of labor and decoupling, ensuring scale uniformity and global accuracy of results across groups and improving the overall system throughput. The two circuits work together through control circuits to form a pipeline architecture of "local computation + global correction", maximizing the computational efficiency and numerical stability of the attention mechanism under limited hardware resources.

[0139] Exemplary data processing method

[0140] This disclosure also provides a data processing method, which will be described below in conjunction with... Figure 6 Please provide an explanation.

[0141] Figure 6 An exemplary flowchart illustrating data processing methods according to some embodiments of this disclosure is shown. Figure 6As shown, the data processing method 600 may include: cyclically performing the following operations in the attention mechanism task according to quantization groups, wherein a quantization group is a group divided into computational units according to sequence length in the attention mechanism: in step S602, performing row maximum calculation on the attention matrix divided by quantization groups in the attention mechanism task to obtain the local maximum value of each quantization group; in step S604, performing exponential operation on the difference between the attention matrix and the local maximum value to obtain a first intermediate result; and in step S606, performing quantization operation on the first intermediate result based on the quantization coefficient to obtain a probability matrix for each quantization group, wherein the quantization coefficient is the reciprocal of the maximum value that the quantized target data type can represent, and the probability matrix is ​​used for softmax operation in the attention mechanism task.

[0142] In some embodiments, the data processing method 600 further includes: performing matrix multiplication on the query matrix Q and the key matrix K divided by quantization groups to obtain a second intermediate result; and performing multiplication on the second intermediate result and the adjustment factor to obtain an attention matrix.

[0143] In other embodiments, the data processing method 600 further includes performing the following operations in the attention mechanism task cyclically per computational unit: performing matrix multiplication and / or normalization operations in the softmax operation on the probability matrix and the value matrix V to obtain the output result of the attention mechanism task.

[0144] In some other embodiments, the data processing method 600 may further include: cyclically updating the global maximum value by computing unit, wherein the global maximum value is the maximum value among the local maximum values ​​of all quantization groups in a computing unit; and updating the summation term in the normalization operation based on the updated global maximum value.

[0145] In some embodiments, the data processing method 600 may further include: correcting the quantization coefficients of each quantization group based on the updated global maximum value to obtain the corrected quantization coefficients of each quantization group; performing matrix multiplication of the probability matrix and the value matrix V based on the corrected quantization coefficients to obtain a third intermediate result; and performing a normalization operation on the third intermediate result and the summation term to obtain an output result.

[0146] In other embodiments, the adjustment factors include attention scaling factors and logarithmic transformation factors.

[0147] The data processing method of this disclosure embodiment has been described in detail above in conjunction with the data processing device, and will not be repeated here.

[0148] Exemplary hardware architecture

[0149] Figure 7 A schematic diagram of the structure of a circuit board according to an embodiment of this disclosure is shown. Figure 7 As shown, board 700 includes one or more chips 701 (i.e., one implementation of the integrated circuit device described herein), which is a system-on-a-chip (SoC) integrating one or more combined processing devices. The combined processing device can be an artificial intelligence computing unit to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence, and a significant characteristic of cloud intelligence applications is the large amount of input data, placing high demands on the platform's storage and computing capabilities. Board 700 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0150] Chip 701 is connected to external device 703 via external interface device 702. External device 703 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 703 to chip 701 via external interface device 702. The calculation results from chip 701 can be transmitted back to external device 703 via external interface device 702. Depending on the application scenario, external interface device 702 may have different interface forms, such as PCIe (Peripheral Component Interconnect express) interface.

[0151] The board 700 also includes a storage device 704 for storing data, which includes one or more memory cells 705. The storage device 704 is connected to and transmits data with the controller 706 and the chip 701 via a bus. The controller 706 in the board 700 is configured to regulate the state of the chip 701. Therefore, in one application scenario, the controller 706 may include a microcontroller, also known as a microcontroller unit (MCU).

[0152] Figure 8 It is shown Figure 7 A structural diagram of the combined processing device in chip 701 of the illustrated embodiment. (As shown) Figure 8 As shown, the combined processing device 800 includes a computing device 801, an interface device 802, a processing device 803, and a storage device 804.

[0153] The computing device 801 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 803 through the interface device 802 to jointly complete the user-specified operations.

[0154] Interface device 802 is used to transmit data and control commands between computing device 801 and processing device 803. For example, computing device 801 can obtain input data from processing device 803 via interface device 802 and write it to on-chip storage device of computing device 801. Further, computing device 801 can obtain control commands from processing device 803 via interface device 802 and write them to on-chip control cache of computing device 801. Alternatively or optionally, interface device 802 can also read data from storage device of computing device 801 and transmit it to processing device 803.

[0155] The processing device 803, as a general-purpose processing device, performs basic controls including but not limited to data transfer and starting / stopping the computing device 801. Depending on the implementation, the processing device 803 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, for the sake of the computing device 801 disclosed herein, it can be considered to have a single-core structure or a homogeneous multi-core structure. However, when the computing device 801 and the processing device 803 are considered together, they are considered to form a heterogeneous multi-core structure.

[0156] The storage device 804 is used to store data to be processed. It may be DRAM (Dynamic Random Access Memory), DDR (Double Data Rate) memory, typically 16G or larger, and is used to store data of computing device 801 and / or processing device 803.

[0157] This disclosure also provides a computing device, including: at least one processor; and at least one memory in communication with the at least one processor, having stored computer-readable instructions thereon, which, when loaded and executed by the at least one processor, cause the at least one processor to perform the aforementioned data processing method.

[0158] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various circuits in the device embodiments described above, this document has decomposed them based on their logical functions, but in actual implementation, there may be other decomposition methods. The physical implementation of the hardware structure of the circuit may include, but is not limited to, physical devices, which may include, but are not limited to, devices such as transistors or memristors. In view of this, the various devices described herein (e.g., computing devices or other processing devices) can be implemented by appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage unit or storage device can be any suitable storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM and RAM, etc.

[0159] In this disclosed embodiment, the term "instruction" can include software instructions, hardware instructions, firmware instructions, or any combination thereof. Software instructions typically refer to statements or commands in a programming language; they are high-level abstractions, such as function calls, machine code, and bytecode. Hardware instructions, also known as machine instructions or instruction set architecture (ISA) instructions, are low-level commands that computer hardware can directly recognize and execute, such as processor instructions and CPU instructions. Firmware instructions include, for example, opcodes and microcode. Furthermore, different CPU architectures and different instruction sets can have different hardware instruction sets. Based on the complexity of the computer instruction set, hardware design, execution speed, compiler complexity, and instruction format, they can be further classified as: CISC (Complex Instruction Set Computer), RISC (Reduced Instruction Set Computer), and VLIW (Very Long Instruction Word).

[0160] The foregoing can be better understood in accordance with the following terms:

[0161] Clause A1. A data processing apparatus, comprising a control circuit and a first processing circuit, wherein:

[0162] The control circuit is configured to control the first processing circuit to cyclically execute at least a portion of the operations in the attention mechanism according to quantization groups, wherein the quantization group is a group divided into computational units split according to sequence length in the attention mechanism;

[0163] The first processing circuit includes:

[0164] The maximum value calculation circuit is configured to perform row maximum value calculation of the attention matrix divided by quantization group in the attention mechanism, and obtain the local maximum value of each quantization group;

[0165] An exponential operation circuit is configured to perform an exponential operation on a first difference between the attention matrix and the local maximum value to obtain a first intermediate result; and

[0166] A quantization circuit is configured to perform a quantization operation on the first intermediate result based on quantization coefficients to obtain a probability matrix for each quantization group, wherein the quantization coefficients are the reciprocals of the maximum value that the quantized target data type can represent, and the probability matrices are used for the softmax operation in the attention mechanism.

[0167] Clause A2. The data processing apparatus according to Clause A1, wherein the first processing circuit further comprises:

[0168] A matrix multiplication circuit is configured to perform matrix multiplication operations on the query matrix Q and the key matrix K, which are divided by quantization groups, to obtain a second intermediate result;

[0169] The multiplication circuit is configured to perform a multiplication operation between the second intermediate result and the adjustment factor to obtain the attention matrix.

[0170] Clause A3. The data processing apparatus according to Clause A1, wherein the first processing circuit further comprises:

[0171] An adder circuit is configured to perform a subtraction operation on the attention matrix and the local maximum value to obtain the first difference.

[0172] Clause A4. The data processing apparatus according to any one of Clauses A1-A3 further includes storage circuitry configured for at least one of the following:

[0173] The query matrix Q and key matrix K in the attention mechanism are stored cyclically by computational unit;

[0174] The key matrix K in the computational unit is stored according to quantization groups;

[0175] The probability matrix is ​​stored cyclically according to the calculation unit.

[0176] Clause A5. The data processing apparatus according to Clause A4 further includes:

[0177] A reading control unit is configured to cyclically read the key matrix K from the storage circuit according to quantization groups.

[0178] Clause A6. The data processing apparatus according to any one of clauses A1-A5, wherein,

[0179] The first processing circuit is further configured to perform normalization operations on the probability matrix and the value matrix V in matrix multiplication and / or softmax operations to obtain an output result; or

[0180] The data processing device further includes a second processing circuit, wherein the control circuit is further configured to control the second processing circuit to cyclically execute attention mechanism operations according to the calculation unit; the second processing circuit is configured to perform matrix multiplication and / or normalization operations in the softmax operation on the probability matrix and the value matrix V to obtain the output result.

[0181] Clause A7. The data processing apparatus according to Clause A6, wherein the first processing circuit or the second processing circuit is further configured to:

[0182] The global maximum value is updated cyclically according to the computing unit, where the global maximum value is the maximum value among the local maximum values ​​of all quantization groups in a computing unit.

[0183] Based on the updated global maximum value, the summation term in the normalization operation is updated.

[0184] Clause A8. The data processing apparatus according to Clause A7, wherein the first processing circuit or the second processing circuit is further configured to:

[0185] Based on the updated global maximum value, the quantization coefficients of each quantization group are corrected to obtain the corrected quantization coefficients of each quantization group;

[0186] Based on the corrected quantization coefficients, a matrix multiplication operation is performed between the probability matrix and the value matrix V to obtain a third intermediate result; and

[0187] The normalization operation is performed on the third intermediate result and the summation term to obtain the output result.

[0188] Clause A9. The data processing apparatus according to Clause A4, wherein the storage circuitry includes SRAM circuitry.

[0189] Clause A10. The data processing apparatus according to Clause A2, wherein the adjustment factor includes an attention scaling factor and a logarithmic transformation factor.

[0190] Clause A11. A data processing method comprising cyclically performing the following operations in an attention mechanism task, per quantization group, wherein the quantization group is a group within computational units split by sequence length in the attention mechanism:

[0191] The row maximum value is calculated for the attention matrix divided into quantization groups in the attention mechanism task to obtain the local maximum value of each quantization group;

[0192] Perform an exponential operation on the first difference between the attention matrix and the local maximum to obtain a first intermediate result; and

[0193] Based on the quantization coefficients, a quantization operation is performed on the first intermediate result to obtain a probability matrix for each quantization group, wherein the quantization coefficients are the reciprocals of the maximum value that the quantized target data type can represent, and the probability matrices are used for the softmax operation in the attention mechanism task.

[0194] Clause A12. The data processing method described in Clause A11 further includes:

[0195] Perform matrix multiplication on the query matrix Q and key matrix K, which are divided by quantization groups, to obtain the second intermediate result;

[0196] The attention matrix is ​​obtained by multiplying the second intermediate result with the adjustment factor.

[0197] Clause A13. The data processing method according to Clause A11 or A12 further includes performing the following operations in the attention mechanism task cyclically per computational unit:

[0198] The normalization operation in the matrix multiplication and / or softmax operation of the probability matrix V is performed to obtain the output result of the attention mechanism task.

[0199] Clause A14. The data processing method described in Clause A13 further includes:

[0200] The global maximum value is updated cyclically according to the computing unit, where the global maximum value is the maximum value among the local maximum values ​​of all quantization groups in a computing unit.

[0201] Based on the updated global maximum value, the summation term in the normalization operation is updated.

[0202] Clause A15. The data processing method described in Clause A14 further includes:

[0203] Based on the updated global maximum value, the quantization coefficients of each quantization group are corrected to obtain the corrected quantization coefficients of each quantization group;

[0204] Based on the corrected quantization coefficients, a matrix multiplication operation is performed between the probability matrix and the value matrix V to obtain a third intermediate result; and

[0205] The normalization operation is performed on the third intermediate result and the summation term to obtain the output result.

[0206] Clause A16. The data processing method according to Clause A12, wherein the adjustment factors include an attention scaling factor and a logarithmic transformation factor.

[0207] Clause A17. An integrated circuit device comprising a data processing apparatus according to any one of Clauses A1-A10.

[0208] Clause A18. A board including an integrated circuit device as described in Clause 17.

[0209] Clause A19. A computing device comprising:

[0210] At least one processor; and

[0211] At least one memory in communication with the at least one processor, having stored computer-readable instructions that, when loaded and executed by the at least one processor, cause the at least one processor to perform any of the data processing methods described in clauses A11-A16.

[0212] Clause 20. A computer-readable storage medium storing program instructions that, when loaded and executed by a processor, cause the processor to perform a data processing method according to any one of Clauses A11-A16.

[0213] While numerous embodiments of this disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and intent of this disclosure. It should be understood that various alternatives to the embodiments of this disclosure described herein may be employed in the practice of this disclosure. The appended claims are intended to define the scope of this disclosure and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A data processing apparatus, comprising a control circuit and a first processing circuit, wherein: The control circuit is configured to control the first processing circuit to cyclically execute at least a portion of the operations in the attention mechanism according to quantization groups, wherein the quantization group is a group divided into computational units split according to sequence length in the attention mechanism; The first processing circuit includes: The maximum value calculation circuit is configured to perform row maximum value calculation of the attention matrix divided by quantization group in the attention mechanism, and obtain the local maximum value of each quantization group; An exponential operation circuit is configured to perform an exponential operation on a first difference between the attention matrix and the local maximum value to obtain a first intermediate result; and A quantization circuit is configured to perform a quantization operation on the first intermediate result based on quantization coefficients to obtain a probability matrix for each quantization group, wherein the quantization coefficients are the reciprocals of the maximum value that the quantized target data type can represent, and the probability matrices are used for the softmax operation in the attention mechanism.

2. The data processing apparatus according to claim 1, wherein the first processing circuit further comprises: A matrix multiplication circuit is configured to perform matrix multiplication operations on the query matrix Q and the key matrix K, which are divided by quantization groups, to obtain a second intermediate result; A multiplication circuit is configured to perform a multiplication operation between the second intermediate result and the adjustment factor to obtain the attention matrix.

3. The data processing apparatus according to claim 1, wherein the first processing circuit further comprises: An adder circuit is configured to perform a subtraction operation on the attention matrix and the local maximum value to obtain the first difference.

4. The data processing apparatus according to any one of claims 1-3, further comprising a storage circuit configured for at least one of the following: The query matrix Q and key matrix K in the attention mechanism are stored cyclically by computational unit; The key matrix K in the computational unit is stored according to quantization groups; The probability matrix is ​​stored cyclically according to the calculation unit.

5. The data processing apparatus according to claim 4, further comprising: A reading control unit is configured to cyclically read the key matrix K from the storage circuit according to quantization groups.

6. The data processing apparatus according to any one of claims 1-5, wherein, The first processing circuit is further configured to perform normalization operations on the probability matrix and the value matrix V in matrix multiplication and / or softmax operations to obtain an output result; or The data processing device further includes a second processing circuit, wherein the control circuit is further configured to control the second processing circuit to cyclically execute attention mechanism operations according to the calculation unit; the second processing circuit is configured to perform matrix multiplication and / or normalization operations in the softmax operation on the probability matrix and the value matrix V to obtain the output result.

7. The data processing apparatus according to claim 6, wherein the first processing circuit or the second processing circuit is further configured to: The global maximum value is updated cyclically according to the computing unit, where the global maximum value is the maximum value among the local maximum values ​​of all quantization groups in a computing unit. Based on the updated global maximum value, the summation term in the normalization operation is updated.

8. The data processing apparatus according to claim 7, wherein the first processing circuit or the second processing circuit is further configured to: Based on the updated global maximum value, the quantization coefficients of each quantization group are corrected to obtain the corrected quantization coefficients of each quantization group; Based on the corrected quantization coefficients, a matrix multiplication operation is performed between the probability matrix and the value matrix V to obtain a third intermediate result; as well as The normalization operation is performed on the third intermediate result and the summation term to obtain the output result.

9. The data processing apparatus according to claim 4, wherein the storage circuit includes an SRAM circuit.

10. The data processing apparatus according to claim 2, wherein the adjustment factor includes an attention scaling factor and a logarithmic transformation factor.

11. A data processing method comprising cyclically performing the following operations in an attention mechanism task by quantization group, wherein the quantization group is a group divided into computational units in the attention mechanism by sequence length: The row maximum value is calculated for the attention matrix divided into quantization groups in the attention mechanism task to obtain the local maximum value of each quantization group; Perform an exponential operation on the first difference between the attention matrix and the local maximum value to obtain a first intermediate result; as well as Based on the quantization coefficients, a quantization operation is performed on the first intermediate result to obtain a probability matrix for each quantization group, wherein the quantization coefficients are the reciprocals of the maximum value that the quantized target data type can represent, and the probability matrices are used for the softmax operation in the attention mechanism task.

12. The data processing method according to claim 11, further comprising: Perform matrix multiplication on the query matrix Q and key matrix K, which are divided by quantization groups, to obtain the second intermediate result; The attention matrix is ​​obtained by multiplying the second intermediate result with the adjustment factor.

13. The data processing method according to claim 11 or 12, further comprising cyclically performing the following operations in the attention mechanism task per computing unit: The normalization operation in the matrix multiplication and / or softmax operation of the probability matrix V is performed to obtain the output result of the attention mechanism task.

14. The data processing method according to claim 13, further comprising: The global maximum value is updated cyclically according to the computing unit, where the global maximum value is the maximum value among the local maximum values ​​of all quantization groups in a computing unit. Based on the updated global maximum value, the summation term in the normalization operation is updated.

15. The data processing method according to claim 14, further comprising: Based on the updated global maximum value, the quantization coefficients of each quantization group are corrected to obtain the corrected quantization coefficients of each quantization group; Based on the corrected quantization coefficients, a matrix multiplication operation is performed between the probability matrix and the value matrix V to obtain a third intermediate result; as well as The normalization operation is performed on the third intermediate result and the summation term to obtain the output result.

16. The data processing method according to claim 12, wherein the adjustment factor includes an attention scaling factor and a logarithmic transformation factor.

17. An integrated circuit device comprising a data processing apparatus according to any one of claims 1-10.

18. A board comprising the integrated circuit device according to claim 17.

19. A computing device, comprising: At least one processor; as well as At least one memory in communication with the at least one processor, having stored computer-readable instructions that, when loaded and executed by the at least one processor, cause the at least one processor to perform the data processing method according to any one of claims 11-16.

20. A computer-readable storage medium storing program instructions that, when loaded and executed by a processor, cause the processor to perform the data processing method according to any one of claims 11-16.

Images