Method and related product for data processing in multi-head self-attention mechanism
By performing on-chip calculations and transposition followed by direct off-chip writing of split data, the method addresses performance issues in multi-head self-attention mechanisms, enhancing network efficiency.
Patent Information
- Application Number
- CN202310778373.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-28
- Publication Date
- 2024-12-31
AI Technical Summary
Existing methods for implementing multi-head self-attention mechanisms in Transformer networks, such as using transpose operations or packing modes, lead to increased IO accesses and redundant data, resulting in decreased network performance.
Perform numerical calculations and transposition operations on-chip to split data, and directly write the split data to off-chip for subsequent self-attention operations, avoiding repeated IO accesses and redundant data.
This approach enhances network performance by reducing IO memory access and eliminating redundant data, thereby improving the efficiency of multi-head self-attention mechanisms.
Smart Images
Figure CN119227736A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to the field of artificial intelligence technology. More specifically, the present disclosure relates to a method, system, device and computer-readable storage medium for data processing in a multi-head self-attention mechanism. Further, the present disclosure also relates to a computing device and a board. Background Art
[0002] The multi-head self-attention ("Multi-head Attention") layer is an important component of the Transformer network and the BERT network, which can perform self-attention operations. Before performing the self-attention operation ("self-attention") in this layer, it is first necessary to generate three intermediate tensors, namely query, key, and value, based on the data of this layer. This process can be called an attention project. Next, the aforementioned three tensors (i.e., query, key, and value) are further split into multiple tensors in preparation for self-attention. At present, in the network implementation of mainstream frameworks (such as PyTorch and TensorFlow), reconstruction ("reshape") and transposition ("transpose") are usually used to implement the aforementioned splitting operations.
[0003] One way to implement the split operation using transpose is to write the intermediate tensor out of the chip, load it from the off-chip to the on-chip, perform the transpose operation on the chip, and finally write the transpose result out of the chip (for example Figure 1 As shown in Figure 1.1.1.1.2, this method requires writing out and reading in an additional intermediate result, which adds two additional IO accesses to read / write. For multi-head self-attention layers with multiple cycles, the aforementioned multiple additional access operations will lead to a decrease in the overall performance of the network. Another method is to use the packing ("pack") mode. In this mode, you first need to restore the pack mode to the padding ("pad") mode, and then use transpose to split (for example Figure 2 This method increases invalid data, resulting in a large amount of redundant IO.
[0004] In view of this, there is an urgent need to provide a solution for data processing in the multi-head self-attention mechanism in order to improve the overall performance of the network adopting the multi-head self-attention mechanism. Summary of the invention
[0005] In order to at least solve one or more of the technical problems mentioned above, the present disclosure proposes a solution for data processing in a multi-head self-attention mechanism in multiple aspects.
[0006] In a first aspect, the present disclosure provides a method for data processing in a multi-head self-attention mechanism, comprising: reading data to be processed in the multi-head self-attention mechanism; performing numerical calculation operations and transposition operations on the data to be processed on-chip to split the data to be processed; and writing the split data from the chip to outside the chip for subsequent use in performing self-attention operations in the multi-head self-attention mechanism.
[0007] In a second aspect, the present disclosure provides a system for data processing in a multi-head self-attention mechanism, comprising: a reading unit, which is used to read the data to be processed in the multi-head self-attention mechanism; an on-chip unit, which is used to perform numerical calculation operations and transposition operations on the data to be processed on the chip to split the data to be processed; and a writing unit, which writes the split data from the chip to outside the chip so that it can be subsequently used to perform self-attention operations in the multi-head self-attention mechanism.
[0008] In a third aspect, the present disclosure provides a device for data processing in a multi-head self-attention mechanism, comprising: a processor; and a memory storing program instructions for data processing in a multi-head self-attention mechanism, wherein when the program instructions are executed by the processor, the device implements an embodiment of the aforementioned first aspect.
[0009] In a fourth aspect, the present disclosure provides a computing device, comprising the apparatus described in the third aspect.
[0010] In a fifth aspect, the present disclosure provides a board comprising the computing device described in the fourth aspect.
[0011] In a sixth aspect, the present disclosure provides a computer-readable storage medium having stored thereon computer-readable instructions for data processing in a multi-head self-attention mechanism, which, when executed by one or more processors, implement the embodiments of the aforementioned first aspect.
[0012] Through the scheme for data processing in the multi-head self-attention mechanism provided above, the disclosed embodiment realizes multi-head splitting by performing numerical calculation operations and transposition operations on the data to be processed on the chip, and writes the split data from the chip to the outside of the chip for subsequent use in self-attention operations. That is, the disclosed embodiment integrates the transposition operation into the attention project and directly writes out the split data. Based on this, the IO memory access caused by repeated read / write can be effectively avoided. Furthermore, through the disclosed embodiment, there is no need to restore the pack mode to the pad mode, which avoids a large amount of redundant IO caused by the pad mode adding a large amount of invalid data, thereby greatly improving the overall performance of the network using the multi-head self-attention mechanism. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:
[0014] Figure 1 is an exemplary schematic diagram showing a method of implementing a split operation using transpose according to the solution of the present disclosure;
[0015] Figure 2 is an exemplary schematic diagram showing another way of implementing a split operation using transpose according to the solution of the present disclosure;
[0016] Figure 3 is an exemplary flowchart showing a method for data processing in a multi-head self-attention mechanism according to an embodiment of the present disclosure;
[0017] Figure 4 is an exemplary schematic diagram showing the overall data processing for a multi-head self-attention mechanism according to an embodiment of the present disclosure;
[0018] Figure 5 is an exemplary structural block diagram showing a system for data processing in a multi-head self-attention mechanism according to an embodiment of the present disclosure;
[0019] Figure 6 is a schematic diagram showing the structure of a board according to an embodiment of the present disclosure; and
[0020] Figure 7 is a schematic diagram showing the structure of a combined processing device in a chip according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0021] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings. It should be understood that the embodiments described in this specification are only some of the embodiments provided by the present disclosure to facilitate a clear understanding of the solutions and to comply with legal requirements, and are not all embodiments that can implement the present disclosure. Based on the embodiments disclosed in this specification, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.
[0022] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.
[0023] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.
[0024] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.
[0025] As described in the background above, one way to use transpose to implement the split operation is to write the intermediate tensor out of the chip, then load it from the chip to the chip, perform the transpose operation on the chip, and finally write the transpose result out of the chip. Another way is to use pack mode. In this mode, you first need to restore the pack mode to pad mode, and then use transpose to split. Figure 1 and Figure 2 Describe in detail the two ways of using transpose to implement the split operation mentioned above.
[0026] Figure 1is an exemplary diagram showing a method of implementing a split operation using transpose. Figure 1 As shown in , first read (or read in) the data in the multi-head self-attention layer, wherein the data in the multi-head self-attention layer includes, for example, input data ("input"), filter data ("qkv_filters"), and bias data ("qkv_bias"). Then, based on the aforementioned input, qkv_filters, and qkv_bias, a numerical operation is performed on the chip, wherein the numerical operation can be, for example, a multiplication ("matmul") operation, and the intermediate results (or intermediate tensors) query, key, and value are obtained after the matmul operation. In an implementation scenario, the aforementioned on-chip is a system-on-chip or an on-chip unit, and the system-on-chip can be, for example, an image processor (Image Processing Unit, "IPU"). Further, by writing the intermediate results query, key, and value out of the chip. In an implementation scenario, the off-chip can be, for example, a double data rate (Double Data Rate, "DDR") memory.
[0027] The figure further shows that when performing the transpose operation, the above intermediate results query, key and value need to be loaded from off-chip to (or "read to") the chip, and the transpose operation is performed on the chip to split the intermediate results query, key and value into multiple tensors query multi-head, key multi-head and value multi-head and write them out to the chip. In this way, multi-head splitting in multi-head self-attention can be achieved to prepare for subsequent self-attention operations.
[0028] By the above Figure 1 It can be seen that when using transpose to implement the split operation in this way, the intermediate results query, key, and value need to be written out of the chip (for example, as shown by arrow A in the figure) and then read from the chip to the chip (for example, as shown by arrow B in the figure), which adds two additional IO memory accesses for reading / writing. For a network with multiple self-attention layers with multiple loops, multiple additional IO memory accesses will be caused, thereby reducing the overall performance of the network.
[0029] Figure 2 FIG. 1 is an exemplary schematic diagram showing another way of implementing the split operation using transpose. Figure 2The left side of the figure shows the pack mode, which is to arrange data of different lengths closely together and record the actual length of each data. As an example, the figure uses rectangular boxes of different lengths to represent data of different lengths. For example, rectangular box L1 in the figure represents short data, and rectangular box L2 represents long data. According to the previous description, in the pack mode, you first need to restore the pack mode to the pad mode, for example Figure 2 The middle shows the pad mode. It can be understood that in pad mode, all data will be extended to a fixed length, and shorter data will have special padding markers added to the end (such as the dotted box in the figure) to make it the same length as the longest data. The processed data can form a regular tensor for subsequent calculations. Next, use transpose to split it and obtain multiple tensors query multi-head, key multi-head, and value multi-head, thereby realizing multi-head splitting in multi-head self-attention, in preparation for subsequent self-attention operations.
[0030] By the above Figure 2 It can be seen that when using transpose to implement the split operation in this way, the pack mode needs to be restored to the pad mode, which will introduce a large number of padding marks (which do not carry valid information, that is, invalid data). The aforementioned padding marks will also occupy computing resources during the calculation process, resulting in a large amount of redundant IO and affecting network performance.
[0031] Based on this, the disclosed embodiment provides a solution for data processing in a multi-head self-attention mechanism, which integrates transpose into the attention project and directly outputs the split data to effectively avoid IO memory access caused by repeated reading / writing and avoids a large amount of redundant IO caused by the addition of a large amount of invalid data due to the pad mode, thereby greatly improving the overall performance of the network.
[0032] The specific implementation of the present disclosure is described in detail below with reference to the accompanying drawings.
[0033] Figure 3 3 is an exemplary flow chart showing a method 300 for data processing in a multi-head self-attention mechanism according to an embodiment of the present disclosure. Figure 3 As shown in , at step S301, the data to be processed in the multi-head self-attention mechanism is read. In one embodiment, the aforementioned data to be processed may include input data and parameter data in the multi-head self-attention mechanism, and the input data and parameter data are both valid data. As an example, the input data may be the above Figure 1 and Figure 2As shown in the input, the parameter data may include filter data and bias data, such as the above Figure 1 and Figure 2 The data input, qkv_filters and qkv_bias to be processed are all valid data (or real data), that is, the disclosed embodiment does not need to fill the input, qkv_filters and qkv_bias (that is, restore to pad mode), but directly performs it in pack mode.
[0034] Based on the data to be processed in the read multi-head self-attention mechanism, at step S302, numerical operation and transposition operation are performed on the data to be processed on the chip to split the data to be processed. In one embodiment, the aforementioned numerical operation includes at least a product operation (for example, denoted as matmul). Specifically, a product operation is first performed on the data to be processed on the chip to obtain an intermediate result. Then, a transposition operation is performed on the intermediate result on the chip to split the data to be processed. As mentioned above, the on-chip is a system on chip or an on-chip unit, and the system on chip can be, for example, an IPU. In one embodiment, the aforementioned intermediate results may include query results, key results, and value results, such as the above Figure 1 and Figure 2 That is, the disclosed embodiment first performs a product operation on the input to be processed, qkv_filters and qkv_bias on the chip to obtain the intermediate results query, key and value, and the aforementioned process is also called attention project.
[0035] Furthermore, in the disclosed embodiment, there is no need to write the intermediate results query, key and value obtained by performing the matmul operation on the chip out of the chip, but directly perform the transpose operation on the intermediate results query, key and value on the chip to split the intermediate results query, key and value into the above Figure 1 and Figure 2 The multiple tensors query multi-head, key multi-head, and value multi-head shown in the figure are prepared for the subsequent self-attention operation.
[0036] In an implementation scenario, before performing numerical calculation operations and transposition operations on the data to be processed on the chip, the data to be processed may be reduced in dimension to the lowest dimension, so as to perform numerical calculation operations and transposition operations on the data to be processed in the lowest dimension on the chip. For example, if the length of the data to be processed is 1024, it may be reduced in dimension to form data with a dimension size of 16*64, so as to perform numerical calculation operations and transposition operations on the data to be processed in the lowest dimension 16, so as to obtain multiple tensors query multi-head, key multi-head and value multi-head after splitting.
[0037] Further, at step S303, the split data is written from the chip to the off-chip, so as to be used for the subsequent self-attention operation in the multi-head self-attention mechanism. That is, the data written to the off-chip in the disclosed embodiment is the split data, not the intermediate result. As can be seen from the foregoing, the off-chip can be, for example, a DDR memory.
[0038] Combined with the above description, it can be seen that the disclosed embodiment integrates transpose into the attention project to perform numerical calculations and transposition operations on the data to be processed on the chip, realize multi-head splitting, and directly write the split data to the outside of the chip, which can effectively avoid IO memory access caused by repeated read / write. Furthermore, the data to be processed in the disclosed embodiment are all valid data, that is, the disclosed embodiment does not need to restore the pack mode to the pad mode, avoiding a large amount of redundant IO due to the increase of a large amount of invalid data in the pad mode, thereby greatly improving the overall performance of the network using the multi-head self-attention mechanism.
[0039] Figure 4 is an exemplary schematic diagram showing the overall data processing for a multi-head self-attention mechanism according to an embodiment of the present disclosure. It should be understood that Figure 4 is the above Figure 3 A specific embodiment of Figure 3 The description also applies to Figure 4 .
[0040] like Figure 4As shown in , first read (or "read in") the data in the multi-head self-attention layer, wherein the data in the multi-head self-attention layer includes input, qkv_filters, and qkv_bias. Based on the aforementioned input, qkv_filters, and qkv_bias. Then, numerical operations (such as matmul operations) can be performed on the chip, and intermediate results (or intermediate tensors) query, key, and value are obtained after the matmul operation. In an implementation scenario, the aforementioned on-chip is a system-on-chip or an on-chip unit, and the system-on-chip can be, for example, an IPU. Then, based on obtaining the intermediate results query, key, and value, a transpose operation is performed on the chip to split the intermediate results query, key, and value into multiple tensors querymulti-head, keymulti-head, and valuemulti-head, thereby realizing multi-head splitting in multi-head self-attention, so as to prepare for subsequent self-attention operations. Further, the split multiple tensors query multi-head, key multi-head, and value multi-head are directly written to the outside of the chip. In one implementation scenario, the off-chip may be, for example, a DDR memory.
[0041] By the above Figure 4 It can be seen that the embodiment of the present disclosure reduces the above Figure 1 The IO access caused by writing the intermediate results query, key and value to the outside of the chip (for example, as shown by arrow A in the figure) and then reading them from the outside to the chip (for example, as shown by arrow B in the figure) also reduces the above Figure 2 The large amount of redundant IO caused by restoring the pack mode to the pad mode greatly improves the overall performance of the network using the multi-head self-attention mechanism.
[0042] Figure 5 is an exemplary structural block diagram showing a system 500 for data processing in a multi-head self-attention mechanism according to an embodiment of the present disclosure. Figure 5 As shown in , the system 500 may include a reading unit 501, an on-chip unit 502, and a writing unit 503. In one embodiment, the reading unit 501 is used to read the data to be processed in the multi-head self-attention mechanism. The data to be processed includes input data and parameter data in the multi-head self-attention mechanism, such as input, filter data qkv_filters, and bias data qkv_bias, and input, qkv_filters, and qkv_bias are all valid data.
[0043] In one embodiment, the aforementioned on-chip unit 502 is used to perform numerical calculation operations and transposition operations on the data to be processed on the chip to split the data to be processed. Among them, the aforementioned numerical calculation operations at least include product operations (for example, recorded as matmul). In an implementation scenario, the on-chip unit 502 may include an on-chip calculation unit and an on-chip transposition unit, and the on-chip calculation unit is used to perform a product operation on the data to be processed on the chip to obtain an intermediate result, and the on-chip transposition unit is used to perform a transposition operation on the intermediate result on the chip to split the data to be processed. As mentioned above, the on-chip is a system on chip or an on-chip unit, and the system on chip can be, for example, an IPU. Specifically, the disclosed embodiment first performs a product operation on the input, qkv_filters and qkv_bias to be processed by the on-chip calculation unit to obtain the intermediate results query, key and value. Next, the intermediate results query, key, and value are transposed through the on-chip transposition unit to split the intermediate results query, key, and value into multiple tensors querymulti-head, keymulti-head, and valuemulti-head to prepare for subsequent self-attention operations.
[0044] In one embodiment, the aforementioned write-out unit 503 is used to write the split data from the chip to the off-chip, so as to be used to perform the self-attention operation in the multi-head self-attention mechanism later. That is, the disclosed embodiment directly writes the split data to the off-chip. As can be seen from the foregoing, the off-chip can be, for example, a DDR memory.
[0045] In an implementation scenario, the system 500 of the disclosed embodiment may further include a dimensionality reduction unit, which is used to reduce the dimension of the data to be processed to the lowest dimension, so as to perform numerical calculation operations and transposition operations on the data to be processed in the lowest dimension on the chip. For example, the 1024 data is reduced to data with a dimension size of 16*64, so as to perform numerical calculation operations and transposition operations on the data to be processed in the lowest dimension 16, and obtain multiple tensors after splitting.
[0046] Figure 6 FIG. 6 is a schematic diagram showing the structure of a board 600 according to an embodiment of the present disclosure. Figure 6As shown, the board 600 includes a chip (or "processing chip") 601, which is a system-on-chip (SoC), or a system on chip or an on-chip unit in the disclosed embodiment, and is integrated with one or more combined processing devices, which are an artificial intelligence computing unit to support various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in the fields of computer vision, speech, natural language processing, data mining, etc. In particular, deep learning technology is widely used in the field of cloud intelligence. A notable feature of cloud intelligence applications is the large amount of input data, which has high requirements for the storage capacity and computing power of the platform. The board 600 of this embodiment is suitable for cloud intelligence applications, and has huge off-chip storage, on-chip storage and a large amount of computing power.
[0047] The chip 601 is connected to an external device 603 via an external interface device 602. The external device 603 is, for example, a server, a computer, a camera, a display, a mouse, a keyboard, a network card, or a WIFI interface. The data to be processed can be transmitted from the external device 603 to the chip 601 via the external interface device 602. The calculation result of the chip 601 can be transmitted back to the external device 603 via the external interface device 602. According to different application scenarios, the external interface device 602 can have different interface forms, such as a PCIe interface.
[0048] The board 600 also includes a storage device 604 for storing data, which includes one or more storage units 605. The storage device 604 is connected to the control device 607 and the chip 601 through a bus and transmits data. The control device 606 in the board 600 is configured to regulate the state of the chip 601. To this end, in an application scenario, the control device 606 may include a single-chip microcomputer (Micro Controller Unit, MCU). In the application scenario of the data processing scheme in the multi-head self-attention mechanism disclosed in the present invention, a driver can be run in the control device and it includes the device of the embodiment of the present disclosure. When the aforementioned driver is controlled and run by the control device, the device executes the aforementioned combined Figure 3 The method flow is described, so that the tasks in each task flow are sent to the processing chip for execution.
[0049] Figure 7 FIG. 1 is a schematic diagram showing the structure of the combined processing device 700 in the chip 601 according to an embodiment of the present disclosure. Figure 7 As shown in , the combined processing device 700 includes a computing device 701 , an interface device 702 , a processing device 703 and a DRAM 704 .
[0050] The computing device 701 is configured to perform user-specified operations, and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 703 through the interface device 702 to jointly complete the user-specified operations.
[0051] The interface device 702 is used to transmit data and control instructions between the computing device 701 and the processing device 703. For example, the computing device 701 can obtain input data from the processing device 703 via the interface device 702 and write it into the storage device on the computing device 701 chip. Further, the computing device 701 can obtain control instructions from the processing device 703 via the interface device 702 and write them into the control cache on the computing device 701 chip. Alternatively or optionally, the interface device 702 can also read data in the storage device of the computing device 701 and transmit it to the processing device 703.
[0052] The processing device 703, as a general processing device, performs basic controls including but not limited to data handling, starting and / or stopping the computing device 701. Depending on the implementation, the processing device 703 can be a central processing unit (CPU), a graphics processing unit (GPU), an IPU in the embodiment of the present disclosure, or other general and / or special processors, one or more types of processors, these processors include but are not limited to digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, only with respect to the computing device 701 disclosed in the present disclosure, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 701 and the processing device 703 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.
[0053] DRAM 704 is used to store data to be processed, and is a DDR memory, which is usually 16G or larger in size, and is used to store data of the computing device 701 and / or the processing device 703. In one or more implementation scenarios, the memory management solution disclosed herein can be applied to the management and maintenance of the DDR, thereby realizing the reuse or recycling operation of events. In this case, the board disclosed herein can be regarded as the device side of the artificial intelligence computing system.
[0054] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teaching of the present disclosure, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present disclosure may be regarded as optional embodiments, i.e., the actions or modules involved therein may not necessarily be necessary for the implementation of one or some of the schemes of the present disclosure. In addition, depending on the different schemes, the present disclosure also has different focuses on the description of some embodiments. In view of this, those skilled in the art may appreciate the parts that are not described in detail in a certain embodiment of the present disclosure, and may also refer to the relevant descriptions of other embodiments.
[0055] In terms of specific implementation, based on the disclosure and teaching of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented by other methods not disclosed herein. For example, with respect to the various modules in the device or apparatus embodiments described above, this article divides them on the basis of considering logical functions, and there may be other division methods in actual implementation. For another example, multiple modules or components can be combined or integrated into another system, or some features or functions in the modules or components can be selectively disabled. With respect to the connection relationship between different modules or components, the connection discussed in the above text in conjunction with the accompanying drawings can be a direct or indirect coupling between modules or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.
[0056] In the present disclosure, the modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules. The aforementioned components or modules may be located in the same location or distributed on multiple systems. In addition, according to actual needs, some or all of the modules may be selected to achieve the purpose of the scheme described in the embodiments of the present disclosure. In addition, in some scenarios, multiple modules in the embodiments of the present disclosure may be integrated into one module or each module may exist physically separately.
[0057] In some implementation scenarios, the above modules can be implemented in the form of software program modules. If implemented in the form of software program modules and sold or used as independent products, the modules can be stored in a computer-readable memory. Based on this, when the scheme of the present disclosure is embodied in the form of a software product (such as a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to enable a computer device (such as a personal computer, a server or a network device, etc.) to perform some or all of the steps of the method described in the embodiment of the present disclosure. The aforementioned memory may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a flash drive, a read-only memory ("Read Only Memory", abbreviated as ROM), a random access memory ("Random Access Memory", abbreviated as RAM), a mobile hard disk, a magnetic disk or an optical disk.
[0058] In some other implementation scenarios, the above modules can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as CPU, GPU, IPU, FPGA, DSP and ASIC. Furthermore, the aforementioned storage unit or storage device may be any appropriate storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), which may be, for example, resistive random access memory ("Resistive Random Access Memory", abbreviated as RRAM), dynamic random access memory ("Dynamic Random Access Memory", abbreviated as DRAM), static random access memory ("Static Random Access Memory", abbreviated as SRAM), enhanced dynamic random access memory ("Enhanced Dynamic Random Access Memory", abbreviated as "EDRAM"), high bandwidth memory ("High Bandwidth Memory", abbreviated as "HBM"), hybrid memory cube ("Hybrid Memory Cube", abbreviated as "HMC"), ROM and RAM, etc.
[0059] The foregoing content can be better understood in accordance with the following terms:
[0060] Item A1. A method for data processing in a multi-head self-attention mechanism, comprising:
[0061] Read the data to be processed in the multi-head self-attention mechanism;
[0062] Performing numerical calculation operations and transposition operations on the data to be processed on-chip to split the data to be processed; and
[0063] The split data is written from the chip to outside the chip so as to be used for subsequent self-attention operations in the multi-head self-attention mechanism.
[0064] Clause A2. The method according to clause A1, further comprising:
[0065] The data to be processed are subjected to a dimensionality reduction operation to the lowest dimension, so as to perform numerical calculation operations and transposition operations on the data to be processed in the lowest dimension on the chip.
[0066] Clause A3. The method according to clause A1, wherein the numerical operation comprises at least a product operation, and performing the numerical operation and the transposition operation on the data to be processed on-chip to split the data to be processed comprises:
[0067] Performing the multiplication operation on the data to be processed on the chip to obtain an intermediate result; and
[0068] A transposition operation is performed on the slice on the intermediate result to split the data to be processed.
[0069] Clause A4. A method according to clause A3, wherein the intermediate results include query results, key results and value results.
[0070] Clause A5. A method according to any one of clauses A1-A4, wherein the data to be processed includes input data and parameter data in a multi-head self-attention mechanism, and the input data and parameter data are both valid data.
[0071] Item A6. A system for data processing in a multi-head self-attention mechanism, comprising:
[0072] A reading unit, which is used to read the data to be processed in the multi-head self-attention mechanism;
[0073] an on-chip unit, which is used to perform numerical calculation operations and transposition operations on the data to be processed on the chip to split the data to be processed; and
[0074] The write-out unit writes the split data from the slice to the outside of the slice for subsequent use in performing the self-attention operation in the multi-head self-attention mechanism.
[0075] Item A7. The system according to Item A6 also includes a dimensionality reduction unit, and the dimensionality reduction unit is used to reduce the dimensionality of the data to be processed to the lowest dimension so as to perform numerical calculation operations and transposition operations on the data to be processed in the lowest dimension on the chip.
[0076] Clause A8. A system according to clause A6, wherein the on-chip unit includes an on-chip arithmetic unit and an on-chip transposition unit, and the numerical operation includes at least a product operation, and the numerical operation and the transposition operation are performed on the data to be processed on the chip to split the data to be processed,
[0077] The on-chip computing unit is used to perform the product operation on the data to be processed on the chip to obtain an intermediate result; and
[0078] The on-chip transposition unit is used to perform a transposition operation on the intermediate result on the chip to split the data to be processed.
[0079] Clause A9. A system according to clause A8, wherein the intermediate results include query results, key results, and value results.
[0080] Item A10. A system according to any one of items A7-A9, wherein the data to be processed includes input data and parameter data in a multi-head self-attention mechanism, and the input data and parameter data are both valid data.
[0081] Item A11. A device for data processing in a multi-head self-attention mechanism, comprising:
[0082] Processor; and
[0083] A memory storing program instructions for data processing in a multi-head self-attention mechanism, which, when executed by the processor, enables the device to implement the method described in any one of clauses A1-A5.
[0084] Clause A12. A computing device comprising the apparatus according to clause A11.
[0085] Item A13. A board comprising a computing device according to Item A12.
[0086] Item A14. A computer-readable storage medium having stored thereon computer-readable instructions for data processing in a multi-head self-attention mechanism, which, when executed by one or more processors, implements a method as described in any one of Items A1-A5.
[0087] Although multiple embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art may think of many changes, modifications, and alternatives without departing from the thought and spirit of the present disclosure. It should be understood that in the process of practicing the present disclosure, various alternatives to the embodiments of the present disclosure described herein may be adopted. The attached claims are intended to define the scope of protection of the present disclosure, and therefore cover equivalents or alternatives within the scope of these claims.
Claims
1. A method for data processing in a multi-head self-attention mechanism, comprising: Read the data to be processed in the multi-head self-attention mechanism; Performing numerical calculation operations and transposition operations on the data to be processed on the chip to split the data to be processed; as well as The split data is written from the chip to outside the chip so as to be used for subsequent self-attention operations in the multi-head self-attention mechanism.
2. The method according to claim 1, further comprising: The data to be processed are subjected to a dimensionality reduction operation to the lowest dimension, so as to perform numerical calculation operations and transposition operations on the data to be processed in the lowest dimension on the chip.
3. The method according to claim 1, wherein the numerical operation comprises at least a product operation, and performing the numerical operation and the transposition operation on the data to be processed on-chip to split the data to be processed comprises: Performing the product operation on the data to be processed on the chip to obtain an intermediate result; as well as A transposition operation is performed on the slice on the intermediate result to split the data to be processed. The method according to claim 3 , wherein the intermediate results include query results, key results and value results.
5. A method according to any one of claims 1-4, wherein the data to be processed includes input data and parameter data in a multi-head self-attention mechanism, and the input data and parameter data are both valid data.
6. A system for data processing in a multi-head self-attention mechanism, comprising: A reading unit, which is used to read the data to be processed in the multi-head self-attention mechanism; An on-chip unit, which is used to perform numerical calculation operations and transposition operations on the data to be processed on the chip to split the data to be processed; as well as The write-out unit writes the split data from the slice to the outside of the slice for subsequent use in performing the self-attention operation in the multi-head self-attention mechanism.
7. The system according to claim 6 further includes a dimensionality reduction unit, and the dimensionality reduction unit is used to reduce the dimensionality of the data to be processed to the lowest dimension, so as to perform numerical calculation operations and transposition operations on the data to be processed in the lowest dimension on the chip.
8. The system according to claim 6, wherein the on-chip unit comprises an on-chip computing unit and an on-chip transposition unit, and the numerical operation comprises at least a product operation, and the numerical operation and the transposition operation are performed on the chip on the data to be processed to split the data to be processed. The on-chip computing unit is used to perform the product operation on the data to be processed on the chip to obtain an intermediate result; and The on-chip transposition unit is used to perform a transposition operation on the intermediate result on the chip to split the data to be processed.
9. The system of claim 8, wherein the intermediate results include query results, key results, and value results.
10. A system according to any one of claims 7-9, wherein the data to be processed includes input data and parameter data in a multi-head self-attention mechanism, and the input data and parameter data are both valid data.
11. A device for data processing in a multi-head self-attention mechanism, comprising: processor; as well as A memory storing program instructions for data processing in a multi-head self-attention mechanism, wherein when the program instructions are executed by the processor, the device implements the method according to any one of claims 1 to 5.
12. A computing device comprising the apparatus according to claim 11.
13. A board comprising the computing device according to claim 12.
14. A computer-readable storage medium storing computer-readable instructions for data processing in a multi-head self-attention mechanism, wherein when the computer-readable instructions are executed by one or more processors, the method described in any one of claims 1-5 is implemented.