Data processing method, method, device, equipment and medium for training attention calculation

Abstract

The present disclosure provides a data processing method, a method for training attention calculation, an apparatus, a device, and a medium, which relate to the field of chip technology, and in particular to the field of artificial intelligence technology. The implementation scheme is as follows: loading a first data block set into a second memory, wherein the first data block set includes a first query data block, a first key data block, and a first value data block; dividing the first query data block and the first key data block into a plurality of query sub-data blocks and a plurality of key sub-data blocks, respectively; for each query sub-data block, generating each score sub-data block corresponding to the query sub-data block based on the query sub-data block and each lower sub-data block corresponding to the query sub-data block; for one or more diagonal sub-data blocks in each score sub-data block, generating a mask sub-data block corresponding to the diagonal sub-data block; and generating an output data block corresponding to the first data block set based on the first score data block and the first value data block.

Description

Technical Field

[0001] The present disclosure relates to the field of chip technology, in particular to the fields of artificial intelligence and image processing, and specifically to a data processing method, a method for training attention calculation, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Artificial intelligence (AI) is the study of how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily encompass computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graphs.

[0003] Currently, attention computing is widely used in artificial intelligence fields such as natural language processing and image processing. However, as the amount of data required for calculation increases, the computational complexity and space complexity also increase, resulting in a corresponding increase in the amount of computation and memory resources required.

[0004] The approaches described in this section are not necessarily approaches that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any approach described in this section is prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, the issues raised in this section should not be considered as having been recognized in any prior art. Summary of the Invention

[0005] The present disclosure provides a data processing method, a method for training attention calculation, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of the present disclosure, a data processing method is provided for performing attention calculation on query data, key data and value data stored in a first memory, wherein the query data includes a plurality of query data blocks, the key data includes a plurality of key data blocks, and the value data includes a plurality of value data blocks. The method comprises: loading a first data block set into a second memory, wherein the first data block set includes a first query data block, a first key data block and a first value data block, and an access speed of the second memory is greater than an access speed of the first memory; dividing the first query data block and the first key data block into a plurality of query sub-data blocks and a plurality of key sub-data blocks, respectively, wherein the plurality of query sub-data blocks and the plurality of key sub-data blocks have the same size; for each query sub-data block, generating respective sub-data blocks corresponding to the query sub-data block based on the query sub-data block and the respective lower sub-data blocks corresponding to the query sub-data block fractional sub-data blocks, wherein the lower sub-data block corresponding to the query sub-data block is a key sub-data block in which the index of data in the multiple key sub-data blocks is less than or equal to the index of the data in the query sub-data block; for one or more diagonal sub-data blocks in each fractional sub-data block, masking is performed on the data in each diagonal sub-data block whose column index is greater than the row index to generate a masked sub-data block corresponding to the diagonal sub-data block, wherein the row index of the diagonal sub-data block in each fractional sub-data block is equal to the column index of the diagonal sub-data block in each fractional sub-data block; and based on the first fractional data block and the first value data block, generating an output data block corresponding to the first data block set, wherein the first fractional data block includes the fractional sub-data blocks in each fractional sub-data block except for the one or more diagonal sub-data blocks and the masked sub-data blocks corresponding to the one or more diagonal sub-data blocks.

[0007] According to another aspect of the present disclosure, a method for training attention calculation is provided, wherein query sample data, key sample data, value sample data and output sample data are stored in a first memory, the query sample data includes a plurality of query sample data blocks, the key sample data includes a plurality of key sample data blocks, the value sample data includes a plurality of value sample data blocks, and the output sample data includes a plurality of output sample data blocks. The method comprises: loading a first data block set into a second memory, wherein the first data block set includes a first query sample data block, a first key sample data block, a first value sample data block and a first output sample data block, and an access speed of the second memory is greater than an access speed of the first memory; dividing the first query sample data block and the first key sample data block into a plurality of query sub-data blocks and a plurality of key sub-data blocks, respectively, wherein the plurality of query sub-data blocks and the plurality of key sub-data blocks have the same size; for each query sub-data block, based on the query sub-data block and the corresponding For each lower sub-data block of the query sub-data block, generate each fractional sub-data block corresponding to the query sub-data block, wherein the lower sub-data block corresponding to the query sub-data block is a key sub-data block whose index of data in multiple key sub-data blocks is less than or equal to the index of data in the query sub-data block; for one or more diagonal sub-data blocks in each fractional sub-data block, mask the data in each diagonal sub-data block whose column index is greater than the row index to generate a masked sub-data block corresponding to the diagonal sub-data block, wherein the row index of the diagonal sub-data block in each fractional sub-data block is equal to the column index of the diagonal sub-data block in each fractional sub-data block; and based on the first fractional sample data block and the first output sample data block, update the parameters of the attention calculation, wherein the first fractional sample data block includes the fractional sub-data blocks other than one or more diagonal sub-data blocks in each fractional sub-data block and the masked sub-data blocks corresponding to the one or more diagonal sub-data blocks.

[0008] According to another aspect of the present disclosure, a data processing device is provided for performing attention calculation on query data, key data and value data stored in a first memory, wherein the query data includes a plurality of query data blocks, the key data includes a plurality of key data blocks, and the value data includes a plurality of value data blocks. The device includes: a data loading unit configured to load a first data block set into a second memory, wherein the first data block set includes a first query data block, a first key data block and a first value data block, and an access speed of the second memory is greater than an access speed of the first memory; a data block partitioning unit configured to partition the first query data block and the first key data block into a plurality of query sub-data blocks and a plurality of key sub-data blocks, respectively, wherein the plurality of query sub-data blocks and the plurality of key sub-data blocks have the same size; a score determination unit configured to generate, for each query sub-data block, a score corresponding to the query sub-data block based on the query sub-data block and the respective lower sub-data blocks corresponding to the query sub-data block. Each fractional sub-data block of the query sub-data block, wherein the lower sub-data block corresponding to the query sub-data block is a key sub-data block whose index of data in multiple key sub-data blocks is less than or equal to the index of data in the query sub-data block; a masking unit is configured to: for one or more diagonal sub-data blocks in each fractional sub-data block, mask the data in each diagonal sub-data block whose column index is greater than the row index to generate a masked sub-data block corresponding to the diagonal sub-data block, wherein the row index of the diagonal sub-data block in each fractional sub-data block is equal to the column index of the diagonal sub-data block in each fractional sub-data block; and an output unit is configured to: generate an output data block corresponding to the first data block set based on the first fractional data block and the first value data block, wherein the first fractional data block includes the fractional sub-data blocks in each fractional sub-data block except for one or more diagonal sub-data blocks and the masked sub-data blocks corresponding to the one or more diagonal sub-data blocks.

[0009] According to another aspect of the present disclosure, a device for training attention calculation is provided, wherein query sample data, key sample data, value sample data and output sample data are stored in a first memory, the query sample data includes a plurality of query sample data blocks, the key sample data includes a plurality of key sample data blocks, the value sample data includes a plurality of value sample data blocks, and the output sample data includes a plurality of output sample data blocks. The device comprises: a data loading unit configured to: load a first data block set into a second memory, wherein the first data block set includes a first query sample data block, a first key sample data block, a first value sample data block and a first output sample data block, and the access speed of the second memory is greater than the access speed of the first memory; a data block partitioning unit configured to: partition the first query sample data block and the first key sample data block into a plurality of query sub-data blocks and a plurality of key sub-data blocks, respectively. a data block, wherein the plurality of query sub-data blocks and the plurality of key sub-data blocks have the same size; a score determining unit configured to: for each query sub-data block, generate each fractional sub-data block corresponding to the query sub-data block based on the query sub-data block and each lower sub-data block corresponding to the query sub-data block, wherein the lower sub-data block corresponding to the query sub-data block is a key sub-data block in which the index of data in the plurality of key sub-data blocks is less than or equal to the index of data in the query sub-data block; a masking unit configured to: for one or more diagonal sub-data blocks in each fractional sub-data block, mask data in each diagonal sub-data block whose column index is greater than the row index to generate a masked sub-data block corresponding to the diagonal sub-data block, wherein the row index of the diagonal sub-data block in each fractional sub-data block is equal to the column index of the diagonal sub-data block in each fractional sub-data block; and

[0010] A parameter updating unit is configured to update parameters of attention calculation based on a first fractional sample data block and a first output sample data block, wherein the first fractional sample data block includes fractional sub-data blocks other than one or more diagonal sub-data blocks in each fractional sub-data block and mask sub-data blocks corresponding to the one or more diagonal sub-data blocks.

[0011] According to another aspect of the present disclosure, a chip is provided, comprising at least one of the data processing device and the device for training attention calculation as described above.

[0012] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0013] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above method.

[0014] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program can implement the above method when executed by a processor.

[0015] According to one or more embodiments of the present disclosure, the amount of attention calculation or the amount of calculation used to train attention calculation can be reduced.

[0016] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings illustrate exemplary embodiments and constitute a part of the specification. Together with the description of the specification, they serve to explain exemplary implementation of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals designate similar, but not necessarily identical, elements.

[0018] Figure 1 A schematic diagram illustrating an exemplary system in which the various methods described herein may be implemented according to exemplary embodiments of the present disclosure;

[0019] Figure 2 A schematic diagram illustrating a causal mask in attention calculation according to an exemplary embodiment of the present disclosure is shown;

[0020] Figures 3A-3B A schematic diagram illustrating attention calculation performed at a coarse granularity according to an exemplary embodiment of the present disclosure is shown;

[0021] Figure 4 A flow chart showing a data processing method according to an exemplary embodiment of the present disclosure is shown;

[0022] Figure 5 A schematic diagram of a causal masking process with refined computation granularity according to an exemplary embodiment of the present disclosure is shown;

[0023] Figure 6 A schematic diagram illustrating a mask data block for masking a diagonal sub-data block according to an exemplary embodiment of the present disclosure is shown;

[0024] Figure 7 The exemplary embodiment of the present disclosure is shown Figure 4A schematic diagram of a process of generating an output data block corresponding to a first data block set based on each fractional sub-data block and a first value data block;

[0025] Figure 8 A schematic diagram of a random deactivation process with refined computational granularity according to an exemplary embodiment of the present disclosure is shown;

[0026] Figure 9 A flowchart of a method for training attention calculation according to an exemplary embodiment of the present disclosure is shown;

[0027] Figure 10 shows a structural block diagram of a data processing device according to an exemplary embodiment of the present disclosure;

[0028] Figure 11 A structural block diagram of an apparatus for training attention calculation according to an exemplary embodiment of the present disclosure is shown; and

[0029] Figure 12 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0030] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0031] In this disclosure, unless otherwise specified, the use of terms such as "first" and "second" to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements. Such terms are only used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of the element, while in some cases, based on the context of the description, they may also refer to different instances.

[0032] The terms used in the descriptions of the various examples described in this disclosure are for the purpose of describing specific examples only and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element may be one or more. In addition, the term "and / or" used in this disclosure encompasses any one and all possible combinations of the listed items.

[0033] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0034] Figure 1FIG2 is a schematic diagram of an exemplary system 100 in which the various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 can be configured to execute one or more applications.

[0035] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable execution of the data processing method and / or the method of training attention calculations.

[0036] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0037] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may, in turn, utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0038] The user may use client devices 101, 102, 103, 104, 105 and / or 106 to send at least one of query data, key data, value data, query sample data, key sample data, value sample data and output sample data. The client device may provide an interface that enables the user of the client device to interact with the client device. The client device may also output information to the user via the interface. Although Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure can support any number of client devices.

[0039] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux, or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. The client device is capable of executing various different applications, such as various Internet-related applications, communication applications (eg, email applications), Short Message Service (SMS) applications, and may use various communication protocols.

[0040] The network 110 may be any type of network known to those skilled in the art that can support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0041] Server 120 may include one or more general-purpose computers, specialized server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0042] The computing units in the server 120 may run one or more operating systems including any of the operating systems described above as well as any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, and the like.

[0043] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display the data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0044] In some embodiments, server 120 may be a distributed system server or a server integrated with blockchain. Server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host equipped with artificial intelligence technology. A cloud server is a host product within the cloud computing service system that addresses the management difficulties and poor scalability of traditional physical hosts and virtual private servers (VPS) services.

[0045] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in a variety of locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0046] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0047] Figure 1The system 100 may be configured and operated in various ways to enable application of the various methods and apparatuses described in accordance with the present disclosure.

[0048] In related attention calculation technologies, by dividing the data to be calculated (for example, query data, key data, and value data) into blocks, the problems of high computational complexity and high memory consumption when the data volume is large (for example, when the sequence is long) are alleviated. However, it is still expected to further reduce the technical complexity and the memory consumed.

[0049] According to some embodiments, in attention calculations, a causal mask is needed to process sequence data to prevent future data from being seen when predicting current data, thereby ensuring that predictions are made based only on past and current information. Therefore, it is desirable to make full use of the causal mask to avoid unnecessary calculations and reduce the memory used.

[0050] According to some embodiments, attention calculation includes the following process:

[0051] 1) Calculate the attention score: Based on the query data and the key data (e.g., multiplying the query data by the key data) to determine the attention score data;

[0052] 2) Normalized attention weights: To ensure that the sum of attention weights is 1, the softmax function is usually used to normalize the score data;

[0053] 3) Weighted summation: Use the normalized attention weights to perform weighted summation on the value data to obtain the output data of the attention calculation.

[0054] According to some embodiments, in attention calculation, in order to ensure that the model only pays attention to the information before the current position when processing sequence data, it is necessary to mask the data in the attention score data whose column index is greater than the row index (for example, set this part of the data to a negative maximum value).

[0055] Figure 2 FIG. 1 shows a schematic diagram of a causal mask in attention calculation according to an exemplary embodiment of the present disclosure. Figure 2 As shown, in the causal mask, the upper fraction data 232 above the diagonal line in the attention score data 230 is masked. According to some embodiments, the calculation of generating the upper fraction data 232 and subsequent calculations associated with the upper fraction data 232 can be omitted to reduce computational complexity.

[0056] Figures 3A-3B A schematic diagram illustrating attention calculation performed at a coarse granularity according to an exemplary embodiment of the present disclosure is shown.

[0057] Figure 3AThe process of calculating the attention score is shown in Figure 2. Figure 3A As shown, when calculating the attention score data 330, a query data block 311 from the query data 310 and a key data block 321 from the key data 320 are taken each time, and based on the query data block 311 and the key data block 321 (for example, multiplying the two together), a corresponding score data block 331 in the attention score data 330 is generated.

[0058] According to some embodiments, the length of query data 310 and key data 320 is L s , the width of the query data 310 and the key data 320 is H s , and the length and width of the attention score data 330 are both L s (For example, in the case where the query data 310 and the key data 320 are multiplied). According to some embodiments, the width H of the query data 310 and the key data 320 s The dimension of each individual attention in the attention calculation.

[0059] According to some embodiments, the length of the query data block 311 is B r , the length of query data block 311 is B c , the width of the query data block 311 and the key data block 321 is the width H of the query data 310 and the key data 320 s , the length of the corresponding score data block 331 is B r , the width of the corresponding fractional data block 331 is B c .

[0060] According to some embodiments, limited by various factors (e.g., the size of the memory inside the data processing unit (e.g., Static Random-Access Memory (SRAM))), the size of the query data block 311 and the key data block 321 is typically larger than the computational granularity of the parallel computing unit (e.g., a single Compute Unified Device Architecture (CUDA) core) that performs the attention computation.

[0061] Figure 3B A schematic diagram of causal masking performed at a coarse granularity according to an exemplary embodiment of the present disclosure is shown.

[0062] like Figure 3BAs shown, there are cross-diagonal score data blocks 331-334 in the attention score data 330, wherein each cross-diagonal score data block has a portion that does not require a causal mask (i.e., the portion below the diagonal) and a portion that requires a causal mask (i.e., the portion above the diagonal). Therefore, it is impossible to directly skip the calculation of generating these cross-diagonal score data blocks 331-334 and the subsequent calculations associated with these cross-diagonal score data blocks 331-334 (e.g., normalization), resulting in the amount of calculation saved by applying the causal mask not being able to reach the ideal situation (e.g., as shown in the reference Figure 2 as much as described).

[0063] According to some embodiments, in order to omit subsequent calculations associated with these cross-diagonal fractional data blocks 331-334, it is necessary to generate a mask matrix to mask the portions of these cross-diagonal fractional data blocks 331-334 that are above the diagonal. However, due to the relatively large size of the cross-diagonal fractional data blocks 331-334, the associated mask calculations are computationally intensive.

[0064] At least in order to solve the above-mentioned problems of large amount of unnecessary calculations and large amount of calculations for related mask calculations caused by coarse calculation granularity, the present disclosure provides a data processing method for performing attention calculations on query data, key data and value data stored in a first memory, wherein the query data includes multiple query data blocks, the key data includes multiple key data blocks, and the value data includes multiple value data blocks. The method comprises: loading a first data block set into a second memory, wherein the first data block set includes a first query data block, a first key data block and a first value data block, and an access speed of the second memory is greater than an access speed of the first memory; dividing the first query data block and the first key data block into multiple query sub-data blocks and multiple key sub-data blocks, respectively, wherein the multiple query sub-data blocks and the multiple key sub-data blocks have the same size; for each query sub-data block, based on the query sub-data block and the respective next The method comprises the steps of: firstly, generating a first fractional sub-data block and generating fractional sub-data blocks corresponding to the query sub-data block, wherein the lower sub-data block corresponding to the query sub-data block is a key sub-data block in which the index of data in the plurality of key sub-data blocks is less than or equal to the index of data in the query sub-data block; for one or more diagonal sub-data blocks in each fractional sub-data block, masking the data in each diagonal sub-data block whose column index is greater than the row index to generate a masked sub-data block corresponding to the diagonal sub-data block, wherein the row index of the diagonal sub-data block in each fractional sub-data block is equal to the column index of the diagonal sub-data block in each fractional sub-data block; and generating an output data block corresponding to a first data block set based on the first fractional data block and the first value data block, wherein the first fractional data block includes the fractional sub-data blocks in each fractional sub-data block except the one or more diagonal sub-data blocks and the masked sub-data blocks corresponding to the one or more diagonal sub-data blocks.

[0065] Figure 4 A flow chart of a data processing method 400 according to an exemplary embodiment of the present disclosure is shown. The method 400 is used to perform attention calculation on query data, key data, and value data stored in a first memory. The query data includes multiple query data blocks, the key data includes multiple key data blocks, and the value data includes multiple value data blocks. Figure 4 As shown, the method 400 includes:

[0066] Step S401: loading a first data block set into a second memory, wherein the first data block set includes a first query data block, a first key data block, and a first value data block, and an access speed of the second memory is greater than an access speed of the first memory;

[0067] Step S402: Divide the first query data block and the first key data block into a plurality of query sub-data blocks and a plurality of key sub-data blocks respectively, wherein the plurality of query sub-data blocks and the plurality of key sub-data blocks have the same size;

[0068] Step S403: For each query sub-data block, generate fractional sub-data blocks corresponding to the query sub-data block based on the query sub-data block and the lower sub-data blocks corresponding to the query sub-data block, wherein the lower sub-data blocks corresponding to the query sub-data block are key sub-data blocks whose indexes of data in the key sub-data blocks are less than or equal to the index of data in the query sub-data block;

[0069] Step S404: for one or more diagonal sub-data blocks in each fractional sub-data block, mask the data in each diagonal sub-data block whose column index is greater than the row index to generate a masked sub-data block corresponding to the diagonal sub-data block, wherein the row index of the diagonal sub-data block in each fractional sub-data block is equal to the column index of the diagonal sub-data block in each fractional sub-data block; and

[0070] Step S405: Based on the first fractional data block and the first value data block, generate an output data block corresponding to the first data block set, wherein the first fractional data block includes the fractional sub-data blocks in each fractional sub-data block except for one or more diagonal sub-data blocks and the mask sub-data blocks corresponding to the one or more diagonal sub-data blocks.

[0071] Therefore, the first query data block and the first key data block loaded into the second memory for attention calculation are further divided into multiple query sub-data blocks and multiple key sub-data blocks, which reduces the size of the data blocks for actual calculation and refines the calculation granularity, so that more unnecessary calculations related to causal masks are omitted; moreover, the size of the query sub-data blocks and key sub-data blocks used for actual calculation is reduced, and the size of the generated fractional sub-data blocks is relatively reduced, thereby reducing the amount of calculation for masking the fractional sub-data blocks on the diagonal.

[0072] According to some embodiments, before performing the attention calculation, the query data, key data, and value data used for the attention calculation are stored in a first memory with a slower access speed (e.g., High Bandwidth Memory (HBM)). According to some embodiments, when performing the attention calculation, a set of data blocks including a query data block, a key data block, and a value data block are loaded into a second memory with a faster access speed (e.g., SRAM) each time to implement block calculation of the attention calculation, thereby reducing the requirements for the storage capacity of the second memory and allowing parallel calculation in multiple computing units to improve computing efficiency. According to some embodiments, the first memory has a larger storage capacity than the second memory to store all query data, key data, and value data.

[0073] According to some embodiments, query data blocks, key data blocks, and value data blocks can be loaded one by one according to an inner loop and an outer loop in the attention calculation to determine one query data block, one key data block, and one value data block in a set of data blocks loaded into the second memory each time.

[0074] For example, the key data block and the value data block can be set as an outer loop, and the query data block can be set as an inner loop, wherein, in the outer loop, a pair of key data blocks and value data blocks with the same index are loaded into the second memory, and different feature parts of the input data are processed sequentially, and, in the inner loop, multiple query data blocks of the query data are traversed and loaded into the second memory one by one to process different positions of the input sequence sequentially.

[0075] According to some embodiments, for key data blocks, value data blocks and query data blocks, the index refers to the sequence number of the data block in multiple data blocks of the corresponding data. For example, if a key data block is the 5th key data block among multiple key data blocks of key data, then the index of the key data block is 5.

[0076] According to some embodiments, the sizes of the multiple query sub-data blocks and the multiple key sub-data blocks include information about the sizes of various dimensions of the multiple query sub-data blocks and the multiple key sub-data blocks. For example, the sizes of the multiple query sub-data blocks and the multiple key sub-data blocks are 64x64, where the first "64" indicates the length of the data block and the second "64" indicates the width of the data block. According to some embodiments, the multiple query sub-data blocks and the multiple key sub-data blocks being the same size means that the sizes of various corresponding dimensions of the multiple query sub-data blocks and the multiple key sub-data blocks are all the same, for example, the sizes are all 64x64.

[0077] According to some embodiments, the sizes of the multiple query sub-data blocks and the multiple key sub-data blocks are determined according to the computational granularity of the parallel computing unit that performs the attention computation. According to some embodiments, the sizes of the multiple query sub-data blocks and the multiple key sub-data blocks may be the minimum computational granularity supported by the parallel computing unit (i.e., the minimum computable data size). For example, if the minimum computational granularity supported by the parallel computing unit is 64x64, then the sizes of the multiple query sub-data blocks and the multiple key sub-data blocks are 64x64. According to some embodiments, the sizes of the multiple query sub-data blocks and the multiple key sub-data blocks may be multiples of the minimum computational granularity supported by the parallel computing unit (i.e., the minimum computable data size). For example, if the minimum computational granularity supported by the parallel computing unit is 64x64, then the sizes of the multiple query sub-data blocks and the multiple key sub-data blocks are 128x128.

[0078] According to the embodiment of the present disclosure, since the reference Figure 4The step S402 described in the above further divides the first query data block and the first key data block into query sub-data blocks and key sub-data blocks of smaller sizes, thereby refining the granularity of subsequent calculations and allowing the first query data block and the first key data block to be further divided into query sub-data blocks and key sub-data blocks of smaller sizes, respectively. Figure 3B The described case of a coarser calculation granularity omits more unnecessary calculations and reduces the computational complexity of mask calculations for data blocks on the diagonal.

[0079] Figure 5 A schematic diagram of a causal masking process with refined computation granularity according to an exemplary embodiment of the present disclosure is shown.

[0080] According to some embodiments, Figure 5 As shown in , since the query data block and the key data block have been divided into smaller query sub-data blocks and key sub-data blocks, respectively, the score data block generated based on the query data block and the key data block may also include multiple score sub-data blocks, wherein each score sub-data block corresponds to a pair of query sub-data block and key sub-data block. For example, in Figure 5 In the example, the fractional sub-data blocks with the same number belong to the same fractional data block. For example, the eight fractional sub-data blocks with the number “1” in the upper left corner of the fractional data 530 belong to the same fractional data block with a length of B. r , width B c The score data block.

[0081] According to some embodiments, since the query data block and the key data block have been divided into smaller query sub-data blocks and key sub-data blocks, respectively, more unnecessary calculations can be omitted, in particular, unnecessary calculations related to fractional data blocks across the diagonal. For example, in addition to the fractional sub-data blocks numbered "5" and "6", the calculation for generating some fractional sub-data blocks numbered "1" (i.e., fractional sub-data blocks with a column index greater than a row index and numbered "1") and subsequent calculations associated with these fractional sub-data blocks can also be omitted.

[0082] According to some embodiments, the column index of a fractional sub-data block is the serial number of the column to which the fractional sub-data belongs in the fractional data. For example, if a fractional sub-data block belongs to the fifth column of the fractional sub-data block in the fractional data, the column index of the fractional sub-data block is 5. According to some embodiments, the row index of a fractional sub-data block is the serial number of the row to which the fractional sub-data belongs in the fractional data. For example, if a fractional sub-data block belongs to the fifth row of the fractional sub-data block in the fractional data, the row index of the fractional sub-data block is 5.

[0083] According to some embodiments, as shown in FIG. Figure 4As described in step S403 of the embodiment, for each query sub-data block, based on the query sub-data block and the lower sub-data blocks corresponding to the query sub-data block, each fractional sub-data block corresponding to the query sub-data block is generated, wherein the lower sub-data block corresponding to the query sub-data block is a key sub-data block whose index of the data in the plurality of key sub-data blocks is less than or equal to the index of the data in the query sub-data block, that is, only for the fractional sub-data blocks whose column index is less than or equal to the row index (for example, Figure 5 All fractional sub-data blocks numbered 3 or 4 and some fractional sub-data blocks numbered 1, 2, 7 or 8 in the data block), and perform calculations to generate these fractional sub-data blocks.

[0084] It should be understood that Figure 5 There are still fractional sub-data blocks across the diagonal 531, that is, fractional sub-data blocks whose row index is equal to the column index (for example, the fractional sub-data block of the first row and the first column, the fractional sub-data block of the second row and the second column, ..., the fractional sub-data block of the eighth row and the eighth column). For this part of the fractional sub-data blocks across the diagonal, it is still necessary to "shield" the data above the diagonal that does not conform to the causal mask.

[0085] According to some embodiments, as shown in FIG. Figure 4 As described in step S404 of FIG4 , for the diagonal sub-data blocks in each fractional sub-data block, masking is performed on the data whose column index in the diagonal sub-data block is greater than the row index, wherein the row index of the diagonal sub-data block in each fractional sub-data block is equal to the column index of the diagonal sub-data block in each fractional sub-data block in the row, that is, for the fractional sub-data block whose column index is less than or equal to the row index (for example, Figure 5 ), masking is performed on the data above the diagonal in these fractional sub-data blocks. According to some embodiments, mask calculations are only performed on the fractional sub-data blocks with smaller sizes that cross the diagonal 531, rather than on the fractional data blocks with larger sizes, thereby reducing the computational complexity of the mask calculations.

[0086] According to some embodiments, the column index of data in a diagonal sub-data block is the sequence number of the column to which the data belongs in the diagonal sub-data block. For example, if a data belongs to the fifth column in the diagonal sub-data block, the column index of the data is 5. According to some embodiments, the row index of data in a diagonal sub-data block is the sequence number of the row to which the data belongs in the diagonal sub-data block. For example, if a data belongs to the fifth row in the diagonal sub-data block, the row index of the data is 5.

[0087] According to some embodiments, masking data in a diagonal sub-data block whose column index is greater than the row index includes: subtracting the mask data block from the diagonal sub-data block, wherein the size of the mask data block is the same as the size of the diagonal sub-data block, and data in the mask data block whose column index is greater than the row index is a negative maximum value, and data in the mask data block whose column index is less than or equal to the row index is zero.

[0088] Figure 6 FIG. 6 is a schematic diagram showing a mask data block 600 for masking a diagonal sub-data block according to an exemplary embodiment of the present disclosure. Figure 6 As shown, the data in the first row and first column, the first to second columns of the second row, the first to third columns of the third row, and the first to fourth columns of the fourth row of the mask data block are zero, and the data in the first row, second to fourth columns, the second row, third to fourth columns, and the third row, fourth column of the mask data block are negative maximum values ​​(for example, -1000).

[0089] According to some embodiments, after Figure 4 After the steps S403 and S404 described above, multiple fractional sub-data blocks that conform to the causal mask mechanism are generated, wherein the amount of data generated due to omission is relatively large compared to the reference Figure 3B More described (e.g., no generation Figure 5 The fractional sub-data blocks of the first row, second column to fourth column, second row, third column to fourth column, third row, fourth column, fifth row, sixth column to eighth column, sixth row, seventh column to eighth column, and seventh row, eighth column) are in reference Figure 4 In the subsequent calculation in step S405 described above, the amount of data related to the omitted fractional sub-data blocks can be saved.

[0090] Figure 7 The exemplary embodiment of the present disclosure is shown Figure 4 In the method, an output data block corresponding to a first data block set is generated based on the first score data block and the first value data block (eg, Figure 4 Schematic diagram of the process of step S405 in FIG. Figure 7 As shown, step S405 includes:

[0091] Step S701: normalizing each fractional sub-data block to generate a normalized data block;

[0092] Step S702: performing random deactivation on the normalized data block to generate a randomly deactivated data block; and

[0093] Step S703: Generate an output data block corresponding to the first data block set based on the random deactivation data block and the first value data block.

[0094] According to some embodiments, normalizing each fractional sub-data block may be applying a normalization exponential (softmax) function to each fractional sub-data block.

[0095] According to some embodiments, in step S702, randomly inactivating the normalized data block includes: randomly generating a random mask data block; comparing the data in the random mask data block with a threshold, wherein if the data is greater than or equal to the threshold, the data is not modified, and if the data is less than the threshold, the data is changed to zero; and generating a randomly inactivated data block based on the normalized data block and the random mask data block (for example, dot multiplying the two).

[0096] According to some embodiments, the random deactivation process described above is also performed during the process of training the attention calculation, and it is desirable that the random mask data used during the attention calculation and the training of the attention calculation be the same. However, the sizes of the fractional data blocks used during the attention calculation and the training of the attention calculation are typically different, and the random mask data is typically generated based on the fractional data blocks, making it difficult to ensure that the random mask data used during the attention calculation and the training of the attention calculation are the same.

[0097] According to some embodiments, normalizing each fractional sub-data block to generate a normalized data block includes: for each fractional sub-data block, normalizing the fractional sub-data block to generate a normalized sub-data block corresponding to the fractional sub-data block, and randomly inactivating the normalized data block to generate a randomly inactivated data block includes: for each fractional sub-data block, generating a mask sub-data block corresponding to the fractional sub-data block; and for each fractional sub-data block, determining a randomly inactivated sub-data block corresponding to the fractional sub-data block based on the normalized sub-data block and the mask sub-data block corresponding to the fractional sub-data block, wherein the randomly inactivated data block includes the randomly inactivated sub-data blocks corresponding to each fractional sub-data block.

[0098] Figure 8 A schematic diagram of a random deactivation process with refined computation granularity according to an exemplary embodiment of the present disclosure is shown.

[0099] According to some embodiments, since the generated fractional sub-data blocks only include fractional sub-data blocks whose row index is greater than or equal to the column index (e.g. Figure 5 The normalized data 830 that needs to be deactivated also only includes the normalized sub-data blocks whose row index is greater than or equal to the column index (for example, Figure 8 831 or the normalized sub-data blocks shown below or across the diagonal line 831).

[0100] According to some embodiments, Figure 8 As shown, each normalized sub-data block in the normalized data 830 is numbered, and for each normalized sub-data block, a mask sub-data block corresponding to the normalized sub-data block is generated based on the number of the normalized sub-data block.

[0101] According to some embodiments, the number of each normalized sub-data block may be generated according to the row index and column index of the normalized sub-data block in the normalized data 830. According to other embodiments, the number of the normalized sub-data block may also be generated according to other rules.

[0102] According to some embodiments, in addition to generating mask sub-data blocks based on each fractional sub-data block as described above, mask sub-data blocks can also be generated based on each group of fractional sub-data blocks, wherein each group of fractional sub-data blocks includes multiple consecutive fractional sub-data blocks, and the length of each group of fractional sub-data blocks is the common divisor of the length of the query data block and the length of the query sample data block, and the width of each group of fractional sub-data is the common divisor of the length of the key data block and the length of the key sample data block, wherein the query sample data and the key sample data are used to train attention calculations, the query sample data is divided into multiple query sample data blocks, and the key sample data is divided into multiple key sample data blocks.

[0103] According to some embodiments, the mask sub-data blocks corresponding to the respective fractional sub-data blocks are also used for the random deactivation process in the training attention calculation. In the embodiments as described in the present disclosure, because the size of the normalized sub-data block depends on the size of the divided query sub-data block and the key sub-data block, rather than the size of the query data block and the key data block, the size of the normalized sub-data block can be kept consistent during the process of performing attention calculation and training attention calculation (for example, when the size of the query sub-data block and the key sub-data block divided during the process of attention calculation is the same as the size of the query sub-data block and the key sub-data block divided during the process of training attention calculation), so that the mask sub-data block generated based on the normalized sub-data block can also be kept consistent during the process of performing attention calculation and training attention calculation.

[0104] According to some embodiments, for each fractional sub-data block, determining a randomly inactivated sub-data block corresponding to the fractional sub-data block based on the normalized sub-data block and the masked sub-data block corresponding to the fractional sub-data block includes: comparing data in the masked sub-data block with a threshold, wherein if the data is greater than or equal to the threshold, the data is not modified, and if the data is less than the threshold, the data is changed to zero; and generating a randomly inactivated data block based on the normalized sub-data block and the masked sub-data block (for example, dot-multiplying the two).

[0105] According to some embodiments, the random deactivation process in step S702 may also be omitted.

[0106] According to some embodiments, the data processing method as described in the present disclosure further includes, before loading the first data block set into the second memory: in the first memory, dividing the query data into multiple query data blocks, dividing the key data into multiple key data blocks, and dividing the value data into multiple value data blocks, wherein the multiple query data blocks have the same size, and the multiple key data blocks and the multiple value data blocks have the same size. According to an embodiment of the present disclosure, the size of the query data block, and the size of the multiple key data blocks and the multiple value data blocks can be determined based on at least one factor of the storage capacity of the second memory, hardware address and length alignment requirements, and software pipelining, and based on the determined size of the query data block, and the size of the multiple key data blocks and the multiple value data blocks, the query data is divided into multiple query data blocks, the key data is divided into multiple key data blocks, and the value data is divided into multiple value data blocks.

[0107] According to some embodiments, when training attention calculations, in order to reduce the amount of data that needs to be stored, the attention scores, normalized data, and random inactivation data are recalculated. Therefore, the above reference Figure 4 The described method of refining the computation granularity in attention computation is similarly applied to the above-mentioned recomputation in training attention computation.

[0108] Figure 9 A flowchart of a method 900 for training attention calculation according to an exemplary embodiment of the present disclosure is shown. The method 900 is used to perform training based on query sample data, key sample data, value sample data, and output sample data stored in a first memory, wherein the query sample data includes a plurality of query sample data blocks, the key sample data includes a plurality of key sample data blocks, the value sample data includes a plurality of value sample data blocks, and the output sample data includes a plurality of output sample data blocks. Figure 9 As shown, method 900 includes:

[0109] Step S901: loading a first data block set into a second memory, wherein the first data block set includes a first query sample data block, a first key sample data block, a first value sample data block, and a first output sample data block, and the access speed of the second memory is greater than the access speed of the first memory;

[0110] Step S902: Divide the first query sample data block and the first key sample data block into a plurality of query sub-data blocks and a plurality of key sub-data blocks respectively, wherein the plurality of query sub-data blocks and the plurality of key sub-data blocks have the same size;

[0111] Step S903: For each query sub-data block, generate fractional sub-data blocks corresponding to the query sub-data block based on the query sub-data block and the lower sub-data blocks corresponding to the query sub-data block, wherein the lower sub-data blocks corresponding to the query sub-data block are key sub-data blocks whose indexes of data in the key sub-data blocks are less than or equal to the index of data in the query sub-data block;

[0112] Step S904: for one or more diagonal sub-data blocks in each fractional sub-data block, mask the data in each diagonal sub-data block whose column index is greater than the row index to generate a masked sub-data block corresponding to the diagonal sub-data block, wherein the row index of the diagonal sub-data block in each fractional sub-data block is equal to the column index of the diagonal sub-data block in each fractional sub-data block; and

[0113] Step S905: Based on the first fractional sample data block and the first output sample data block, update the parameters of the attention calculation, wherein the first fractional sample data block includes the fractional sub-data blocks other than one or more diagonal sub-data blocks in each fractional sub-data block and the mask sub-data blocks corresponding to the one or more diagonal sub-data blocks.

[0114] According to some embodiments, similar to attention calculation, the first query sample data block and the first key sample data block loaded into the second memory for training of attention calculation are further divided into multiple query sub-data blocks and multiple key sub-data blocks, thereby reducing the size of the data blocks for actual calculation, refining the calculation granularity, and omitting more unnecessary calculations related to causal masks; and, the size of the query sub-data blocks and key sub-data blocks used for actual calculation is reduced, and the size of the generated fractional sub-data blocks is relatively reduced, thereby reducing the amount of calculation for masking the fractional sub-data blocks on the diagonal.

[0115] According to some embodiments, the sizes of the query sub-data block and the key sub-data block are determined according to the computational granularity of the parallel computing units that perform the attention computation.

[0116] According to some embodiments, masking data in a diagonal sub-data block whose column index is greater than the row index includes: subtracting the mask data block from the diagonal sub-data block, wherein the size of the mask data block is the same as the size of the diagonal sub-data block, and data in the mask data block whose column sequence number is greater than the row sequence number is a negative maximum value, and data in the mask data block whose column index is less than or equal to the row index is zero.

[0117] According to some embodiments, updating the parameters of attention calculation based on the first fractional sample data block and the first output sample data block includes: normalizing each fractional sub-data block to generate a normalized data block; randomly deactivating the normalized data block to generate a randomly deactivated data block; and updating the parameters of attention calculation based on the randomly deactivated data block, the first value sample data block and the first output sample data block.

[0118] According to some embodiments, normalizing each fractional sub-data block to generate a normalized data block includes: for each fractional sub-data block, normalizing the fractional sub-data block to generate a normalized sub-data block corresponding to the fractional sub-data block, and randomly inactivating the normalized data block to generate a randomly inactivated data block includes: for each fractional sub-data block, generating a mask sub-data block corresponding to the fractional sub-data block; and for each fractional sub-data block, determining the randomly inactivated sub-data block corresponding to the fractional sub-data block based on comparison between the normalized sub-data block and the mask sub-data block corresponding to the fractional sub-data block, wherein the randomly inactivated data block includes the randomly inactivated sub-data blocks corresponding to each fractional sub-data block.

[0119] According to some embodiments, the mask sub-data blocks corresponding to the respective fractional sub-data blocks are also used for the random dropout process in the attention calculation.

[0120] According to some embodiments, the method for training attention calculation as described in the present disclosure further includes, before loading the first data block set into the second memory: in the first memory, dividing the query sample data into multiple query sample data blocks, dividing the key sample data into multiple key sample data blocks, dividing the value data into multiple value sample data blocks, and dividing the output sample data into multiple output sample data blocks, wherein the multiple query sample data blocks and the multiple output sample data blocks have the same size, and the multiple key sample data blocks and the multiple value sample data blocks have the same size.

[0121] According to some embodiments, since the recalculation of attention scores, normalized data and random inactivation data during training attention calculation is similar to the corresponding process in attention calculation, the various aspects and corresponding technical effects described above with reference to attention calculation can be applied to the process of training attention calculation, and therefore will not be repeated here.

[0122] Figure 10 A structural block diagram of a data processing device 1000 according to an exemplary embodiment of the present disclosure is shown, wherein the device 1000 is used to perform attention calculation on query data, key data, and value data stored in a first memory, wherein the query data includes multiple query data blocks, the key data includes multiple key data blocks, and the value data includes multiple value data blocks. The device 1000 includes:

[0123] The data loading unit 1001 is configured to: load a first data block set into a second memory, wherein the first data block set includes a first query data block, a first key data block, and a first value data block, and an access speed of the second memory is greater than an access speed of the first memory;

[0124] The data block division unit 1002 is configured to: divide the first query data block and the first key data block into a plurality of query sub-data blocks and a plurality of key sub-data blocks respectively, wherein the plurality of query sub-data blocks and the plurality of key sub-data blocks have the same size;

[0125] The score determination unit 1003 is configured to: for each query sub-data block, generate scores for the sub-data blocks corresponding to the query sub-data block based on the query sub-data block and the lower sub-data blocks corresponding to the query sub-data block, wherein the lower sub-data blocks corresponding to the query sub-data block are key sub-data blocks whose indexes of data in the key sub-data blocks are less than or equal to the index of data in the query sub-data block;

[0126] a masking unit 1004 configured to: for one or more diagonal sub-data blocks in each fractional sub-data block, mask data in each diagonal sub-data block whose column index is greater than the row index, to generate a masked sub-data block corresponding to the diagonal sub-data block, wherein the row index of the diagonal sub-data block in each fractional sub-data block is equal to the column index of the diagonal sub-data block in each fractional sub-data block; and

[0127] The output unit 1005 is configured to: generate an output data block corresponding to the first data block set based on the first fractional data block and the first value data block, wherein the first fractional data block includes the fractional sub-data blocks in each fractional sub-data block except for one or more diagonal sub-data blocks and the mask sub-data blocks corresponding to the one or more diagonal sub-data blocks.

[0128] Figure 11 A structural block diagram of an apparatus 1100 for training attention calculation according to an exemplary embodiment of the present disclosure is shown, wherein query sample data, key sample data, value sample data, and output sample data are stored in a first memory, the query sample data includes a plurality of query sample data blocks, the key sample data includes a plurality of key sample data blocks, the value sample data includes a plurality of value sample data blocks, and the output sample data includes a plurality of output sample data blocks. The apparatus 1100 includes:

[0129] The data loading unit 1101 is configured to: load a first data block set into a second memory, wherein the first data block set includes a first query sample data block, a first key sample data block, a first value sample data block, and a first output sample data block, and an access speed of the second memory is greater than an access speed of the first memory;

[0130] The data block division unit 1102 is configured to: divide the first query sample data block and the first key sample data block into a plurality of query sub-data blocks and a plurality of key sub-data blocks, respectively, wherein the plurality of query sub-data blocks and the plurality of key sub-data blocks have the same size;

[0131] The score determination unit 1103 is configured to: for each query sub-data block, generate scores for the sub-data blocks corresponding to the query sub-data block based on the query sub-data block and the lower sub-data blocks corresponding to the query sub-data block, wherein the lower sub-data blocks corresponding to the query sub-data block are key sub-data blocks whose indexes of data in the key sub-data blocks are less than or equal to the index of data in the query sub-data block;

[0132] a masking unit 1104 configured to: for one or more diagonal sub-data blocks in each fractional sub-data block, mask data in each diagonal sub-data block whose column index is greater than its row index, to generate a masked sub-data block corresponding to the diagonal sub-data block, wherein the row index of the diagonal sub-data block in each fractional sub-data block is equal to the column index of the diagonal sub-data block in each fractional sub-data block; and

[0133] The parameter updating unit 1105 is configured to update the parameters of the attention calculation based on the first fractional sample data block and the first output sample data block, wherein the first fractional sample data block includes the fractional sub-data blocks other than one or more diagonal sub-data blocks in each fractional sub-data block and the mask sub-data blocks corresponding to the one or more diagonal sub-data blocks.

[0134] According to one aspect of the present disclosure, a chip is further provided, comprising the data processing device 1000 or the device 1100 for training attention calculation as described above.

[0135] According to another aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned data processing method and / or method for training attention calculation.

[0136] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned data processing method and / or method for training attention calculation.

[0137] According to another aspect of the present disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the above-mentioned data processing method and / or method for training attention calculation.

[0138] refer to Figure 12 , a block diagram of an electronic device 1200 that can serve as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0139] like Figure 12 As shown, the device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. Various programs and data required for the operation of the device 1200 can also be stored in the RAM 1203. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0140] Multiple components in the device 1200 are connected to the I / O interface 1205, including: an input unit 1206, an output unit 1207, a storage unit 1208, and a communication unit 1209. The input unit 1206 can be any type of device that can input information to the device 1200. The input unit 1206 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone, and / or a remote control. The output unit 1207 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1208 can include but is not limited to a magnetic disk and an optical disk. The communication unit 1209 allows the device 1200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0141] The computing unit 1201 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1201 performs the various methods and processes described above, such as data processing methods and / or methods for training attention calculations. For example, in some embodiments, the data processing methods and / or methods for training attention calculations can be implemented as computer software programs that are tangibly contained in a machine-readable medium, such as a storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, the data processing methods and / or methods for training attention calculations described above can be executed.

[0142] Alternatively, in other embodiments, the computing unit 1201 may be configured to perform the data processing method and / or the method for training attention calculation in any other suitable manner (e.g., by means of firmware).

[0143] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0144] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0145] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0147] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0148] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0149] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0150] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. In addition, the steps may be performed in an order different from that described in this disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear after this disclosure.

Claims

1. A data processing method for performing attention calculation on query data, key data, and value data stored in a first memory, wherein the query data includes a plurality of query data blocks, the key data includes a plurality of key data blocks, and the value data includes a plurality of value data blocks, the method comprising: Loading a first data block set into a second memory, wherein the first data block set includes a first query data block, a first key data block, and a first value data block, and an access speed of the second memory is greater than an access speed of the first memory; Dividing the first query data block and the first key data block into a plurality of query sub-data blocks and a plurality of key sub-data blocks respectively, wherein the plurality of query sub-data blocks and the plurality of key sub-data blocks have the same size; For each query sub-data block, generating respective fractional sub-data blocks corresponding to the query sub-data block based on the query sub-data block and respective lower sub-data blocks corresponding to the query sub-data block, wherein the lower sub-data block corresponding to the query sub-data block is a key sub-data block whose index of data in the plurality of key sub-data blocks is less than or equal to the index of data in the query sub-data block; For one or more diagonal sub-data blocks in each of the fractional sub-data blocks, masking data whose column index is greater than the row index in each diagonal sub-data block to generate a masked sub-data block corresponding to the diagonal sub-data block, wherein the row index of the diagonal sub-data block in each of the fractional sub-data blocks is equal to the column index of the diagonal sub-data block in each of the fractional sub-data blocks; and Based on a first fractional data block and the first value data block, an output data block corresponding to the first data block set is generated, wherein the first fractional data block includes the fractional sub-data blocks in the respective fractional sub-data blocks except the one or more diagonal sub-data blocks and the mask sub-data blocks corresponding to the one or more diagonal sub-data blocks.

2. The method according to claim 1, wherein The sizes of the query sub-data block and the key sub-data block are determined according to the computational granularity of the parallel computing unit that performs the attention computation.

3. The method according to claim 1 or 2, wherein: The masking of data in the diagonal sub-data block whose column index is greater than the row index comprises: Subtract the mask data block from the diagonal sub-data block, wherein the size of the mask data block is the same as the size of the diagonal sub-data block, and data in the mask data block whose column index is greater than the row index is a negative maximum value, and data in the mask data block whose column index is less than or equal to the row index is zero.

4. The method according to claim 1, wherein Generating an output data block corresponding to the first data block set based on the first score data block and the first value data block includes: Normalizing each fractional sub-data block to generate a normalized data block; performing random deactivation on the normalized data block to generate a randomly deactivated data block; and An output data block corresponding to the first data block set is generated based on the random deactivation data block and the first value data block.

5. The method according to claim 4, wherein Normalizing each fractional sub-data block to generate a normalized data block includes: For each fractional sub-data block, normalize the fractional sub-data block to generate a normalized sub-data block corresponding to the fractional sub-data block, and The randomly deactivating the normalized data block to generate a randomly deactivated data block includes: For each fractional sub-data block, generate a mask sub-data block corresponding to the fractional sub-data block; and For each fractional sub-data block, based on the normalized sub-data block and the mask sub-data block corresponding to the fractional sub-data block, a randomly inactivated sub-data block corresponding to the fractional sub-data block is determined. The randomly inactivated data block includes randomly inactivated sub-data blocks corresponding to the respective fractional sub-data blocks.

6. The method according to claim 5, wherein: The masked sub-data blocks corresponding to the respective score sub-data blocks are also used to train the random dropout process in the attention calculation.

7. The method according to claim 1 or 2, further comprising, before loading the first set of data blocks into the second memory: In the first memory, the query data is divided into a plurality of query data blocks, the key data is divided into a plurality of key data blocks, and the value data is divided into a plurality of value data blocks. in, The multiple query data blocks have the same size, and the multiple key data blocks and the multiple value data blocks have the same size.

8. A method for training attention computation, wherein: Query sample data, key sample data, value sample data, and output sample data are stored in a first memory, wherein the query sample data includes a plurality of query sample data blocks, the key sample data includes a plurality of key sample data blocks, the value sample data includes a plurality of value sample data blocks, and the output sample data includes a plurality of output sample data blocks. The method includes: Loading a first data block set into a second memory, wherein the first data block set includes a first query sample data block, a first key sample data block, a first value sample data block, and a first output sample data block, and an access speed of the second memory is greater than an access speed of the first memory; Dividing the first query sample data block and the first key sample data block into a plurality of query sub-data blocks and a plurality of key sub-data blocks respectively, wherein the plurality of query sub-data blocks and the plurality of key sub-data blocks have the same size; For each query sub-data block, generating respective fractional sub-data blocks corresponding to the query sub-data block based on the query sub-data block and respective lower sub-data blocks corresponding to the query sub-data block, wherein the lower sub-data block corresponding to the query sub-data block is a key sub-data block whose index of data in the plurality of key sub-data blocks is less than or equal to the index of data in the query sub-data block; For one or more diagonal sub-data blocks in each of the fractional sub-data blocks, masking data whose column index is greater than the row index in each diagonal sub-data block to generate a masked sub-data block corresponding to the diagonal sub-data block, wherein the row index of the diagonal sub-data block in each of the fractional sub-data blocks is equal to the column index of the diagonal sub-data block in each of the fractional sub-data blocks; and updating the parameters of the attention calculation based on a first fractional sample data block and the first output sample data block, wherein the first fractional sample data block includes the fractional sub-data blocks other than the one or more diagonal sub-data blocks in the respective fractional sub-data blocks and the mask sub-data blocks corresponding to the one or more diagonal sub-data blocks.

9. The method according to claim 8, wherein The sizes of the query sub-data block and the key sub-data block are determined according to the computational granularity of the parallel computing unit that performs the attention computation.

10. The method according to claim 8 or 9, wherein: The masking of data in the diagonal sub-data block whose column index is greater than the row index comprises: Subtract the mask data block from the diagonal sub-data block, wherein the size of the mask data block is the same as the size of the diagonal sub-data block, and data in the mask data block whose column number is greater than the row number is a negative maximum value, and data in the mask data block whose column index is less than or equal to the row index is zero.

11. The method according to claim 8, wherein The updating of the parameters of the attention calculation based on the first score sample data block and the first output sample data block includes: Normalizing each fractional sub-data block to generate a normalized data block; performing random deactivation on the normalized data block to generate a randomly deactivated data block; and Parameters of the attention calculation are updated based on the random deactivation data block, the first value sample data block and the first output sample data block.

12. The method according to claim 11, wherein Normalizing each fractional sub-data block to generate a normalized data block includes: For each fractional sub-data block, normalize the fractional sub-data block to generate a normalized sub-data block corresponding to the fractional sub-data block, and The randomly deactivating the normalized data block to generate a randomly deactivated data block includes: For each fractional sub-data block, generate a mask sub-data block corresponding to the fractional sub-data block; and For each fractional sub-data block, based on the comparison between the normalized sub-data block and the mask sub-data block corresponding to the fractional sub-data block, a randomly inactivated sub-data block corresponding to the fractional sub-data block is determined. The randomly inactivated data block includes randomly inactivated sub-data blocks corresponding to the respective fractional sub-data blocks.

13. The method according to claim 12, wherein: The mask sub-data blocks corresponding to the respective score sub-data blocks are also used for the random deactivation process in the attention calculation.

14. The method according to claim 12 or 13, further comprising, before loading the first set of data blocks into the second memory: In the first memory, the query sample data is divided into the plurality of query sample data blocks, the key sample data is divided into the plurality of key sample data blocks, the value sample data is divided into the plurality of value sample data blocks, and the output sample data is divided into the plurality of output sample data blocks. in, The plurality of query specimen data chunks and the plurality of output specimen data chunks are of the same size, and the plurality of key specimen data chunks and the plurality of value specimen data chunks are of the same size.

15. A data processing device for performing attention calculation on query data, key data, and value data stored in a first memory, wherein the query data comprises a plurality of query data blocks, the key data comprises a plurality of key data blocks, and the value data comprises a plurality of value data blocks, the device comprising: a data loading unit configured to: load a first data block set into a second memory, wherein the first data block set includes a first query data block, a first key data block, and a first value data block, and an access speed of the second memory is greater than an access speed of the first memory; a data block division unit configured to: divide the first query data block and the first key data block into a plurality of query sub-data blocks and a plurality of key sub-data blocks, respectively, wherein the plurality of query sub-data blocks and the plurality of key sub-data blocks have the same size; a score determination unit configured to: for each query sub-data block, generate, based on the query sub-data block and each lower sub-data block corresponding to the query sub-data block, each score sub-data block corresponding to the query sub-data block, wherein the lower sub-data block corresponding to the query sub-data block is a key sub-data block whose index of data in the plurality of key sub-data blocks is less than or equal to the index of data in the query sub-data block; a masking unit configured to: for one or more diagonal sub-data blocks in each of the fractional sub-data blocks, mask data whose column index is greater than the row index in each diagonal sub-data block to generate a masked sub-data block corresponding to the diagonal sub-data block, wherein the row index of the diagonal sub-data block in each of the fractional sub-data blocks is equal to the column index of the diagonal sub-data block in each of the fractional sub-data blocks; and An output unit is configured to generate an output data block corresponding to the first data block set based on a first fractional data block and the first value data block, wherein the first fractional data block includes the fractional sub-data blocks in the respective fractional sub-data blocks except the one or more diagonal sub-data blocks and the mask sub-data blocks corresponding to the one or more diagonal sub-data blocks.

16. A device for training attention computing, wherein: Query sample data, key sample data, value sample data, and output sample data are stored in a first memory, wherein the query sample data includes a plurality of query sample data blocks, the key sample data includes a plurality of key sample data blocks, the value sample data includes a plurality of value sample data blocks, and the output sample data includes a plurality of output sample data blocks. The apparatus includes: a data loading unit configured to: load a first data block set into a second memory, wherein the first data block set includes a first query sample data block, a first key sample data block, a first value sample data block, and a first output sample data block, and an access speed of the second memory is greater than an access speed of the first memory; a data block division unit configured to: divide the first query sample data block and the first key sample data block into a plurality of query sub-data blocks and a plurality of key sub-data blocks, respectively, wherein the plurality of query sub-data blocks and the plurality of key sub-data blocks have the same size; a score determination unit configured to: for each query sub-data block, generate, based on the query sub-data block and each lower sub-data block corresponding to the query sub-data block, each score sub-data block corresponding to the query sub-data block, wherein the lower sub-data block corresponding to the query sub-data block is a key sub-data block whose index of data in the plurality of key sub-data blocks is less than or equal to the index of data in the query sub-data block; a masking unit configured to: for one or more diagonal sub-data blocks in each of the fractional sub-data blocks, mask data whose column index is greater than the row index in each diagonal sub-data block to generate a masked sub-data block corresponding to the diagonal sub-data block, wherein the row index of the diagonal sub-data block in each of the fractional sub-data blocks is equal to the column index of the diagonal sub-data block in each of the fractional sub-data blocks; and A parameter updating unit is configured to update the parameters of the attention calculation based on a first fractional sample data block and the first output sample data block, wherein the first fractional sample data block includes the fractional sub-data blocks other than the one or more diagonal sub-data blocks in the respective fractional sub-data blocks and the mask sub-data blocks corresponding to the one or more diagonal sub-data blocks.

17. A chip comprising the device according to any one of claims 15 to 16.

18. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause a computer to execute the method according to any one of claims 1-14.

20. A computer program product comprising a computer program, wherein The computer program implements the method according to any one of claims 1 to 14 when executed by a processor.

Images