A chip, a data processing method, an electronic device, a computer readable storage medium and a computer program product

Abstract

The application provides a chip, a data processing method, an electronic device, a computer readable storage medium and a computer program product. The chip comprises: a static random access memory, a microcontroller unit, a coprocessor and a computing engine; the static random access memory is used for storing a first data block and is also used for storing M causal mask blocks corresponding to M mask data distributions; the mask data distribution is a data distribution of the first data block changed by masking; the microcontroller unit is used for sending a mask instruction for the first data block to the coprocessor; the coprocessor is used for determining a target causal mask block from the M causal mask blocks based on a first index and a second index of the first data block in response to the mask instruction; and the computing engine is used for masking the first data block based on the target causal mask block to obtain a fourth data block. Through the application, the resource consumption of masking can be reduced.

Description

Technical Field

[0001] This application relates to artificial intelligence technology in the field of computer applications, and more particularly to a chip, a data processing method, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] In the field of artificial intelligence, attention mechanisms are often used to improve the predictive accuracy of AI. To enhance the effectiveness of attention mechanisms, it is necessary to mask the relevance score matrices of the query matrix and key matrix based on causal masking. However, the causal mask blocks used for masking correspond one-to-one with the data blocks in the relevance score matrix, meaning that the amount of data in each causal mask block used for masking is the same as the amount of data in the relevance score matrix. This affects the space occupied by the causal mask blocks and thus the resource consumption of the masking. Summary of the Invention

[0003] This application provides a chip, a data processing method, an electronic device, a computer-readable storage medium, and a computer program product that can reduce the resource consumption of the mask.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a chip, which includes a static random access memory, a microcontroller unit, a coprocessor, and a computing engine;

[0006] The static random access memory is used to store a first data block, wherein the first data block is any data block in the relevance score matrix of the query matrix and the key matrix; it is also used to store M causal mask blocks corresponding to M mask data distributions, wherein the mask data distributions are the data distributions of the first data block that have been masked, and M is a positive integer;

[0007] The microcontroller unit is used to send a mask instruction for the first data block to the coprocessor;

[0008] The coprocessor is configured to, in response to the masking instruction, determine a target causal masking block from the M causal masking blocks based on a first index and a second index of the first data block, wherein the first index is the index of the second data block in the query matrix used to calculate the first data block, and the second index is the index of the third data block in the key matrix used to calculate the first data block;

[0009] The computing engine is used to mask the first data block based on the target causal mask block to obtain the fourth data block.

[0010] In this embodiment, the coprocessor is further configured to control the loading storage unit to load the fifth data block from the dynamic random access memory (DRAM) into the static random access memory (SRAM), wherein the fifth data block is a data block in the value matrix; control the computing engine to perform calculations on the fourth data block and the fifth data block to obtain a feature data block; and control the loading storage unit to store the feature data block into the DRAM, so as to perform prediction based on the feature data block in the DRAM to obtain an artificial intelligence prediction result.

[0011] This application provides a data processing method for a chip, the method comprising:

[0012] The microcontroller unit sends a mask instruction to the coprocessor for a first data block in the static random access memory, wherein the first data block is any data block in the relevance score matrix of the query matrix and the key matrix;

[0013] The coprocessor performs the following processing in response to the mask instruction:

[0014] Based on the first index and the second index of the first data block, a target causal mask block is determined from M causal mask blocks pre-stored in static random access memory. The first index is the index of the second data block used to calculate the first data block in the query matrix, and the second index is the index of the third data block used to calculate the first data block in the key matrix. The M causal mask blocks correspond to M mask data distributions, and the mask data distributions are the data distributions of the first data block that are changed by the mask. M is a positive integer.

[0015] The control computing engine masks the first data block based on the target causal mask block to obtain the fourth data block.

[0016] In this embodiment of the application, determining the target causal mask block from M causal mask blocks pre-stored in static random access memory based on the first index and the second index of the first data block includes:

[0017] Based on the first index and the second index of the first data block, the target mask data distribution of the first data block that is masked is determined, wherein the M mask data distributions include the target mask data distribution;

[0018] Based on the correspondence between the M causal mask block identifiers and the M mask data distributions pre-stored in the static random access memory, the target causal mask block identifier corresponding to the target mask data distribution is determined;

[0019] From the M causal mask blocks pre-stored in the static random access memory, determine the target causal mask block corresponding to the target causal mask block identifier.

[0020] In this embodiment of the application, the chip further includes a central processing unit;

[0021] Before determining the target causal mask block from M causal mask blocks pre-stored in static random access memory based on the first and second indices of the first data block, the method further includes:

[0022] The central processing unit performs the following processing:

[0023] Based on the first number of the second data blocks in the query matrix and the second number of the third data blocks in the key matrix, the distribution of the M mask data that the first data blocks are masked is determined.

[0024] Determine the M causal mask blocks corresponding to the M mask data distributions;

[0025] The M causal mask blocks are stored in the static random access memory.

[0026] In this embodiment of the application, determining the distribution of the M masked data whose first data blocks are masked based on the first number of the second data blocks in the query matrix and the second number of the third data blocks in the key matrix includes:

[0027] Determine the multiple between the first number of the second data block in the query matrix and the second number of the third data block in the key matrix;

[0028] Determine the mask data distribution for the specified multiples;

[0029] Based on the data distribution of the first data block that was completely changed by the mask and the data distribution that was not changed by the mask, as well as the mask data distribution of the number multiple, the M mask data distributions of the first data block that were changed by the mask are obtained.

[0030] In this embodiment of the application, storing the M causal mask blocks into the static random access memory includes:

[0031] The M causal mask blocks are stored in a dynamic random access memory;

[0032] The coprocessor controls the loading storage unit to store M causal mask blocks from the dynamic random access memory to the static random access memory.

[0033] In this embodiment of the application, before the microcontroller unit sends a mask instruction for the first data block in the static random access memory to the coprocessor, the method further includes:

[0034] The coprocessor performs the following processing:

[0035] The control loading storage unit loads the second data block and the third data block from the dynamic random access memory into the static random access memory. The second data block is obtained by dividing the query matrix into data blocks based on the first dimension granularity in the preset calculation granularity, and the third data block is obtained by dividing the key matrix into data blocks based on the second dimension granularity in the preset calculation granularity.

[0036] The computing engine is controlled to calculate the correlation score between the second data block and the third data block to obtain the first data block.

[0037] In this embodiment of the application, after the control computing engine masks the first data block based on the target causal mask block to obtain the fourth data block, the method further includes:

[0038] The coprocessor performs the following processing:

[0039] The control loading storage unit loads the fifth data block from the dynamic random access memory into the static random access memory, wherein the fifth data block is a data block in the value matrix;

[0040] The computing engine is controlled to perform calculations on the fourth data block and the fifth data block to obtain the feature data block;

[0041] The loading storage unit is controlled to store the feature data block into the dynamic random access memory, so as to make a prediction based on the feature data block in the dynamic random access memory and obtain an artificial intelligence prediction result.

[0042] This application provides an electronic device, which includes the chip provided in this application.

[0043] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed, implement the data processing method of the chip provided in this application.

[0044] This application provides a computer program product, including computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, they implement the data processing method of the chip provided in this application.

[0045] The embodiments of this application have at least the following beneficial effects: When the microcontroller unit masks any first data block in the relevance score matrix of the query matrix and the key matrix, the control coprocessor selects a target causal mask block corresponding to the first data block from M causal mask blocks in the static random access memory to mask the first data block; since the M causal mask blocks are determined based on the M mask data distributions that the first data block changes during the masking process, and the mask data distribution is the data distribution that the first data block changes during masking, first data blocks with the same data distribution that has been changed by masking can reuse the same causal mask block, reducing the number of causal mask blocks; furthermore, by selecting a target causal mask block from the M causal mask blocks to mask the first data block in the relevance score matrix, the number of causal mask blocks can be reduced, the space occupied by the causal mask blocks can be reduced, and the resource consumption of masking can be reduced. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the architecture of the data processing system provided in the embodiments of this application;

[0047] Figure 2 This is a schematic diagram of a chip for data processing provided in an embodiment of this application. Figure 1 ;

[0048] Figure 3 This is a schematic diagram of a chip for data processing provided in an embodiment of this application. Figure 2 ;

[0049] Figure 4 This is a schematic diagram of a chip for data processing provided in an embodiment of this application. Figure 3 ;

[0050] Figure 5 This is an exemplary schematic diagram of calculating a relevance score provided in an embodiment of this application;

[0051] Figure 6 This is a schematic flowchart of the data processing method for the chip provided in the embodiments of this application;

[0052] Figure 7 This is an exemplary chip schematic diagram provided in an embodiment of this application;

[0053] Figure 8 This is a schematic diagram of an exemplary data block matrix for correlation scores provided in an embodiment of this application. Figure 1 ;

[0054] Figure 9 This is an exemplary causal mask block illustration provided in the embodiments of this application. Figure 1 ;

[0055] Figure 10This is another exemplary data block matrix diagram of correlation scores provided in this application embodiment. Figure 2 ;

[0056] Figure 11 This is an exemplary causal mask block illustration provided in the embodiments of this application. Figure 2 . Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0058] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0059] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0060] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0061] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0062] 1) Artificial Intelligence (AI) is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology in computer science used to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making functions. The chip in the embodiments of this application is an artificial intelligence chip.

[0063] 2) Attention Mechanism is a neural network component that mimics the way humans allocate attention. It is used to control artificial intelligence models to dynamically focus on the most important part of the input sequence when processing sequential data and to assign different weights to different parts.

[0064] 3) Causal masking is a technique used in sequence generation tasks (such as text generation) to mask information after the current time step. It ensures that the AI ​​model generates output sequentially according to the sequence and avoids using information after the current time step, thus improving causal consistency. A causal mask is a binary matrix (or Boolean matrix) that is a square matrix corresponding to the relevance score matrix (i.e., the rows and columns of the matrix are the sequence length). When masking based on causality, the causal mask value for the masked position (the position of the element above but not including the diagonal of the relevance score matrix) is set to negative infinity (-∞), for example. After passing through an activation function (e.g., Softmax), the attention weight at the masked position will approach 0, thus achieving the masking of information at that position.

[0065] 4) Relevance score is a numerical value used to measure the similarity or matching degree between a specific query and each key in the input sequence. It determines the importance or relevance of each key in the input sequence to the current specific query. The higher the relevance score, the more closely the key matches the specific query.

[0066] 5) In response to, used to indicate the conditions or states on which the operation performed depends. When the conditions or states on which it depends are met, one or more operations performed may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0067] It should be noted that in the field of artificial intelligence, in order to improve the rationality of the attention mechanism, it is necessary to mask the relevance score matrix of the query matrix and the key matrix based on causal masking. However, the causal mask blocks used for masking correspond one-to-one with the data blocks in the relevance score matrix. That is, the amount of data in each causal mask block used for masking is the same as the amount of data in the relevance score matrix, which affects the space occupied by the causal mask blocks and thus affects the resource consumption of the mask.

[0068] Based on this, embodiments of this application provide a chip, a data processing method, an electronic device, a computer-readable storage medium, and a computer program product, which can reduce the space occupied by causal mask blocks and reduce the resource consumption of the mask. The following describes exemplary applications of the electronic device provided in this application. The electronic device provided in this application can be implemented as various types of terminals such as robots, smartphones, smartwatches, laptops, tablets, desktop computers, smart home appliances, set-top boxes, smart vehicle devices, portable music players, personal digital assistants, dedicated messaging devices, intelligent voice interaction devices, portable gaming devices, and smart speakers. It can also be implemented as a server, or a combination of both. The following will describe exemplary applications when the electronic device is implemented as a server.

[0069] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the data processing system provided in the embodiments of this application; as shown Figure 1 As shown, to support a data processing application, in the data processing system 100, terminals 200 (terminals 200-1 and 200-2 are shown as examples) are connected to server 400 via network 300. Network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. Additionally, the data processing system 100 also includes a database 500 for providing data support to server 400; and... Figure 1 The example shown illustrates a scenario where the database 500 is independent of the server 400. However, the database 500 can also be integrated into the server 400, and this embodiment does not limit this to any particular case.

[0070] Terminal 200 is used to send task description information (e.g., text to be translated, questions to be answered, etc.) of the sequence generation task to server 400 via network 300 when performing the sequence generation task; it is also used to receive the information sequence sent by server 400 via network 300 and present the information sequence (graphical interfaces 200-11 and 200-21 are shown as examples). In addition, terminal 200 can also perform masking processing on the relevance score matrix during the attention mechanism processing based on its own chip.

[0071] Server 400 is used to receive task description information sent by terminal 200 through network 300. When performing attention mechanism processing based on task description information, it performs masking processing on the correlation score matrix through its own chip to obtain information sequence; and sends the information sequence to terminal 200 through network 300.

[0072] In some embodiments, server 400 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0073] The exemplary applications of the chips provided in the embodiments of this application will be described below.

[0074] See Figure 2 , Figure 2 This is a schematic diagram of a chip for data processing provided in an embodiment of this application. Figure 1 ;like Figure 2 As shown, the chip 2-1 includes a data processing unit 2-11, which includes a microcontroller unit (MCU) 2-111, a static random access memory 2-112, a coprocessor 2-114, and a computing engine 2-116.

[0075] It should be noted that chip 2-1 may include one data processing unit 2-11 or multiple data processing units 2-11.

[0076] In this embodiment, Static Random Access Memory (SRAM) 2-112 is used to store a first data block, wherein the first data block is any data block in the relevance score matrix of the query matrix and the key matrix; SRAM 2-112 is also used to store M causal mask blocks corresponding to M mask data distributions, wherein the mask data distributions are the data distributions of the first data block that are changed by masking, and M is a positive integer; microcontroller unit 2-111 is used to send a masking instruction for the first data block to coprocessor 2-114; coprocessor 2-114 is used to determine a target causal mask block from the M causal mask blocks based on a first index and a second index of the first data block in response to the masking instruction, wherein the first index is the index of the second data block in the query matrix used to calculate the first data block, and the second index is the index of the third data block in the key matrix used to calculate the first data block; computing engine 2-116 is used to mask the first data block based on the target causal mask block to obtain a fourth data block.

[0077] It should be noted that chip 2-1 uses a block-based approach to implement the causal mask-based attention mechanism. This block-based approach involves dividing the matrix into multiple data blocks and then performing calculations at the data block level. Therefore, the relevance score calculated based on the query matrix (called the Q matrix) and the key matrix (called the K matrix) is the data block matrix. Here, the data block matrix corresponding to the relevance score is called the first data block matrix (also known as the relevance score matrix). Thus, the first data block matrix represents the relevance score of the query matrix and the key matrix. It is easy to see that the first data block matrix includes multiple first data blocks. A first data block refers to a data block within the first data block matrix, and the amount of data included in each first data block is positively correlated with the granularity of the data block division of the query matrix and the key matrix.

[0078] In this embodiment, the masking instruction refers to an instruction to mask any data block in the relevance score matrix of the query matrix and the key matrix. The microcontroller unit 2-111 sends the masking instruction for the first data block to the coprocessor 2-114, so that the coprocessor 2-114 can mask the first data block by executing the masking instruction.

[0079] In this embodiment, the coprocessor 2-114 is used to mask the first data block in the static random access memory 2-112. Additionally, the dynamic random access memory 2-113 pre-stores M causal mask blocks corresponding to the first data block matrix, where M is a positive integer. Each of the M causal mask blocks corresponds one-to-one with one of the M mask data distributions; that is, one causal mask block among the M causal mask blocks has a corresponding mask data distribution among the M mask data distributions. The mask data distributions are the data distributions of the first data block that have been masked, including data distributions that have been completely masked, data distributions that have been partially masked, and data distributions that have not been masked. Here, "pre-stored" refers to the period before masking the first data block. Furthermore, a causal mask block refers to a data block of causal mask with the same size as the first data block.

[0080] In this embodiment, each first data block includes a first index and a second index. The first index is the index in the query matrix used to calculate the second data block of the first data block, and the second index is the index in the key matrix used to calculate the third data block of the first data block. The second data block is the data block after the query matrix is ​​divided into blocks, and the third data block is the data block after the key matrix is ​​divided into blocks. The size of the second data block can be the same as or different from the size of the third data block, and this embodiment does not limit this. The coprocessor 2-114 determines the data distribution that has been masked in the first data block based on the first index and the second index, and then determines the corresponding causal mask block from the M causal mask blocks based on the determined data distribution that has been masked in the first data block, thus obtaining the target causal mask block. The coprocessor 2-114 controls the computing engine 2-116 in the static random access memory 2-112 to mask the first data block based on the target causal mask block. The masking method can be corresponding addition or corresponding multiplication, which is not limited in this embodiment. The masked first data block is the fourth data block, which can be stored in the static random access memory 2-112 for subsequent calculations. Furthermore, by performing masking on each first data block in the correlation score matrix, the corresponding fourth data block is obtained. Based on the position of the first data block in the correlation score matrix, the corresponding fourth data blocks are combined to obtain the data block matrix, which is the masked correlation score matrix.

[0081] Understandably, when the microcontroller unit 2-111 masks any first data block in the relevance score matrix of the query matrix and the key matrix, the control coprocessor 2-114 selects a target causal mask block corresponding to the first data block from the M causal mask blocks in the static random access memory 2-112 to mask the first data block. Since the M causal mask blocks are determined based on the M mask data distributions that the first data block changes after being masked, and the mask data distribution is the data distribution that the first data block changes after being masked, first data blocks with the same data distribution that has been masked can reuse the same causal mask block, reducing the number of causal mask blocks. Furthermore, by selecting a target causal mask block from the M causal mask blocks to mask the first data block in the relevance score matrix, the number of causal mask blocks can be reduced, the space occupied by the causal mask blocks can be reduced, and the resource consumption of the mask can be reduced.

[0082] In this embodiment of the application, the data processing unit 2-11 may include a sub-data processing unit, which includes a static random access memory 2-112, a coprocessor 2-114, and a computing engine 2-116. Based on Figure 2 See Figure 3 , Figure 3This is a schematic diagram of a chip for data processing provided in an embodiment of this application. Figure 2 ;like Figure 3 As shown, the data processing unit 2-11 may also contain multiple sub-data processing units, each of which includes a static random access memory 2-112, a coprocessor 2-114, and a computing engine 2-116.

[0083] It is understandable that when the data processing unit 2-11 includes multiple sub-data processing units, the microcontroller unit 2-111 can control multiple coprocessors 2-114 in the multiple sub-data processing units to perform masking on multiple first data blocks in parallel, so as to improve masking efficiency.

[0084] In this embodiment, the static random access memory 2-112 is further used to store the correspondence between M causal mask block identifiers and M mask data distributions; the coprocessor 2-114 is further used to determine the target mask data distribution of the first data block being masked based on the first index and the second index of the first data block; to determine the target causal mask block identifier corresponding to the target mask data distribution based on the correspondence between the M causal mask block identifiers and the M mask data distributions; and to determine the target causal mask block corresponding to the target causal mask block identifier from the M causal mask blocks pre-stored in the static random access memory 2-112.

[0085] In this embodiment, the coprocessor 2-114 can determine the position of the first data block in the correlation score matrix based on the first index and the second index of the first data block. Then, based on the determined position, it can determine the data distribution of the first data block that has been altered by the mask, thus obtaining the target mask data distribution. The position of the first data block in the correlation score matrix can be, for example, the row and column of the data block in the correlation score matrix, or it can be the index of at least one of the top-left, bottom-left, top-right, and bottom-right corners of the first data block, etc. This embodiment does not limit this; and the index of the first data block can be determined by the first index, the second index, and a preset calculation granularity.

[0086] It should be noted that since there is a one-to-one correspondence between the M causal mask blocks and the M mask data distributions, and each causal mask block includes a causal mask block identifier, a correspondence can be established between the M causal mask block identifiers and the M mask data distributions. The causal mask block identifier is used to identify the causal mask block. The target mask data distribution refers to the mask data distribution of the first data block to be masked, while the M mask data distributions are all the various mask data distributions included in the first data block matrix. Therefore, the M mask data distributions include the target mask data distribution. Furthermore, the coprocessor 2-114 matches the target mask data distribution with the M mask data distributions one-to-one, and determines the causal mask block identifier corresponding to the mask data distribution that matches the target mask data distribution as the target causal mask block identifier. The causal mask block identified by the target causal mask block identifier is the target causal mask block. Here, matching means that the target causal mask block identifier and the causal mask block identifier are identical.

[0087] Understandably, by pre-creating a correspondence between M causal mask block identifiers and M mask data distributions, the microcontroller unit 2-111 can determine the target causal mask block identifier corresponding to the target mask data distribution of the first data block based on this correspondence; and since the M causal mask block identifiers are used to represent M causal mask blocks, the microcontroller unit 2-111 can determine the target causal mask block from the M causal mask blocks based on the target causal mask block identifier.

[0088] based on Figure 2 See Figure 4 , Figure 4 This is a schematic diagram of a chip for data processing provided in an embodiment of this application. Figure 3 ;like Figure 4 As shown, chip 2-1 also includes a central processing unit 4-12, a load storage unit (LSU) 2-115, and a data processing unit 2-11 of chip 2-1, which also includes dynamic random-access memory (DRAM) 2-113.

[0089] In this embodiment of the application, the central processing unit 4-12 is used to determine M mask data distributions of the first data block that are masked based on the first number of the second data blocks of the query matrix and the second number of the third data blocks of the key matrix; and to determine M causal mask blocks corresponding to the M mask data distributions; and to store the M causal mask blocks in the static random access memory 2-112.

[0090] It should be noted that the first quantity is the number of data blocks after the query matrix is ​​divided, which can be the ratio of the sequence length corresponding to the query matrix (e.g., 1024) to the first dimension granularity in the preset calculation granularity (e.g., 128) (e.g., 8), and here the data blocks in the query matrix are referred to as the second data blocks; the second quantity is the number of data blocks after the key matrix is ​​divided, which can be the ratio of the sequence length corresponding to the key matrix (e.g., 1024) to the second dimension granularity in the preset calculation granularity (e.g., 256) (e.g., 4), and here the data blocks in the key matrix are referred to as the third data blocks. Since the first and second quantities determine the position of each first data block in the correlation score matrix, the CPU 4-12 can determine M masked data distributions based on the first and second quantities. Then, it determines the causal mask block corresponding to each masked data distribution, thus obtaining M causal mask blocks. It is easy to see that the element values ​​in the causal mask block are determined based on the distribution of data changed by the masking of the first data block by that causal mask block, and that the causal mask block and the first data block have a one-to-one correspondence in terms of elements. The CPU 4-12 determines the position of each masked data distribution based on the masking... The code data distribution determines each value in the causal mask block. Here, for the first position of data in the mask data distribution that has not been changed by the mask, the value at the same position in the causal mask block is set to a first value. The first value is used to keep the value at the same position in the first data block unchanged, for example, 0. For the second position of data in the mask data distribution that has been changed by the mask, the value at the same position in the causal mask block is set to a second value. The second value is used to change the value at the same position in the first data block, for example, negative infinity.

[0091] In this embodiment, the central processing unit 4-12 can obtain mask parameters, which include a preset computational granularity and a sequence length. It can then obtain a first quantity based on the ratio of the first dimension granularity to the sequence length within the preset computational granularity, and a second quantity based on the ratio of the second dimension granularity to the sequence length within the preset computational granularity. Based on the first and second quantities, it can determine M causal mask blocks. Here, the central processing unit 4-12 can store the mask parameters and the corresponding M causal mask blocks. This allows it to directly reuse the stored M causal mask blocks when a newly obtained mask parameter matches the stored mask parameter, improving the efficiency of obtaining the M causal mask blocks. Therefore, before determining causal mask blocks based on the obtained mask parameters, the central processing unit 4-12 first matches the obtained mask parameters with each of the stored mask parameters. Only when no stored mask parameter matching the obtained mask parameter is found is the causal mask block determination performed based on the obtained mask parameters.

[0092] It should be noted that when the information obtained by the central processing unit 4-12 for attention mechanism processing includes a causal mask, the causal mask is marked as disabled, and M causal mask blocks corresponding to the mask parameters are regenerated. Each first data block is masked based on the M causal mask blocks, thus achieving compatibility with the attention mechanism processing process.

[0093] Understandably, the CPU 4-12 determines the number of causal mask blocks based on the mask data distribution of the first data block after the mask has been changed, and then determines the element value of each causal mask block based on the data layout of the first data block after the mask has been changed; thus, it realizes that the masking is performed by replacing the causal mask with M causal mask blocks, reducing the number of causal mask blocks and improving the utilization of storage space.

[0094] In this embodiment of the application, the central processing unit 4-12 is further configured to determine the quantity multiple between the first quantity and the second quantity; and determine the mask data distribution of the quantity multiple; and obtain M mask data distributions of the first data block that are masked based on the data distribution of the first data block that are fully changed by the mask and the data distribution that are not changed by the mask, and the mask data distribution of the quantity multiple.

[0095] It should be noted that the quantity multiple refers to the multiple of the quantity relationship between the first quantity and the second quantity; for example, if the first quantity is 4 and the second quantity is 8, then the quantity multiple is 2; for another example, if both the first and second quantities are 8, then the quantity multiple is 1; and for yet another example, if the first quantity is 8 and the second quantity is 4, then the quantity multiple is 2; thus, the quantity multiple is a multiple of the first quantity and the second quantity. The M mask data distributions include the data distributions where the first data block is completely changed by the mask, the data distributions where the first data block is not changed by the mask, and the mask data distributions with quantity multiples; the mask data distributions with quantity multiples correspond to the cases where part of the first data block is changed by the mask, that is, the first data block in the first data block matrix that is crossed by the diagonal.

[0096] In this embodiment of the application, the central processing unit 4-12 is further configured to store M causal mask blocks into dynamic random access memory 2-113; the coprocessor 2-114 is further configured to control the loading storage unit 2-115 to store the M causal mask blocks from dynamic random access memory 2-113 into static random access memory 2-112.

[0097] It should be noted that after the central processing unit 4-12 stores the generated M causal mask blocks into the dynamic random access memory 2-113, the coprocessor 2-114 controls the loading storage unit 2-115 to store the M causal mask blocks from the dynamic random access memory 2-113 into the static random access memory 2-112.

[0098] Understandably, since M is less than the number of first data blocks in the first data block matrix, storing M causal mask blocks reduces the space occupied by the causal mask. Therefore, the generated M causal mask blocks can be stored in static random access memory 2-112; thus, the number of accesses to dynamic random access memory 2-113 is reduced. Since the access speed of static random access memory 2-112 is greater than that of dynamic random access memory 2-113, the access speed of the M causal mask blocks can be improved.

[0099] In this embodiment of the application, the coprocessor 2-114 is further configured to control the loading storage unit 2-115 to load the second data block and the third data block from the dynamic random access memory 2-113 into the static random access memory 2-112; and control the computing engine 2-116 to calculate the correlation score between the second data block and the third data block to obtain the first data block.

[0100] It should be noted that the second data block is obtained by partitioning the query matrix into data blocks based on the first dimension of the preset computational granularity. In other words, the query matrix is ​​partitioned into data blocks based on the first dimension of the preset computational granularity to obtain the second data block matrix. The first dimension of granularity refers to the amount of data included in a column of a data block in the query matrix. Similarly, the third data block is obtained by partitioning the key matrix into data blocks based on the second dimension of the preset computational granularity. In other words, the key matrix is ​​partitioned into data blocks based on the second dimension of the preset computational granularity to obtain the third data block matrix. The second dimension of granularity is the amount of data included in a column of a data block in the key matrix. The second and third data block matrices are stored in dynamic random access memory 2-113. Since the access speed of static random access memory 2-112 is greater than that of dynamic random access memory 2-113, when calculating the correlation score in two loops based on the first and second quantities, the coprocessor 2-114 controls the loading memory unit 2-115 to load the i-th second data block and the j-th third data block from dynamic random access memory 2-113 to static random access memory 2-112 before calculating the correlation score between the i-th second data block and the j-th third data block.

[0101] It should be noted that the preset computational granularity refers to the granularity of the first data block in the attention mechanism, such as [256, 128], [128, 128], etc. The preset computational granularity includes a first-dimensional granularity and a second-dimensional granularity. The first-dimensional granularity represents the number of rows in the first data block, i.e., the amount of data in each column containing the first-dimensional granularity. The second-dimensional granularity represents the number of columns in the first data block, i.e., the amount of data in each row containing the second-dimensional granularity. Additionally, the first-dimensional granularity is also used to represent the amount of data in each column of the second data block divided by the query matrix. Therefore, the result of dividing the query matrix into data blocks based on the first-dimensional granularity is the second data block matrix. The number of second data blocks included in the second data block matrix is ​​the ratio of the sequence length corresponding to the query matrix to the first-dimensional granularity. Similarly, the second dimension granularity is also used to represent the data volume of each column in the third data block divided by the key matrix. Therefore, the partitioning result obtained by dividing the key matrix into data blocks based on the second dimension granularity is the third data block matrix. The number of third data blocks included in the third data block matrix is ​​the ratio of the sequence length corresponding to the key matrix to the second dimension granularity. It is easy to see that the first dimension granularity and the second dimension granularity can be equal or unequal; this embodiment does not limit this. Here, the calculation engine 2-116 can multiply the second data block and the third data block to calculate the correlation score, and the obtained calculation result is the first data block; therefore, the first data block matrix is ​​obtained by multiplying the second data block matrix and the third data block matrix.

[0102] For example, see Figure 5 , Figure 5 This is an exemplary schematic diagram of calculating a relevance score provided in an embodiment of this application; as shown... Figure 5 As shown, the second data block matrix 5-1 is obtained by partitioning the query matrix based on the first dimension granularity, the third data block matrix 5-2 is obtained by partitioning the key matrix based on the second dimension granularity, and the first data block matrix 5-3 is obtained by calculating the relevance score between the second data block matrix 5-1 and the third data block matrix 5-2; wherein, the first data block 5-31 is calculated from the second data block 5-11 and the third data block 5-21.

[0103] Understandably, by using a computational model that employs data blocks, it becomes possible to replace the causal mask matrix with M causal mask matrices, thereby reducing the storage space required for the causal mask's related data.

[0104] In this embodiment of the application, the correspondence between the M causal mask block identifiers and the M mask data distributions can be created by the central processing unit 4-12, and the central processing unit 4-12 stores the correspondence in the dynamic random access memory 2-113. The coprocessor 2-114 controls the loading storage unit 2-115 to load the correspondence from the dynamic random access memory 2-113 to the static random access memory 2-112.

[0105] In this embodiment, the coprocessor 2-114 is further configured to control the loading storage unit 2-115 to load the fifth data block from the dynamic random access memory 2-113 into the static random access memory 2-112; control the computing engine 2-116 to perform calculations on the fourth and fifth data blocks to obtain a feature data block; and control the loading storage unit 2-115 to store the feature data block into the dynamic random access memory 2-113, so as to perform prediction based on the feature data block in the dynamic random access memory 2-113 and obtain an artificial intelligence prediction result.

[0106] It should be noted that the fifth data block is a data block within the value matrix (V matrix), and it is the data block in the value matrix that needs to be computed in correspondence with the fourth data block. The feature data block refers to the computational result of the fourth and fifth data blocks, and it is also the data block that completes the attention mechanism based on causal masking. The artificial intelligence prediction result refers to the output of the relevant task predicted based on the feature data block, such as the generated text.

[0107] It is understandable that by using M causal mask matrices to implement a causal mask-based attention mechanism, and M causal mask matrices reduce the SRAM space usage and improve memory access efficiency, the resource consumption of the attention mechanism algorithm can be reduced, thus reducing the resource requirements of artificial intelligence prediction.

[0108] In the embodiments of this application, the descriptions of the top left corner, bottom left corner, bottom right corner, top right corner, row, and column are all determined based on the orientation of the screen when the information is presented.

[0109] See Figure 6 , Figure 6 This is a schematic flowchart of the data processing method for the chip provided in the embodiments of this application; the following will be combined with Figure 6 The steps shown are explained.

[0110] Step 101: Microcontroller unit 2-111 sends a mask instruction to coprocessor 2-114 for the first data block in static random access memory 2-112.

[0111] It should be noted that the first data block is any data block in the relevance score matrix of the query matrix and the key matrix.

[0112] In this embodiment of the application, the coprocessor 2-114 executes steps 102 and 103 in response to the mask instruction.

[0113] Step 102: Coprocessor 2-114 determines the target causal mask block from M causal mask blocks pre-stored in static random access memory 2-112 based on the first index and the second index of the first data block.

[0114] It should be noted that the first index is the index of the second data block used to calculate the first data block in the query matrix, and the second index is the index of the third data block used to calculate the first data block in the key matrix. The M causal mask blocks correspond to the M mask data distributions, and the mask data distributions are the data distributions of the first data block that have been changed by the mask. M is a positive integer.

[0115] Step 103: The coprocessor 2-114 controls the computing engine 2-116 to mask the first data block based on the target causal mask block to obtain the fourth data block.

[0116] It should be noted that the implementation process of steps 101 to 103 is the same as the implementation process of the microcontroller unit 2-111, coprocessor 2-114 and computing engine 2-116 in the chip provided in this application embodiment to mask the first data block, and will not be described again in this application embodiment.

[0117] In step 102 of this embodiment, the coprocessor 2-114 determines a target causal mask block from M causal mask blocks pre-stored in static random access memory 2-112 based on a first index and a second index of the first data block. This includes: the coprocessor 2-114 determining a target mask data distribution whose first data block is masked based on the first index and the second index of the first data block, wherein the M mask data distributions include the target mask data distribution; and determining a target causal mask block identifier corresponding to the target mask data distribution based on the correspondence between the M causal mask block identifiers and the M mask data distributions pre-stored in static random access memory 2-112; and finally, determining the target causal mask block corresponding to the target causal mask block identifier from the M causal mask blocks pre-stored in static random access memory 2-112.

[0118] In this embodiment of the application, before step 102, the process of determining M causal mask blocks is further included; that is, before the coprocessor 2-114 determines the target causal mask block from the M causal mask blocks pre-stored in the static random access memory 2-112 based on the first index and the second index of the first data block, the data processing method further includes: the central processing unit 4-12 first determines the distribution of M mask data that the first data block is masked based on the first number of the second data blocks of the query matrix and the second number of the third data blocks of the key matrix; and determines the M causal mask blocks corresponding to the M mask data distributions; finally, the M causal mask blocks are stored in the static random access memory 2-112.

[0119] In this embodiment of the application, the central processing unit 4-12 determines the M mask data distributions of the first data block that are masked based on the first number of the second data blocks of the query matrix and the second number of the third data blocks of the key matrix. This includes: the central processing unit 4-12 first determines the multiple between the first number of the second data blocks of the query matrix and the second number of the third data blocks of the key matrix; and determines the mask data distribution of the multiple; finally, based on the data distribution of the first data block that is fully masked and the data distribution that is not masked, as well as the mask data distribution of the multiple, the M mask data distributions of the first data block that are masked are obtained.

[0120] In this embodiment of the application, the central processing unit 4-12 stores M causal mask blocks into static random access memory 2-112, including: the central processing unit 4-12 stores the M causal mask blocks into dynamic random access memory 2-113; and controls the loading storage unit 2-115 through the coprocessor 2-114 to store the M causal mask blocks from dynamic random access memory 2-113 into static random access memory 2-112.

[0121] In this embodiment of the application, before the microcontroller unit 2-111 sends a mask instruction for the first data block in the static random access memory 2-112 to the coprocessor 2-114, the data processing method further includes: the coprocessor 2-114 first controls the loading storage unit 2-115 to load the second data block and the third data block from the dynamic random access memory 2-113 into the static random access memory 2-112, wherein the second data block is obtained by dividing the query matrix into data blocks based on the first dimension granularity in the preset calculation granularity, and the third data block is obtained by dividing the key matrix into data blocks based on the second dimension granularity in the preset calculation granularity; and controls the calculation engine 2-116 to calculate the correlation score between the second data block and the third data block to obtain the first data block.

[0122] In this embodiment, after the coprocessor 2-114 controls the computing engine 2-116 to mask the first data block based on the target causal mask block to obtain the fourth data block, the data processing method further includes: the coprocessor 2-114 first controls the loading storage unit 2-115 to load the fifth data block from the dynamic random access memory 2-113 into the static random access memory 2-112, wherein the fifth data block is a data block in the value matrix; then controls the computing engine 2-116 to perform calculations on the fourth data block and the fifth data block to obtain the feature data block; finally, controls the loading storage unit 2-115 to store the feature data block into the dynamic random access memory 2-113, so as to perform prediction based on the feature data block in the dynamic random access memory 2-113 to obtain the artificial intelligence prediction result.

[0123] The following describes an exemplary application of the embodiments of this application in a real-world application scenario. This exemplary application describes the process of causal masking in an attention mechanism operator.

[0124] For example, see Figure 7 , Figure 7 This is an exemplary chip schematic diagram provided in an embodiment of this application; as shown... Figure 7 As shown, the computing unit (Die) 7-1 (also called the data processing unit 2-11) is a chip that accelerates computation using Single Instruction Multiple Data (SIMD). It includes a microcontroller unit 7-10, static random access memory (SRAM) 7-11, dynamic random access memory (DRAM) 7-12, a coprocessor 7-14, a load memory unit 7-15, and a computing engine 7-16 (including a matrix multiplication module (GEMM) and a vector operation module (VECTOR)). In the computing unit 7-1, the load memory unit 7-15 moves data from DRAM 7-12 to SRAM 7-11. The computing engine 7-16 retrieves data from SRAM 7-11, performs calculations based on the retrieved data, and writes the calculation results back to SRAM 7-11. Then, the load memory unit 7-15 moves the calculation results from SRAM 7-11 to DRAM 7-12.

[0125] In the attention mechanism of this application embodiment, the correlation score matrices of the Q matrix and K matrix are masked. Due to the space limitations of SRAM in the computing unit, the attention mechanism is implemented at the data block level. Here, the attention mechanism algorithm (Flash Attention) is used as an example for explanation.

[0126] During the forward propagation of the attention mechanism algorithm, the computational granularity is [256, 128]. If the size of the relevance score matrix is ​​[1024, 1024], then see [link to relevant algorithm]. Figure 8 , Figure 8 This is a schematic diagram of an exemplary data block matrix for correlation scores provided in an embodiment of this application. Figure 1 ;like Figure 8 As shown, the data block matrix 8-1 of the correlation score matrix obtained based on the computational granularity [256, 128] (referred to as the preset computational granularity, 256 as the first dimension granularity, and 128 as the second dimension granularity) includes 4 rows (rows 0 to 3) and 8 columns (columns 0 to 7) of data blocks; when performing causal masking on the data block matrix 8-1, there are four types of data blocks (referred to as M types of masked data layouts), one of which is a data block in which all data remains unchanged after masking (see...). Figure 8 One type is a data block filled with diagonal lines, where the data is changed by a quarter after masking (see...). Figure 8 One type is a data block filled with square grids, where three-quarters of the data is changed after masking (see [link]). Figure 8 Another type is a data block filled with horizontal lines, and another type is a data block whose data is completely changed after masking (see...). Figure 8 (Data blocks filled with vertical lines).

[0127] It should be noted that, Figure 8 The data block matrix 8-1 in the diagram is obtained by dividing the Q matrix into... ( (referred to as the first quantity) data blocks, dividing the K matrix into ( (This is called the second quantity) data blocks are used for a two-dimensional loop (an outer loop based on i, and an inner loop based on j, where i is less than...) An integer variable, j is less than It is calculated from an integer variable.

[0128] based on Figure 8 The four data blocks in the matrix are used to generate four causal mask blocks (called M causal mask blocks). Thus, the space occupied by the four causal mask blocks is only 1 / 8 of the space occupied by the causal mask corresponding one-to-one with the data blocks of the correlation score matrix, reducing the storage space of the causal mask blocks. See also Figure 9 , Figure 9 This is an exemplary causal mask block illustration provided in the embodiments of this application. Figure 1 ;like Figure 9 As shown, corresponding to Figure 8 The four data blocks generated four causal mask blocks, namely causal mask blocks 9-1 to causal mask block 9-4; in these blocks, the data in the dot-filled portion is 0, and the data in the square-filled portion is negative infinity; and causal mask block 9-1 is used for... Figure 8 The data blocks filled with square grids are masked, and the causal mask block 9-2 is used to mask the data. Figure 8 The data block filled with horizontal lines is masked, and the causal mask block 9-3 is used to mask the data. Figure 8 The data block filled with vertical lines is masked, and the causal mask block 9-4 is used to mask the data. Figure 8 Mask the data blocks filled with diagonal lines.

[0129] In the embodiments of this application, when generating causal mask blocks, if the data processing instruction carries mask parameters (such as sequence length and computation granularity), the corresponding function is called based on the mask parameters to generate various causal mask blocks, and the various causal mask blocks are stored in a temporary workspace and loaded into SRAM from the workspace, thereby reducing the number of DRAM accesses.

[0130] It should be noted that the i and j (called the first and second indices) of each data block in the data block matrix 8-1 (called the correlation score matrix) indicate that the data block is obtained by multiplying the i-th data block in the Q matrix and the j-th data block in the K matrix. Based on the i and j of each data block in the data block matrix 8-1, the position of the data block in the data block matrix 8-1 can be determined, and thus the data distribution that is changed after masking can be determined. Based on the changed data distribution after masking, a causal mask block for masking the data block can be determined. Therefore, based on Figure 8 and Figure 9 The data block masked using causal masking block 9-1 is Data blocks, Data blocks, Data blocks and The data block; the data block masked using causal masking block 9-2 is Data blocks, Data blocks, Data blocks and The data block; the data block masked using causal masking block 9-4 is The data block; the data block masked using causal masking block 9-3 is Data blocks.

[0131] Accordingly, based on the i and j of the data block and the computational granularity [256, 128], the top-left index of the data block can be determined as (256*i, 128*j), the bottom-left index as (256*(i+1), 128*j), the top-right index as (256*i, 128*(j+1)), and the top-right index as (256*(i+1), 128*(j+1)). Therefore, when 256*(i+1) is less than or equal to 128*j, causal mask block 9-3 is used; when 256*i is greater than or equal to 128*(j+1), causal mask block 9-4 is used; when 256*(i+1) is equal to 128*(j+1), causal mask block 9-2 is used; and when 256*i is equal to 128*j, causal mask block 9-1 is used.

[0132] Similarly, in the backpropagation process of the attention mechanism algorithm, the computational granularity is [128, 128]. If the size of the relevance score matrix is ​​[1024, 1024], then see... Figure 10 , Figure 10 This is a schematic diagram of an exemplary data block matrix for correlation scores provided in an embodiment of this application. Figure 2 ;like Figure 10 As shown, the data block matrix 10-1 of the correlation score matrix obtained based on the computational granularity [128, 128] includes 8 rows (rows 0 to 7) and 8 columns (columns 0 to 7) of data blocks; when masking the data block matrix 10-1, there are three types of data blocks, one of which is a data block whose data remains unchanged after masking (see...). Figure 10 One type is a data block filled with diagonal lines, where half of the data is changed after masking (see [link]). Figure 10 Another type is a data block filled with square grids, and another type is a data block whose data is completely changed after masking (see...). Figure 10 (Data blocks filled with vertical lines).

[0133] It should be noted that, Figure 10 The data block matrix 10-1 in the middle is obtained by dividing the Q matrix into ( ) data blocks, dividing the K matrix into ( The data was calculated using a two-dimensional loop from 10 data blocks.

[0134] based on Figure 10 The three data blocks generate three causal mask blocks; see [link to relevant documentation]. Figure 11 , Figure 11 This is an exemplary causal mask block illustration provided in the embodiments of this application. Figure 2 ;like Figure 11 As shown, corresponding to Figure 10The three data blocks generated three causal mask blocks, namely causal mask blocks 11-1 to causal mask blocks 11-3; in these blocks, the data in the dot-filled portion is 0, and the data in the square-filled portion is negative infinity; and causal mask block 11-1 is used for... Figure 10 Causal masking is performed on the data blocks filled with square grids. Causal masking block 11-2 is used to... Figure 10 Causal masking is performed on the data block filled with vertical lines. Causal masking block 11-3 is used to... Figure 10 Causal masking is performed on data blocks filled with diagonal lines.

[0135] It should be noted that the values ​​i and j of each data block in the data block matrix 10-1 indicate that the data block is obtained by multiplying the i-th data block of the Q matrix and the j-th data block of the K matrix. Based on the values ​​i and j of each data block in the data block matrix 10-1, the position of the data block in the matrix can be determined. This allows us to determine the altered data distribution after masking, and based on the altered data distribution, to determine the causal masking block for that data block. Therefore, based on... Figure 10 and Figure 11 The data block masked using causal masking block 11-1 is: Data blocks, Data blocks, Data blocks, Data blocks, Data blocks, Data blocks, Data blocks and The data block; the data block masked using causal masking block 11-2 is The data block; the data block masked using causal masking block 11-3 is Data blocks.

[0136] Accordingly, based on the i and j of the data block and the computational granularity [128, 128], the top-left index of the data block can be determined as (128*i, 128*j), the bottom-left index as (128*(i+1), 128*j), and the top-right index as (128*i, 128*(j+1)). Therefore, when 128*(i+1) is less than or equal to 128*j, causal mask block 11-2 is used; when 128*i is greater than or equal to 128*(j+1), causal mask block 11-3 is used; and when 128*i is equal to 128*j, causal mask block 11-1 is used.

[0137] It is understood that the embodiments of this application are based on the chip's own design architecture characteristics, including the layout of its arithmetic units, the structure of its storage hierarchy, the data interaction method, and the characteristics of its instruction set. Based on these underlying characteristics, the attention mechanism operator is specifically adapted and modified. By optimizing the computational logic of the attention mechanism operator to improve the computational efficiency of the chip's arithmetic units, adjusting the data access mode to match the storage hierarchy, and refactoring some code in conjunction with the chip's instruction set, the attention mechanism operator on this chip platform can ultimately maintain consistency with the expected function while also unlocking the chip's hardware potential, improving the chip's computational performance, and providing stable underlying support for the operation of upper-layer applications.

[0138] This application provides a computer program product, which includes computer-executable instructions or a computer program stored in a computer-readable storage medium. An electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the chip data processing method described above in this application.

[0139] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed, they will cause the execution of the data processing method of the chip provided in this application embodiment. For example, ... Figure 6 The data processing method of the chip is shown.

[0140] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0141] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0142] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0143] As an example, computer-executable instructions may be deployed to execute on a single electronic device (in which case, this single electronic device is an electronic device including the chip provided in the embodiments of this application), or to execute on multiple electronic devices located at one location (in which case, the multiple electronic devices located at one location are electronic devices including the chip provided in the embodiments of this application), or to execute on multiple electronic devices distributed across multiple locations and interconnected via a communication network (in which case, the multiple electronic devices distributed across multiple locations and interconnected via a communication network are electronic devices including the chip provided in the embodiments of this application).

[0144] It is understood that in the embodiments of this application, data related to query matrices, key matrices and causal mask blocks are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required. The collection, use and processing of related data must comply with relevant laws, regulations and standards, and conform to the principles of legality, legitimacy and necessity. It does not involve obtaining data types prohibited or restricted by laws and regulations, and will not hinder the normal operation of the target website.

[0145] In summary, this embodiment of the application, when the microcontroller unit masks any first data block in the relevance score matrix of the query matrix and the key matrix, controls the coprocessor to select a target causal mask block corresponding to the first data block from M causal mask blocks in static random access memory to mask the first data block. Since the M causal mask blocks are determined based on the M mask data distributions that the first data block changes during the masking process, and the mask data distribution is the data distribution of the first data block that has been changed by the masking, first data blocks with the same data distribution that has been changed by the masking can reuse the same causal mask block, reducing the number of causal mask blocks. Furthermore, by selecting a target causal mask block from the M causal mask blocks to mask the first data block in the relevance score matrix, the number of causal mask blocks can be reduced, the space occupied by the causal mask blocks can be reduced, and the resource consumption of the mask can be reduced.

[0146] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A chip, characterized in that, The chip includes static random access memory, a microcontroller unit, a coprocessor, and a computing engine, and the chip also includes a central processing unit; The static random access memory is used to store a first data block, wherein the first data block is any data block in the relevance score matrix of the query matrix and the key matrix; it is also used to store M causal mask blocks corresponding to M mask data distributions, wherein the mask data distributions are the data distributions of the first data block that have been masked, and M is a positive integer; The microcontroller unit is used to send a mask instruction for the first data block to the coprocessor; The coprocessor is configured to, in response to the masking instruction, determine a target causal masking block from the M causal masking blocks based on a first index and a second index of the first data block, wherein the first index is the index of the second data block in the query matrix used to calculate the first data block, and the second index is the index of the third data block in the key matrix used to calculate the first data block; The computing engine is used to mask the first data block based on the target causal mask block to obtain a fourth data block; The central processing unit is configured to, when it obtains a causal mask for attention mechanism processing, mark the causal mask as disabled and regenerate the M causal mask blocks.

2. The chip according to claim 1, characterized in that, The static random access memory is also used to store the correspondence between the M causal mask block identifiers and the distribution of the M mask data; The coprocessor is further configured to: determine the target mask data distribution of the first data block being masked based on the first index and the second index of the first data block, wherein the M mask data distributions include the target mask data distribution; determine the target causal mask block identifier corresponding to the target mask data distribution based on the correspondence between the M causal mask block identifiers and the M mask data distributions; and determine the target causal mask block corresponding to the target causal mask block identifier from the M causal mask blocks.

3. The chip according to claim 1 or 2, characterized in that, The central processing unit is further configured to determine the M mask data distributions of the first data block being masked based on the first number of the second data blocks in the query matrix and the second number of the third data blocks in the key matrix; determine the M causal mask blocks corresponding to the M mask data distributions; and store the M causal mask blocks in the static random access memory.

4. The chip according to claim 3, characterized in that, The central processing unit is further configured to determine the quantity multiple between the first quantity and the second quantity; determine the mask data distribution of the quantity multiple; and obtain the M mask data distributions of the first data block that are masked based on the data distribution of the first data block that is fully masked and the data distribution that is not masked, as well as the mask data distribution of the quantity multiple.

5. The chip according to claim 3, characterized in that, The chip also includes a dynamic random access memory and a load memory unit; The central processing unit is also used to store the M causal mask blocks into the dynamic random access memory; The loading storage unit is used to load the M causal mask blocks from the dynamic random access memory to the static random access memory.

6. The chip according to claim 1 or 2, characterized in that, The dynamic random access memory in the chip is used to store the second data block and the third data block, wherein the second data block is obtained by dividing the query matrix into data blocks based on the first dimension granularity in the preset calculation granularity, and the third data block is obtained by dividing the key matrix into data blocks based on the second dimension granularity in the preset calculation granularity. The coprocessor is also configured to control the loading memory unit in the chip to load the second data block and the third data block from the dynamic random access memory into the static random access memory; The computing engine is also used to calculate the correlation score between the second data block and the third data block in the static random access memory to obtain the first data block; The coprocessor is also configured to control the loading storage unit to store the first data block into the static random access memory.

7. A data processing method for a chip, characterized in that, The method includes: The microcontroller unit sends a mask instruction to the coprocessor for a first data block in the static random access memory, wherein the first data block is any data block in the relevance score matrix of the query matrix and the key matrix; The coprocessor performs the following processing in response to the mask instruction: Based on the first and second indices of the first data block, a target causal mask block is determined from M causal mask blocks pre-stored in the static random access memory. The first index is the index of the second data block in the query matrix used to calculate the first data block, and the second index is the index of the third data block in the key matrix used to calculate the first data block. The M causal mask blocks correspond to M mask data distributions, where the mask data distribution is the data distribution of the first data block that has been changed by the mask, and M is a positive integer. Furthermore, the M causal mask blocks are generated when the central processing unit obtains a causal mask used for attention mechanism processing, marks the causal mask as disabled, and regenerates it. The control computing engine masks the first data block based on the target causal mask block to obtain the fourth data block.

8. An electronic device, characterized in that, The electronic device includes the chip according to any one of claims 1 to 6.

9. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed, the data processing method of the chip described in claim 7 is implemented.

10. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed, the data processing method of the chip described in claim 7 is implemented.

Images