Method, computing device and medium for generating input mask tensor

By generating bit-type input mask tensors on the device side, the problem of long processing time in traditional methods is solved, thus improving model training performance and efficiency.

CN118567610BActive Publication Date: 2024-11-08北京壁仞科技开发有限公司 +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411052381.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-08
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

In traditional solutions, the process of generating the input mask tensor on the host side and transmitting it to the device side takes a long time, resulting in poor model training performance.

Method used

The input mask tensor is generated directly on the device side, omitting the transmission process between the host and the device side. The mask tensor data is converted into bit data to reduce bandwidth pressure, and the mask tensor is generated in parallel using the computing unit on the device side.

Benefits of technology

It improves model training performance, reduces transmission time, and lowers computational bandwidth pressure.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118567610B_ABST
    Figure CN118567610B_ABST
Patent Text Reader

Abstract

Embodiments of the present invention relate to a method, a computing device, a storage medium, and a computer program product for generating an input mask tensor. The method is executed at a device end and includes: determining the tensor dimension of the input mask tensor and data for filling the input mask tensor based at least on the input tensor; and generating the input mask tensor based on the tensor dimension of the input mask tensor and the data for filling the input mask tensor by at least one computing unit in the device end, wherein the data in the generated input mask tensor is bit-type data. Embodiments of the present invention can directly generate an input mask tensor for a masked_fill function at the device end, omitting the transmission process of the input mask tensor between the host end and the device end, saving time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention generally relate to the field of data computing, and more specifically to a method, a computing device, a storage medium, and a computer program product for generating an input mask tensor. Background Art

[0002] In the process of training Transformer models (such as language models and text-based graph models), based on the attention mechanism, data calculations based on the masked_fill function are required multiple times. The traditional input mask tensor for the masked_fill function is usually generated on the host side and transmitted to the device side, and then the device side performs related operations based on the received input mask tensor.

[0003] However, in the above solution, the host side can only generate the input mask tensor in a point-to-point manner, which is time-consuming, and it also takes extra time for the host side to transmit the input mask tensor generated on it to the device side.

[0004] In summary, the traditional solution for generating input mask tensors for the masked_fill function has the following disadvantages: it can only be performed on the host side and is time-consuming. Summary of the invention

[0005] In response to the above problems, the present invention provides a method, a computing device, a storage medium and a computer program product for generating an input mask tensor, so that the input mask tensor for the masked_fill function can be directly generated at the device end, omitting the transmission process of the input mask tensor between the host end and the device end, thereby saving time.

[0006] According to a first aspect of the present invention, a method for generating an input mask tensor is provided, characterized in that it includes: determining the tensor dimension of the input mask tensor and the data used to fill the input mask tensor based at least on the input tensor; and generating the input mask tensor based on the tensor dimension of the input mask tensor and the data used to fill the input mask tensor by at least one computing unit in the device end, wherein the data in the input mask tensor is bit-type data, and the provided method is executed at the device end.

[0007] In some embodiments, determining the tensor dimension of the input mask tensor and the data used to fill the input mask tensor based at least on the input tensor includes: determining the tensor dimension of the original mask tensor based on the input tensor, wherein the data of the original mask tensor is Boolean data; and determining the tensor dimension of the input mask tensor based on the tensor dimension of the original mask tensor and a predetermined conversion rule.

[0008] In some embodiments, the data of the input tensor is Boolean data, and the tensor dimension of the input tensor is the same as the tensor dimension of the original mask tensor.

[0009] In some embodiments, the predetermined conversion rule is: a conversion rule for converting the tensor dimension of a Boolean tensor into the tensor dimension of a bit tensor.

[0010] In some embodiments, determining the tensor dimension of the input mask tensor and the data used to fill the input mask tensor based at least on the input tensor includes: determining a sequence length and a mask type corresponding to the input tensor based on the input tensor; and determining the data used to fill the input mask tensor based on the sequence length and the mask type.

[0011] In some embodiments, the mask types include lower triangle and square.

[0012] In some embodiments, determining data used to fill the input mask tensor includes: determining the column index and row index of each data in the input mask tensor; for each data, comparing the column index with the row index; and based on the comparison result, determining the bit value corresponding to each data to facilitate determining the value of the data used to fill the input mask tensor.

[0013] In some embodiments, determining the bit value corresponding to each data based on the comparison result includes: in response to the comparison result satisfying a predetermined condition, determining that the bit value corresponding to the current data is 1; and in response to the comparison result not satisfying the predetermined condition, determining that the bit value corresponding to the current data is 0, wherein the predetermined condition includes at least: the row index of the current data is greater than or equal to the column index of the current data.

[0014] In some embodiments, each computing unit in at least one computing unit includes multiple threads; and generating an input mask tensor based on the tensor dimension of the input mask tensor and the data used to fill the input mask tensor includes: determining the starting column index and the starting row index of the input mask tensor; matching the starting column index and the starting row index with the computing unit index and the thread index to obtain a correspondence between the data processed by each thread of each computing unit and the data in the input mask tensor; and generating the input mask tensor based on the obtained correspondence and the data used to fill the input mask tensor.

[0015] In some embodiments, generating an input mask tensor also includes: determining a dimensional range of the input mask tensor that can be processed each time based on at least the number of at least one computing unit, the number of threads included in each computing unit, and the size of the registers corresponding to each thread; in response to the dimensional range of the input mask tensor that can be processed each time being different from the tensor dimension of the input mask tensor, dividing the input mask tensor based on the dimensional range of the input mask tensor that can be processed each time to obtain a plurality of input mask tensor partitions; and for each input mask tensor partition, generating an input mask tensor portion corresponding to the input mask tensor partition based on data used to fill the input mask tensor, so as to generate the input mask tensor.

[0016] According to a second aspect of the present invention, a computing device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute the method of the first aspect of the present invention.

[0017] According to a third aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method of the first aspect of the present invention.

[0018] According to a fourth aspect of the present invention, there is provided a computer program product, wherein the computer program product is tangibly stored on a non-transitory computer readable medium and comprises machine executable instructions which, when executed, cause a machine to perform the steps in the method of the first aspect of the present invention.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements.

[0021] Figure 1 A schematic structural diagram of a heterogeneous computing system according to an embodiment of the present invention is shown.

[0022] Figure 2 A schematic diagram of a system for generating an input mask tensor according to an embodiment of the present invention is shown.

[0023] Figure 3 A schematic diagram of a method for generating an input mask tensor according to an embodiment of the present invention is shown.

[0024] Figure 4A A schematic diagram of an exemplary input mask tensor to be generated according to an embodiment of the present invention is shown.

[0025] Figure 4B A schematic diagram of the layout of a computing unit in a device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0027] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0028] In the process of training Transformer models (such as language models and text graph models), based on the attention mechanism, the masked_fill function needs to be used multiple times to replace at least some elements in the input tensor. Generally speaking, the input parameters of the masked_fill function include at least: input tensor, input mask tensor. Among them, the input mask tensor is usually a Boolean tensor with the same dimension as the input tensor, which is used to indicate the position of the element to be replaced in the input tensor.

[0029] Typically, for systems with heterogeneous structures, the input mask tensor is generated on the host side and then transmitted from the host side to the device side for corresponding operations on the device side. Figure 1 The heterogeneous structure shown is taken as an example and described in detail.

[0030] Figure 1 A schematic structural diagram of a heterogeneous computing system 100 according to an embodiment of the present invention is shown.

[0031] like Figure 1As shown, the heterogeneous computing system 100 includes a host end 110 (e.g., a central processing unit (CPU)) and at least one device end 130 (a processor such as a graphics processing unit (GPU) or a general-purpose graphics processing unit (GPGPU)), wherein the device end 130 is configured with at least a computing unit 132 and a register 134.

[0032] As described above, in the conventional scheme, an input mask tensor for the masked_fill function can be generated on the host side 110, and then the generated input mask tensor is transmitted to the device side 130 through methods such as host2device (host side to device side), and the computing unit 132 at the device side 130 loads the received input mask tensor to the corresponding register 134, so as to perform operations related to the masked_fill function on the device side 130. However, in the above scheme, since the host side can only generate the input mask tensor in a point-to-point manner, it takes a long time, and the host side also needs additional time to transmit the input mask tensor generated on it to the device side. In addition, at the device side, it takes a certain amount of time and bandwidth to load the received input mask tensor into the register, and the performance of this receiving and loading process accounts for a large proportion, which puts a certain burden on the model training performance. In summary, the shortcomings of the conventional scheme for generating an input mask tensor for the masked_fill function are: it takes a long time and the model training performance is poor.

[0033] In order to at least partially solve one or more of the above problems and other potential problems, an exemplary embodiment of the present invention proposes a scheme for generating an input mask tensor. In this scheme, the tensor dimension of the input mask tensor and the data used to fill the input mask tensor are determined based at least on the input tensor; and the input mask tensor is generated by at least one computing unit in the device side based on the tensor dimension of the input mask tensor and the data used to fill the input mask tensor, so that the input mask tensor for the masked_fill function can be directly generated at the device side, omitting the transmission process of the input mask tensor between the host side and the device side, saving time. In addition, in the above scheme of the present invention, the data in the generated input mask tensor is made bit-type data, so as to reduce the bandwidth pressure in the subsequent function calculation process and improve the model training performance.

[0034] The following will be combined Figures 2 to 4B A scheme for generating an input mask tensor on a device side according to an embodiment of the present invention is described in detail.

[0035] Figure 2A schematic diagram of a system 200 for generating an input mask tensor according to an embodiment of the present invention is shown. It should be understood that the system 100 may further include additional modules not shown and / or may omit the modules shown, and the scope of the present invention is not limited in this respect.

[0036] like Figure 2 As shown, the system 200 includes: an input mask tensor determination module 220 and an input mask tensor generation module 240 .

[0037] Regarding the input mask tensor determination module 220 , it may be configured to determine the tensor dimension of the input mask tensor and data for filling the input mask tensor based at least on the input tensor.

[0038] Regarding the input mask tensor generation module 240, it may be configured to generate an input mask tensor based on the tensor dimension of the input mask tensor and the data used to fill the input mask tensor. According to an embodiment of the present invention, the data in the input mask tensor is bit-type data.

[0039] Figure 3 A schematic diagram of a method 300 for generating an input mask tensor according to an embodiment of the present invention is shown. It should be understood that the method 300 may further include additional steps not shown and / or may omit the steps shown, and the scope of the present invention is not limited in this respect.

[0040] In step 302 , at the device end, based at least on the input tensor, a tensor dimension of an input mask tensor and data for filling the input mask tensor are determined.

[0041] Regarding the input tensor, it can be any tensor suitable for large model training. According to an embodiment of the present invention, the input tensor can be a floating point tensor or an integer tensor commonly used for large model training, that is, the type of data in the input tensor can be a floating point type (such as fp16, fp32, bf16) or an integer type (such as int16, int32).

[0042] As mentioned above, the traditional mask tensor is usually a Boolean tensor with the same tensor dimension as the input tensor. According to the inventive concept of the present invention, in order to reduce the bandwidth pressure in the calculation process of the masked_fill function and improve the model training performance, the input mask tensor to be generated can be a bit-type tensor, that is, the data in the input mask tensor needs to be bit-type data, and the tensor dimension and filling data of the bit-type input mask tensor are determined based on the input tensor. Correspondingly, the filling data here is bit data.

[0043] Specifically, according to an embodiment of the present invention, determining the tensor dimension of an input mask tensor based on an input tensor may include: determining the tensor dimension of an original mask tensor based on the input tensor, wherein the data of the original mask tensor is Boolean data; and determining the tensor dimension of the input mask tensor based on the tensor dimension of the original mask tensor and a predetermined conversion rule.

[0044] Regarding the original mask tensor, it can be a Boolean tensor, and the tensor dimension of the original mask tensor is the same as the tensor dimension of the input tensor.

[0045] According to an embodiment of the present invention, the tensor dimension of a tensor can be expressed in the form of (N, H, W) commonly used in the attention mechanism, for example, where N represents the batch size (batch_size), H represents the height (height), and W represents the width (width). On this basis, the tensor dimension of the input tensor can be expressed as (N_input, H_input, W_input). Similarly, the tensor dimension of the original mask tensor can be expressed as (N_original, H_original, W_original), where N_original = N_input, H_original = H_input, and W_original = W_input.

[0046] Regarding the predetermined conversion rule, it may be a rule for determining the tensor dimension of the corresponding bit-type tensor based on the tensor dimension of the Boolean tensor.

[0047] For example, in an embodiment of the present invention, the tensor dimension of the input mask tensor can be expressed as (N_transferred, H_transferred, W_transferred). According to the inventive concept of the present invention, for the expected input mask tensor to be an fp32 type tensor, the tensor dimension of the input mask tensor and the tensor dimension of the original mask tensor have the following corresponding relationship:

[0048] N_transferred = N_original;

[0049] H_transferred = (H_original + 1) / / 2;

[0050] W_transferred = (W_original + 511) / / 512 × 32.

[0051] Here, “ / / ” is the integer division operator, which is used to obtain the integer part of the division operation.

[0052] Taking the input tensor with a tensor dimension of (1, 2, 512) as an example, it can be determined that the tensor dimension of the original mask tensor of the input tensor is (1, 2, 512), and further based on the above predetermined conversion rule, it can be determined that the tensor dimension of the input mask tensor is (1, 1, 32). It can be seen that compared with the original mask tensor, the input mask tensor is compressed by 2 times in the height dimension and 16 times in the width dimension. In other words, the tensor dimension of the input mask tensor in the scheme of the present invention can be significantly smaller than the tensor dimension of the mask tensor used in the traditional scheme. Therefore, performing operations such as masked_fill function based on the above input mask tensor can significantly reduce the bandwidth pressure in the calculation process and improve the model training performance.

[0053] It should be understood that the correspondence between the tensor dimension of the above-mentioned input mask tensor and the tensor dimension of the original mask tensor is related to the distribution mode of the data. When the distribution mode of the data in the input tensor changes, the technicians in this field will adjust the above-mentioned correspondence accordingly based on the distribution mode of the data. For example, if the width dimension is 1024, the corresponding height dimension will become aligned with 1. This is because for each computing unit, the total amount of data bits of a register is determined, so the total amount of data (i.e. 32 x 32 = 1024) can be determined by the product of the number of threads of each computing unit (i.e. 32) and the size of the register corresponding to each thread (i.e. 32 bits). In other words, the correspondence between the tensor dimension of the above-mentioned input mask tensor and the tensor dimension of the original mask tensor is not unique.

[0054] Further, according to an embodiment of the present invention, determining data for filling an input mask tensor based on an input tensor may include: determining a sequence length and a mask type corresponding to the input tensor based on the input tensor; and determining data for filling the input mask tensor based on the sequence length and the mask type.

[0055] Regarding the sequence length (seq_len), it may indicate the length of the sequence input into the model. According to an embodiment of the present invention, the sequence length may be related to the input tensor. For example, the sequence length may be positively correlated with the input tensor. In some embodiments, the sequence length may be consistent with the size of, for example, only the last two dimensions of the tensor dimension of the input tensor, and has nothing to do with the size of other dimensions.

[0056] Regarding the mask type, it may refer to masks with different shapes. According to an embodiment of the present invention, the mask type may include: a lower triangle type and a square type.

[0057] As described above, the data used to fill the input mask tensor is bit data, and its value is one of "0" and "1". According to an embodiment of the present invention, the data used to fill the input mask tensor may be related to the mask type and the sequence length.

[0058] For example, in one embodiment, based on the input tensor, it can be determined that the sequence length corresponding to the input tensor is 100 (i.e., seq_len = 100) and the mask type is lower triangular. Then, according to the sequence length seq_len = 100 and the mask type is lower triangular, it can be known that for the bit-type input mask tensor to be generated, it will include a lower triangular area with a size of 100*100 in the height dimension (H dimension) and the width dimension (W dimension), and it can be determined that the value of the data to be filled in the lower triangular area is 1, and the value of the data to be filled in other areas outside the lower triangular area is 0. The following will be combined with Figure 4A It is not necessary to describe in detail how to determine the data used to fill the input mask tensor.

[0059] In step 304 , at the device end, at least one computing unit in the device end generates an input mask tensor based on the tensor dimension of the input mask tensor and the data for filling the input mask tensor.

[0060] Regarding the computing unit, generally, one or more computing units may be included in the device end. Taking the device end as a GPU as an example, multiple computing units may be included thereon, such as hundreds to thousands of computing units. Thus, according to an embodiment of the present invention, multiple computing units on the device end may be used to generate input mask tensors in parallel, thereby improving computing efficiency.

[0061] Further, each computing unit may include multiple threads. For example, according to some embodiments of the present invention, a computing unit may be provided with at least one row of threads, and each row of threads further includes multiple threads. Typically, each row of threads in a computing unit may include 32 threads. It should be understood that the number of threads that each row of threads may include is not limited to 32. In some other embodiments, a row of threads may include, for example, 16, 48, or 64 threads, and the present invention is not limited to this.

[0062] In addition, according to some embodiments of the present invention, each thread can be used to generate, for example, 32 bits of data. However, it should be understood that in some other embodiments, each thread can be used to generate other numbers (such as 16 or 48) of bit data, and the present invention is not limited to this.

[0063] In some embodiments, the device end may include multiple computing units, each of which may include multiple rows of threads. According to the inventive concept of the present invention, each computing unit includes the same number of rows of threads. For example, the device end includes four computing units, and each computing unit includes two rows of threads.

[0064] In some other embodiments, each row of threads at the device end includes the same number of threads. For example, in the example described above where the device end includes four computing units and each computing unit includes two rows of threads, each row of threads includes, for example, 32 threads, that is, each computing unit includes 2*32 threads.

[0065] According to the inventive concept of the present invention, by pairing the threads on each computing unit with the data included in the input mask tensor, the input mask tensor is generated in parallel by multiple threads on the computing unit. Specifically, according to an embodiment of the present invention, the starting column index and the starting row index of the input mask tensor can be first determined; then, the starting column index and the starting row index can be matched with the computing unit index and the thread index to obtain the correspondence between the data processed by each thread of each computing unit and the data in the input mask tensor; and, based on the obtained correspondence and the data used to fill the input mask tensor, the input mask tensor is generated. The following will be combined with Figure 4A and Figure 4B Describes in detail how to generate input mask tensors in parallel via multiple threads on multiple compute units.

[0066] Figure 4A A schematic diagram of an exemplary input mask tensor to be generated according to an embodiment of the present invention is shown.

[0067] For example, to generate an input mask tensor with a size of 100*100 and a mask type of lower triangle in the height dimension (H dimension) and width dimension (W dimension), Figure 2 It shows that the input mask tensor is obtained when the mask type is lower triangular and the sequence length is 100.

[0068] like Figure 4A As shown, it can be determined that the starting column index (col_start_idx) of the input mask tensor is 0, and the starting row index (row_start_idx) of the input mask tensor is 0. It can be seen that in Figure 4A In the example shown, the column index (col_idx) of the input mask tensor to be generated ranges from 0 to 99, and the row index (row_idx) ranges from 0 to 99.

[0069] According to an embodiment of the present invention, the column index and row index of each data in the input mask tensor can be compared to obtain data for filling the input mask tensor. Specifically, the column index and row index of each data in the input mask tensor can be determined; for each data, its column index is compared with its row index; and based on the comparison result, the value of the data for filling the input mask tensor is determined.

[0070] For example, for Figure 4A The input mask tensor shown in , the column index (col_idx) and row index (row_idx) of each data in the input mask tensor can be determined first, and then the column index col_idx of each data is compared with the row index row_idx, so as to determine the number of cells to be filled based on the comparison result. Figure 4A The value of the input mask tensor data. Figure 4A In the example input mask tensor shown, since the mask type of the input mask tensor is lower triangular and the sequence length is 100, it is used to fill Figure 4A The shaded bit data in the input mask tensor must be 1 to fill Figure 4A The values ​​of the bit-type data in other parts of the input mask tensor need to be 0. That is, in this example, the column index col_idx and the row index row_idx of each data in the input mask tensor can be compared. If the comparison result of the column index col_idx and the row index row_idx of the current data satisfies row_idx ≥ col_idx, the bit value corresponding to the current data is determined to be 1; accordingly, if the comparison result of the column index col_idx and the row index row_idx of the current data does not satisfy row_idx ≥ col_idx, the bit value corresponding to the current data is determined to be 0.

[0071] For example, for the data with column index col_idx = 6 and row index row_idx = 2 in the input mask tensor, since row_idx < col_idx, that is, the comparison result of the column index col_idx and the row index row_idx of the data does not satisfy row_idx ≥ col_idx, the corresponding bit value is 0. For another example, for the data with column index col_idx = 88 and row index row_idx = 88 in the input mask tensor, since row_idx = col_idx, that is, the comparison result of the column index col_idx and the row index row_idx of the data satisfies row_idx ≥ col_idx, the corresponding bit value is 1.

[0072] Figure 4BA schematic diagram of the layout of a computing unit in a device end 200 according to an embodiment of the present invention is shown, wherein a Figure 4A An example input mask tensor is shown.

[0073] Reference Figure 4B ,like Figure 4B As shown, the device 200 includes at least three computing units, namely, a first computing unit compute_unit_0, a second computing unit compute_unit_1, and a third computing unit compute_unit_2. Further, each computing unit includes two rows of threads, and each row of threads includes 32 threads. Taking the first computing unit compute_unit_0 as an example, its first row of threads includes the first thread CU0_L1_T0 to the thirty-second thread CU0_L1_T31. Further, as Figure 4B As shown, for each thread, it can be used to generate 32 bits of data.

[0074] Therefore, according to an embodiment of the present invention, based on the number of computing units and the number of rows of threads included in each computing unit, as well as the position of the current thread in the current row of threads, the correspondence between the data processed by each computing unit and the data in the input mask tensor to be generated can be determined, and the correspondence between the data processed by each thread and the data in the input mask tensor to be generated can be further determined.

[0075] In order to better distinguish different threads set on the device side, according to the inventive concept of the present invention, a computing unit index (compute_unit_idx) and a thread index (thread_idx) are introduced, and then the starting column index and the starting row index are matched with the computing unit index and the thread index to obtain the correspondence between the data processed by each thread on the device side and the data in the input mask tensor to be generated.

[0076] Regarding the computing unit index, it can be used to identify different computing units on the device side. For example, the computing unit index of the first computing unit compute_unit_0 can be recorded as 0 (that is, compute_unit_idx = 0), the computing unit index of the second computing unit compute_unit_1 can be recorded as 1 (that is, compute_unit_idx = 1), and so on.

[0077] Regarding the thread index, it can be used to identify the position of the thread in the current row of threads. For example, in Figure 4B, thread CU0_L1_T0 is the first thread in the first row of threads of the first computing unit compute_unit_0, so the thread index of thread CU0_L1_T0 can be recorded as 0 (i.e., thread_idx = 0). Similarly, the thirty-second thread CU0_L1_T31 is the thirty-second thread in the first row of threads of the first computing unit compute_unit_0, so the thread index of the thirty-second thread CU0_L1_T31 can be recorded as 31 (i.e., thread_idx = 31).

[0078] On this basis, the starting column index and the starting row index can then be matched with the computing unit index and the thread index to obtain the correspondence between the data processed by each thread of each computing unit and the data in the input mask tensor.

[0079] Specifically, according to an embodiment of the present invention, the starting column index of the input mask tensor can be matched with the above-mentioned thread index, and the starting row index of the input mask tensor can be matched with the above-mentioned computing unit index to obtain the correspondence between the data processed by different computing units at the device end and the data in the input mask tensor.

[0080] Assume that each thread can be used to generate k According to an embodiment of the present invention, the data processed by a thread at the device end may correspond to the column index range of [col_start_idx + thread_idx * k , col_start_idx + thread_idx * k + k )of k data, where col_start_idx is the starting column index of the input mask tensor, thread_idx is the thread index of the thread, k is the number of bits of data that a thread can generate. Figure 4A Take the input mask tensor shown as an example, where the starting column index col_start_idx = 0, then Figure 4B The data processed by the thread with thread index thread_idx=0 at the device end 200 shown may correspond to Figure 4A The column index range of the input mask tensor is [0, 32) 32 data; similarly, the data processed by the thread with thread index thread_idx = 31 can correspond to Figure 4A The input mask tensor has 32 data with column index range [992, 1024).

[0081] Further, according to an embodiment of the present invention, the data processed by multiple threads in a computing unit at the device end may correspond to data in the input mask tensor with a row index range of [row_start_idx + compute_unit_idx *row_num, row_start_idx + compute_unit_idx *row_num + row_num - 1], where row_start_id is the starting row index of the input mask tensor, compute_unit_idx is the index of the computing unit, and row_num is the number of rows of consecutive threads assigned to the computing unit. As described above, each computing unit includes the same number of rows of threads. Figure 4A Take the input mask tensor shown as an example, where the starting row index row_start_idx = 0, then Figure 4B The computing unit with the computing unit index compute_unit_idx = 1 (ie, the second computing unit compute_unit_1) at the device end shown in the figure may process data corresponding to Figure 4A The row index range of the input mask tensor is [2,3]; similarly, for the computing unit with the computing unit index compute_unit_idx = 2, the data it processes can correspond to Figure 4A The data in the row index range of [4, 5] in the input mask tensor of . On this basis, the data processed by each row thread in the computing unit corresponds one by one to each row of data in the corresponding row index range in the input mask tensor. For example, Figure 4B The first row of the compute unit with compute_unit_idx = 1 can be processed by the thread Figure 4A The row index of the input mask tensor is 2, and Figure 4B The second row of threads for the compute unit with compute_unit_idx = 1 can process Figure 4A The data with row index 3 in the input mask tensor of .

[0082] In summary, we can then get the correspondence between the data processed by each computing unit on the device and the data in the input mask tensor. Figure 4B The second thread of the first row of the first computing unit in , that is, thread CU0_L1_T1, can process data corresponding to Figure 4A The input mask tensor of has a column index range of [32, 64) and a row index of 0, totaling 32 data. In another example, for Figure 4B The thirty-second thread of the second row of the third computing unit in , that is, thread CU2_L2_T31, can process data corresponding to Figure 4A The input mask tensor has a column index range of [992, 1024) and a row index of 5, totaling 32 data.

[0083] In response to obtaining the correspondence between the data that can be processed by each thread and the data in the input mask tensor, the input mask tensor can then be generated in parallel by the threads of the multiple computing units based on the data used to fill the input mask tensor.

[0084] According to an embodiment of the present invention, the value of the data processed by each thread can be determined based on the correspondence between the processed data and the data in the input mask tensor and the data used to fill the input mask tensor. Figure 4B For example, thread CU0_L1_T1 processes the data corresponding to Figure 4A The input mask tensor of has a column index range of [32,64) and a row index of 0, which is a total of 32 data. Combined with the above description of determining the data used to fill the input mask tensor, it can be seen that the bit values ​​corresponding to the data with a row index of 0 and a column index range of [32, 64) in the input mask tensor are all 0. Therefore, it can be seen that thread CU0_L1_T1 will generate 32 data with a bit value of 0 in the 32nd to 64th columns of the first row of the input mask tensor. For another example, for Figure 4B The thread CU2_L2_T0 processes the data corresponding to Figure 4A There are 32 data with a column index range of [0, 32) and a row index of 5 in the input mask tensor. Combined with the above description of determining the data used to fill the input mask tensor, it can be seen that in the data with a row index of 5 and a column index range of [0, 32) in the input mask tensor, the bit value corresponding to the data with a column index range of [0, 5] is 1, and the bit value corresponding to the data with a column index range of [6, 32) is 0. Therefore, thread CU2_L2_T0 will generate 5 data with a bit value of 1 in the 1st to 5th columns of the fifth row of the input mask tensor, and generate 17 data with a bit value of 0 in the 6th to 32nd columns of the same row.

[0085] Further, according to an embodiment of the present invention, the data in the corresponding input mask tensor can be generated in parallel by multiple threads in the first row of threads of each computing unit. Figure 4B Taking the device side as an example, thirty-two threads in the first row of threads of the first computing unit compute_unit_0, thirty-two threads in the first row of threads of the second computing unit compute_unit_1, and thirty-two threads in the first row of threads of the third computing unit compute_unit_2 can generate data in the corresponding input mask tensor in parallel.

[0086] In an embodiment of the present invention, each thread can generate data from left to right in sequence. Combined with the correspondence between the data that each thread can process and the data in the input mask tensor described above, Figure 4B For example, thread CU0_L1_T0 can process data corresponding to Figure 4A There are 32 data with a column index range of [0, 32) and a row index of 0 in the input mask tensor. According to the inventive concept of the present invention, the order in which thread CU0_L1_T0 generates data is: data with a row index of 0 and a column index of 0, data with a row index of 0 and a column index of 1, data with a row index of 0 and a column index of 2, until data with a row index of 0 and a column index of 31.

[0087] On this basis, the thirty-two threads in the first row of threads of the first computing unit compute_unit_0 can generate data with a row index of 0 in parallel, wherein thread CU0_L1_T0 sequentially generates data with a column index range of [0, 32), thread CU0_L1_T1 sequentially generates data with a column index range of [32, 64), and so on. While the first computing unit compute_unit_0 generates data with a row index of 0, the second computing unit compute_unit_1 generates data with a row index of 2 in parallel, and the third computing unit compute_unit_2 generates data with a row index of 4 in parallel, and the manner in which each thread in the second computing unit compute_unit_1 and the third computing unit compute_unit_2 generates data is similar to that of the first computing unit compute_unit_0, which will not be repeated here. When the first row of threads of all computing units at the device end completes data generation, all computing units simultaneously switch to starting to generate data in parallel from the threads in their respective second row of threads.

[0088] In summary, according to an embodiment of the present invention, multiple computing units at the device end can generate input mask tensors in parallel, thereby improving the generation efficiency of the input mask tensor, and the data in the generated input mask tensor is bit-type data, thereby reducing bandwidth pressure and improving model training performance during the subsequent calculation of the masked_fill function.

[0089] According to the inventive concept of the present invention, the scheme for generating an input mask tensor on the device side provided by the present invention may also include: determining the dimensional range of the input mask tensor that can be processed each time based on at least the number of at least one computing unit, the number of threads included in each computing unit, and the size of the register corresponding to each thread; in response to the dimensional range of the input mask tensor that can be processed each time being different from the tensor dimension of the input mask tensor, dividing the input mask tensor based on the dimensional range of the input mask tensor that can be processed each time to obtain multiple input mask tensor partitions; and for each input mask tensor partition, generating an input mask tensor part corresponding to the input mask tensor partition based on the data used to fill the input mask tensor, so as to generate the input mask tensor.

[0090] Regarding the number of threads included in each computing unit, here, it may refer to any one of the number of rows of threads included in each computing unit or the number of threads included in each row of threads in the computing unit.

[0091] The size of the register corresponding to each thread may be the same as the number of bit data that each thread can generate. For example, if a thread can be used to generate 32 bits of data, the size of the register corresponding to the thread is 32 bits.

[0092] According to an embodiment of the invention, determining the dimensional range of the processable input mask tensor each time may include: determining the height dimensional range of the processable input mask tensor based on the number of computing units (compute_unit_num) and the number of rows of threads included in each computing unit (row_num), and determining the width dimensional range of the processable input mask tensor based on the number of threads included in each row of threads in the computing unit (thread_num) and the size of the register corresponding to each thread (threat_bit). Specifically, assuming that the starting column index col_start_idx = a and the starting row index row_start_idx = b of the currently generated input mask tensor partition, the dimensional range of the processable input mask tensor each time can be calculated based on the following formula:

[0093] Height dimension = [b, b + compute_unit_num * row_num)

[0094] Width dimension = [a, a + thread_num * threat_bit)

[0095] As Figure 4BTaking the device end 200 shown in the figure as an example, assuming that the device end 200 only includes the three computing units shown in the figure, each computing unit includes two rows of threads, and each row of threads includes 32 threads, and the size of the register corresponding to each thread is 32 bits, then the dimension range of the input mask tensor that can be processed by the device end 200 each time is:

[0096] Height dimension = [b, b + 3 * 2) = [b, b + 6)

[0097] Width dimension = [a, a + 32 * 32) = [a, a + 1024)

[0098] From the above, we can see that Figure 4B The device end 200 shown can be used to generate data in an input mask tensor of dimension 6*1024 each time, that is, the device end 200 can be used to generate 6 rows of data each time, each row of data includes 1024 bits. Figure 4A The input mask tensor shown is an input mask tensor with a dimension of 100*100. Obviously, the device end 200 cannot generate all the input mask tensors at once. Therefore, according to an embodiment of the present invention, the input mask tensor can be divided to obtain multiple input mask tensor partitions, and the device end 200 generates an input mask tensor part corresponding to one input mask tensor partition in the input mask tensor each time, and then updates the starting column index and the starting row index of the input mask tensor part to achieve the offset of the position of the data processed by the device end 200 in the input mask tensor, thereby achieving the generation of the input mask tensor.

[0099] For example, for Figure 4A The 100*100 input mask tensor shown in FIG. 1 determines its starting column index col_start_idx = 0 and starting row index row_start_idx = 0, and generates data with a row index range of [0, 6) by the device end 200. In response to the generation of all data with a row index range of [0, 6) in the input mask tensor, the starting column index and starting row index of the input mask tensor are updated to col_start_idx = 0 and row_start_idx = 6, and the device end 200 generates data with a row index range of [6, 12). And so on, until all 100 rows of data of the input mask tensor are generated.

[0100] Similarly, in some other embodiments, if the width dimension of the input mask tensor is larger than 1024 bits that the device end 200 can process at a time, then after each time the data of the input mask tensor part is generated, the starting row index of the input mask tensor part can be updated to row_start_idx = 0, and the starting column index can be updated to the previous value plus 1024, and then the data of the input mask tensor part can be generated based on the updated starting row index and starting column index until all the column data of the input mask tensor are generated.

[0101] According to the inventive concept of the present invention, in some embodiments, the generated input mask tensor can also be applicable to multiple input sequences. In this case, the input mask tensor can be divided into multiple lower triangular areas, and the length of the right angle of each lower triangular area can be related to the sequence length of one of the multiple input sequences. For example, if the sequence length is 5000, the length of the right angle of the corresponding lower triangular area is 5000. Furthermore, the value of the data to be filled in each lower triangular area is 1, and the value of the data to be filled in the areas outside these lower triangular areas is 0.

[0102] According to the inventive concept of the present invention, in some embodiments, in order to adapt the arrangement of the input mask tensor that can be loaded by the masked_fill function in the computing unit, each generated bit data can be corresponded to two rows of data in the input mask tensor. The matching principle is as described above and will not be repeated here.

[0103] In summary, the scheme for generating an input mask tensor on the device side provided by the present invention enables the input mask tensor for the masked_fill function to be directly generated on the device side, omitting the transmission process of the input mask tensor between the host side and the device side, saving time. In addition, in the above scheme of the present invention, by making the data in the generated input mask tensor bit-type data, it is possible to reduce bandwidth pressure and improve model training performance in subsequent function calculations.

[0104] The various processes and processing described herein, such as method 300, may be performed at a computing device. The computing device includes, for example: at least one processor (at least one graphics processor and at least one central processing unit); and a memory connected to at least one processor in communication; wherein the memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor. In some embodiments, method 300 may be implemented as a computer software program or program product, which is tangibly contained in a machine-readable medium. In some embodiments, part or all of the computer program may be loaded and / or installed on a computing device via a read-only memory (ROM) and / or a communication unit. When the computer program is loaded into a random-access memory (RAM) and executed by a GPU and a GPGPU, one or more actions of method 300 described above may be performed.

[0105] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions for executing various aspects of the present invention. The computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof.

[0106] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each box of the flowchart and / or block diagram, and the combination of boxes in the flowchart and / or block diagram, can be implemented by computer-readable program instructions.

[0107] These computer-readable program instructions can be provided to a central processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the central processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0108] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0109] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0110] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0111] The above are only optional embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating an input mask tensor, characterized in that include: Based on at least the input tensor, determine tensor dimensions of the input mask tensor and data for filling the input mask tensor; as well as At least one computing unit in the device generates the input mask tensor based on the tensor dimension of the input mask tensor and the data used to fill the input mask tensor, wherein the data in the input mask tensor is bit-type data, The method is performed at the device end. Each of the at least one computing unit comprises a plurality of threads, and Wherein, generating the input mask tensor includes: The plurality of threads included in each computing unit are paired with the data in the input mask tensor to generate the input mask tensor in parallel.

2. The method according to claim 1, characterized in that Determining, based at least on the input tensor, a tensor dimension of an input mask tensor and data for filling the input mask tensor comprises: Based on the input tensor, determining the tensor dimension of the original mask tensor, wherein the data of the original mask tensor is Boolean data; and Based on the tensor dimension of the original mask tensor and a predetermined conversion rule, the tensor dimension of the input mask tensor is determined.

3. The method according to claim 2, characterized in that The data of the input tensor is Boolean data, and the tensor dimension of the input tensor is the same as the tensor dimension of the original mask tensor.

4. The method according to claim 2, characterized in that: The predetermined conversion rule is: a conversion rule for converting the tensor dimension of a Boolean tensor into the tensor dimension of a bit tensor.

5. The method according to claim 1, characterized in that Determining, based at least on the input tensor, a tensor dimension of an input mask tensor and data for filling the input mask tensor comprises: Based on the input tensor, determining a sequence length and a mask type corresponding to the input tensor; and The data for filling the input mask tensor is determined according to the sequence length and the mask type.

6. The method according to claim 5, characterized in that The mask types include lower triangle type and square type.

7. The method according to claim 5, characterized in that Determining the data used to fill the input mask tensor includes: Determine the column and row indices of each data in the input mask tensor; For each data, comparing the column index with the row index; and Based on the comparison result, a bit value corresponding to each data is determined to facilitate determining the value of the data used to fill the input mask tensor.

8. The method according to claim 7, characterized in that Determining the bit value corresponding to each data based on the comparison result includes: In response to the comparison result satisfying a predetermined condition, determining that the bit value corresponding to the current data is 1; and In response to the comparison result not satisfying the predetermined condition, determining that the bit value corresponding to the current data is 0, The predetermined condition at least includes: the row index of the current data is greater than or equal to the column index of the current data.

9. The method according to claim 5, characterized in that Each of the at least one computing unit comprises a plurality of threads; and, Generating the input mask tensor based on the tensor dimension of the input mask tensor and the data for filling the input mask tensor includes: Determine a starting column index and a starting row index of the input mask tensor; Matching the starting column index and the starting row index with the computing unit index and the thread index to obtain a correspondence between data processed by each thread of each computing unit and data in the input mask tensor; and Based on the obtained correspondence and the data for filling the input mask tensor, the input mask tensor is generated.

10. The method according to claim 9, characterized in that Generating the input mask tensor further includes: Determining a dimensional range of the input mask tensor that can be processed each time based at least on the number of the at least one computing unit, the number of threads included in each computing unit, and the size of a register corresponding to each thread; In response to the dimensional range of the input mask tensor that can be processed each time being different from the tensor dimension of the input mask tensor, dividing the input mask tensor based on the dimensional range of the input mask tensor that can be processed each time to obtain a plurality of input mask tensor partitions; and For each input mask tensor partition, a portion of the input mask tensor corresponding to the input mask tensor partition is generated based on the data for filling the input mask tensor, so as to generate the input mask tensor.

11. A computing device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 1 to 10.

13. A computer program product, characterized in that The computer program product is tangibly stored on a non-transitory computer readable medium and comprises machine executable instructions which, when executed, cause a machine to perform the steps of the method according to any one of claims 1-10.