Matrix row direction maximum value determination method, electronic equipment and storage medium
By applying for space in the general matrix main cache and high bandwidth memory of the artificial intelligence chip, and using multi-threaded groups to calculate the maximum value of matrix elements, the problem of low efficiency in determining the maximum value of matrix row direction in the prior art is solved, and more efficient computing and resource utilization is achieved.
Patent Information
- Application Number
- CN202510637126.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-16
- Publication Date
- 2025-08-19
AI Technical Summary
In the prior art, the efficiency of determining the maximum value of matrix row direction is low and occupies a lot of hardware computing resources. Especially in parallel computing architectures such as GPU, GPGPU or TPU, the limitations of broadcasting mechanisms and high bandwidth memory lead to low computing efficiency.
Apply for the matrix layout space of the target matrix in the general matrix main cache of the artificial intelligence chip, and apply for the vector layout space in the high-bandwidth memory. The maximum value of each row of elements of the matrix is calculated through multiple thread groups, the intermediate result is covered and written into the matrix layout space, and finally the result is written into the vector layout space to avoid the broadcast mechanism, and the low latency characteristics of the general matrix main cache are used to reduce the read and write to high-bandwidth memory.
It improves the calculation efficiency of determining the maximum value of matrix row direction, reduces the use of hardware computing resources, reduces the number of read and writes to high-bandwidth memory, and improves computing performance.
Smart Images
Figure CN120508389A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence chip technology, and in particular to a method for determining a maximum value in a matrix row direction, an electronic device, and a storage medium. Background Art
[0002] Determining the maximum value along a matrix row is a crucial operation in the fields of artificial intelligence and high-performance computing. Operators such as the Softmax operator, Batch Normalization (Batchnorm), and Attention (Attention) in neural networks require this technique to calculate the maximum value of each row element in the matrix. This is used to compute the reverse gradient and perform normalization calculations during exponential summations to prevent data overflow. Accurate and efficient determination of the maximum value along a matrix row can significantly reduce the computation time of a single operator and improve the end-to-end performance of neural networks.
[0003] In related technologies, parallel computing architectures such as graphics processing units (GPUs), general-purpose computing on graphics processing units (GPGPUs), or tensor processing units (TPUs) typically use warps as the granularity to determine the maximum value in the row direction of a matrix.
[0004] However, on the one hand, the aforementioned parallel computing architecture triggers a broadcast mechanism when determining the maximum value in the matrix row direction at the thread group granularity. This broadcast mechanism consumes a large amount of hardware computing resources and reduces the computational efficiency of determining the maximum value in the matrix row direction. On the other hand, the aforementioned parallel computing architecture stores the temporary data generated during this process in High Bandwidth Memory (HBM). Due to the low bandwidth and high latency of HBM, frequent reads and writes to HBM further reduce the computational efficiency of determining the maximum value in the matrix row direction. Summary of the Invention
[0005] The present invention provides a method for determining a maximum value in a matrix row direction, an electronic device and a storage medium, which are used to solve the defects of low efficiency and large occupation of hardware computing resources in the prior art for determining a maximum value in a matrix row direction.
[0006] The present invention provides a method for determining a maximum value in a matrix row direction, comprising the following steps.
[0007] When a target calculation task for calculating the maximum value of a target matrix in the row direction is triggered, a matrix layout space corresponding to the target matrix is requested in the general matrix main cache of the artificial intelligence chip, and a vector layout space corresponding to the target matrix is requested in the high-bandwidth memory of the artificial intelligence chip.
[0008] Call multiple thread groups to calculate the maximum value of each row of the target matrix, and then call the first The thread group calculates the target matrix In the process of finding the maximum value of the elements in the row, Each set of intermediate results calculated by the thread group is overwritten and written into the matrix layout space corresponding to the target matrix. For each element in the row, , Indicates the total number of rows in the target matrix.
[0009] In determining the The thread group completes the target matrix In the case of calculating the maximum value of the elements in a row, the matrix layout space corresponding to the target matrix is Any element in the row overwrites the first element in the vector layout space corresponding to the target matrix. row, obtain the target matrix The maximum value of the elements in the row.
[0010] According to a method for determining the maximum value in a matrix row direction provided by the present invention, the method of calling the first The thread group calculates the target matrix In the process of finding the maximum value of the elements in the row, Each set of intermediate results calculated by the thread group is overwritten and written into the matrix layout space corresponding to the target matrix. Each element in the row includes: calling the Each thread in the thread group reads the target matrix No. Each element in the element group, A positive integer that starts at 0. The maximum value is based on the The number of threads in the thread group is determined by the target matrix Each element in the row is based on the The number of threads in the thread group is divided into multiple element groups in order from front to back; obtain the target matrix No. The maximum value of each element in the element group; based on the target matrix No. The maximum value of each element in the element group, get the The first thread group calculates an intermediate result; The first thread group calculates The intermediate result is overwritten and written into the matrix layout space corresponding to the target matrix. Each element of the row; in determining When the maximum value is reached, the The thread group completes the target matrix The maximum value of the elements in the row is calculated in If the maximum value is not reached, Increase by 1, return to execute the call Each thread in the thread group reads the target matrix No. Steps for each element in the element group until the Reached maximum value.
[0011] According to a matrix row direction maximum value determination method provided by the present invention, the target matrix No. The maximum value of each element in the element group, get the The first thread group calculates An intermediate result, including: reading the first Each element in the row; compare the first element in the matrix layout space corresponding to the target matrix Each element in the row and the target matrix No. The maximum value of each element in the element group; the first Each element in the row and the target matrix No. The largest of the maximum values of the elements in the element group is determined as the The first thread group calculates An intermediate result.
[0012] According to a method for determining a maximum value in a matrix row direction provided by the present invention, applying for a matrix layout space corresponding to the target matrix in a general matrix main cache of an artificial intelligence chip includes: Applying for a matrix layout space in the general matrix main cache of the artificial intelligence chip, the matrix layout space having the same row dimension as the target matrix and the same column dimension as the number of threads in the thread group; After initializing each element in the matrix layout space to negative infinity, the matrix layout space is determined as the matrix layout space corresponding to the target matrix.
[0013] According to a method for determining the maximum value in a matrix row direction provided by the present invention, applying for a matrix layout space corresponding to the target matrix in the general matrix main cache of an artificial intelligence chip includes: applying for a matrix layout space in the general matrix main cache of the artificial intelligence chip whose row and column dimensions are the same as the row and column dimensions of the target matrix; after initializing each element in the matrix layout space to negative infinity, determining the matrix layout space as the matrix layout space corresponding to the target matrix.
[0014] According to a method for determining the maximum value in a matrix row direction provided by the present invention, applying for a matrix layout space corresponding to the target matrix in the general matrix main cache of an artificial intelligence chip includes: applying for a matrix layout space in the general matrix main cache of the artificial intelligence chip whose row dimension is the same as the row dimension of the target matrix but whose column dimension is 1; after initializing each element in the matrix layout space to negative infinity, determining the matrix layout space as the matrix layout space corresponding to the target matrix.
[0015] According to a method for determining the maximum value in the row direction of a matrix provided by the present invention, applying for a vector layout space corresponding to the target matrix in the high-bandwidth memory of the artificial intelligence chip includes: applying for a vector layout space in the high-bandwidth memory of the artificial intelligence chip whose row dimension is the same as the row dimension of the target matrix and whose column dimension is 1, as the vector layout space corresponding to the target matrix.
[0016] The present invention also provides a device for determining a maximum value in a matrix row direction, comprising the following modules: The space application module is used to apply for the matrix layout space corresponding to the target matrix in the general matrix main cache of the artificial intelligence chip and apply for the vector layout space corresponding to the target matrix in the high-bandwidth memory of the artificial intelligence chip when a target calculation task for calculating the maximum value of the target matrix in the row direction is triggered.
[0017] The numerical calculation module is used to call multiple thread groups to calculate the maximum value of the elements in each row of the target matrix, and to call the first The thread group calculates the target matrix In the process of finding the maximum value of the elements in the row, Each set of intermediate results calculated by the thread group is overwritten and written into the matrix layout space corresponding to the target matrix. For each element in the row, , Indicates the total number of rows in the target matrix.
[0018] The result output module is used to determine the The thread group completes the target matrix In the case of calculating the maximum value of the elements in a row, the matrix layout space corresponding to the target matrix is Any element in the row overwrites the first element in the vector layout space corresponding to the target matrix. row, obtain the target matrix The maximum value of the elements in the row.
[0019] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-described methods for determining the maximum value in the row direction of a matrix is implemented.
[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for determining the maximum value in the row direction of a matrix as described above is implemented.
[0021] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for determining the maximum value in the row direction of a matrix.
[0022] The present invention provides a matrix row direction maximum value determination method, electronic device and storage medium. When a target calculation task for calculating the row direction maximum value of a target matrix is triggered, a matrix layout space corresponding to the target matrix is applied for in the general matrix main cache of an artificial intelligence chip, and a vector layout space corresponding to the target matrix is applied for in the high-bandwidth memory of the artificial intelligence chip. Then, multiple thread groups are called to respectively calculate the maximum value of the elements in each row of the target matrix. When the first thread group in each thread group is called, the maximum value of the elements in each row of the target matrix is calculated. The thread group calculates the target matrix In the process of finding the maximum value of the elements in the row, Each set of intermediate results calculated by the thread group is overwritten and written into the matrix layout space corresponding to the target matrix. For each element in the row, The thread group completes the target matrix In the case of calculating the maximum value of the elements in a row, the matrix layout space corresponding to the target matrix is Any element in the row is overwritten in the vector layout space corresponding to the target matrix. row, get the target matrix The maximum value of the elements in the row can avoid triggering the broadcast mechanism in the process of determining the maximum value in the matrix row direction, thereby reducing the hardware computing resource occupation of the artificial intelligence chip and improving the computational efficiency of determining the maximum value in the matrix row direction. It can utilize the low latency characteristics of the general matrix main cache of the artificial intelligence chip to store the temporary data generated in the process of determining the maximum value in the matrix row direction to the general matrix main cache of the artificial intelligence chip, which can significantly reduce the number of times high-bandwidth memory is read and written during the calculation process in the matrix row direction, thereby further improving the computational efficiency of determining the maximum value in the matrix row direction. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0024] Figure 1 It is a schematic diagram of the principle of a traditional method for determining the maximum value in a matrix row direction in related technology.
[0025] Figure 2 It is a structural diagram of the universal image processor provided by the present invention.
[0026] Figure 3 It is a flow chart of the method for determining the maximum value in the matrix row direction provided by the present invention.
[0027] Figure 4 It is a schematic diagram of the principle of the method for determining the maximum value in the matrix row direction provided by the present invention.
[0028] Figure 5 It is a structural schematic diagram of the device for determining the maximum value in the matrix row direction provided by the present invention.
[0029] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0030] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.
[0031] It should be noted that for a target matrix with a shape of (H, W), when calculating the maximum value of elements in each row of the target matrix, the parallel computing architecture in the related art usually first applies for a vector layout space with a shape of (H, 1) in the high-bandwidth memory as the vector layout space corresponding to the target matrix. This space is used to store temporary data and the final calculation results generated during the process of calculating the maximum value of elements in each row of the target matrix.
[0032] Parallel computing architectures typically use thread groups as the granularity for determining the maximum value in a matrix row direction. Each thread group contains multiple threads. This parallel computing architecture can call on multiple thread groups to independently calculate the maximum value for each row of the target matrix.
[0033] Figure 1 This is a schematic diagram of the principle of the traditional matrix row direction maximum value determination method in the related art. Figure 1 As shown, for the first OK( ), the vector layout space corresponding to the above target matrix in high bandwidth memory Elements of a row Initialized to negative infinity (-INFINITY).
[0034] Call each thread in a thread group to read the first No. Each element in the element group. A positive integer indicating that the starting value is 0; The maximum value of W / N is rounded up; N represents the number of threads in the above thread group; Each element in the row is divided into an element group every N elements in the order from front to back. For example, in the target matrix above, The first element group in the row includes the first element in the target matrix The first N elements in the row.
[0035] Comparison of the above The maximum value of each element in the element group Afterwards, the thread group can read the first Elements of a row , and then the above The maximum value of each element in the element group The first vector layout space corresponding to the target matrix Elements of a row Compare and write the larger one back to the vector layout space corresponding to the target matrix Elements of a row .
[0036] In determining If the maximum value is not reached, Increase by 1, return to execute each thread in the above thread group to read the first No. Steps for each element in the element group until the Reaching the maximum value, Determine the target matrix The final computed maximum value of the elements in the row.
[0037] In the related calculation, the thread group reads the vector layout space corresponding to the target matrix. Elements of a row When , the broadcast mechanism will be triggered, and the vector layout space corresponding to the above target matrix will be Elements of a row The result is broadcast to each thread in the thread group. However, the broadcast mechanism will occupy more hardware computing resources and reduce the computational efficiency of determining the maximum value in the matrix row direction.
[0038] And, comparing the above The maximum value of each element in the element group The first vector layout space corresponding to the target matrix Elements of a row After the larger one between the two, it is necessary to overwrite the larger one back to the vector layout space corresponding to the above target matrix. Elements of a row , calculate the first The number of times the process of obtaining the maximum value of the row reads and writes to the high-bandwidth memory is However, due to the high latency of HBM, frequent reading and writing of HBM will further reduce the computational efficiency of determining the maximum value in the matrix row direction.
[0039] Therefore, the traditional matrix row direction maximum value determination method in the related art has low efficiency in determining the matrix row direction maximum value and occupies more hardware computing resources.
[0040] To address this issue, the present invention provides a method for determining a maximum value in a matrix row direction. The method, provided by the present invention, stores temporary data generated during the matrix row direction maximum value determination process in the general matrix main cache of an artificial intelligence chip. This method can utilize the low-latency characteristics of the general matrix main cache of the artificial intelligence chip to significantly reduce the number of reads and writes to high-bandwidth memory during the matrix row direction calculation process, thereby improving the computational efficiency of determining the maximum value in the matrix row direction. Furthermore, the method can avoid triggering a broadcast mechanism during the process, thereby reducing the hardware computing resource usage of the artificial intelligence chip and further improving the computational efficiency of determining the maximum value in the matrix row direction.
[0041] It should be noted that determining the maximum value in the row direction of a matrix is a core operation in deep learning and is widely used in fields such as image processing, signal processing, and natural language processing. The method provided by the present invention is mainly used to determine the maximum value in the row direction of a target matrix, wherein the target matrix can be determined based on an actual application scenario. For example, when the method for determining the maximum value in the row direction of a matrix provided by the present invention is applied to image processing tasks such as image classification, target detection, or image segmentation, the target matrix can be the input data matrix of the Softmax operator, Batchnorm operator, and Attention operator in the image classification model, target detection model, or image segmentation model.
[0042] Furthermore, the method for determining the maximum value in a matrix row direction provided by the present invention can be executed by an artificial intelligence chip such as a GPU, GPGPU, or TPU. These chips are highly optimized for determining the maximum value in a matrix row direction to accelerate computation. The following briefly describes the structure of the execution element of the present invention, using a GPGPU as an example.
[0043] Figure 2 Schematic diagram of the structure of the universal image processor provided by the present invention, such as Figure 2 As shown in the figure, GPGPU, as a core architecture for high-performance parallel computing, features a hardware design deeply adapted to the demands of large-scale matrix operations. The GPGPU architecture comprises multiple layers of parallel computing units: the bottom layer comprises multiple thread groups, each with its own register file and local cache resources; the middle layer integrates high-bandwidth memory, achieving terabyte / s-level data throughput through 3D stacking technology; and the top layer houses the Gemm Main Buffer (GMB), optimizing data reuse for matrix multiplication and addition operations. This architectural design enables GPGPU to demonstrate significant energy efficiency advantages in data-intensive tasks such as deep learning inference and scientific computing.
[0044] Figure 3 FIG. 1 is a flow chart of the method for determining the maximum value in the matrix row direction provided by the present invention. Figure 3As shown, the method includes the following: Step 301, when a target calculation task for calculating the maximum value of a target matrix in a row direction is triggered, a matrix layout space corresponding to the target matrix is applied for in a general matrix main cache of an artificial intelligence chip, and a vector layout space corresponding to the target matrix is applied for in a high-bandwidth memory of the artificial intelligence chip.
[0045] Specifically, the target matrix is the target of calculation for the matrix row-wise maximum value determination method provided by the present invention. Based on the matrix row-wise maximum value determination method provided by the present invention, the maximum value in each row of the target matrix can be calculated. In embodiments of the present invention, the dimension of the target matrix can be (H, W).
[0046] It is understood that the target matrix in the embodiments of the present invention can be determined based on actual needs. For example, the input data matrix of the Softmax operator, Batchnorm operator, and Attention operator in an image classification model, object detection model, or image segmentation model can be determined as the target matrix. The target matrix is not specifically limited in the embodiments of the present invention.
[0047] In the embodiment of the present invention, a computing task for computing the maximum value in the row direction of a target matrix may be determined as a target computing task.
[0048] In embodiments of the present invention, the target calculation task can be triggered in a variety of ways. For example, the target calculation task can be triggered based on user input. Alternatively, during the calculation process of the neural network model, based on the flow of data between operators, when a matrix row-wise maximum value determination is required, the matrix for which the matrix row-wise maximum value determination is required can be automatically determined as the target matrix, thereby automatically triggering the target calculation task. The specific method for triggering the target calculation task is not limited in embodiments of the present invention.
[0049] When the above-mentioned target technical task is triggered, a matrix layout space (matrix layout) can be applied for the target matrix in the GMB of the artificial intelligence chip as the matrix layout space corresponding to the target matrix. A vector layout space (vector layout) can also be applied for the target matrix in the HBM of the artificial intelligence chip as the vector layout space corresponding to the target matrix.
[0050] It's important to note that the HBM used in AI chips is built on 3D stacked DRAM technology. Multiple DRAMs (Dynamic Random Access Memory) are stacked vertically using through-silicon vias (TSVs) and integrated into the same package with logic chips (such as GPUs / accelerators). HBM has high latency (typically tens of nanoseconds), requiring communication between multiple I / O interfaces and stacked layers over long physical distances. Frequent, small-scale HBM accesses (such as row-by-row reads) significantly amplify HBM latency issues. Furthermore, because the temporary data generated during the matrix row maximum determination process is stored non-contiguously, HBM's wide bus advantage is not fully utilized.
[0051] The GMB of an AI chip typically refers to on-chip or near-processor cache or SRAM (static random access memory, such as a GPU's L2 cache or dedicated memory buffer), which is physically closer to the computing unit. GMB has a smaller storage capacity but a shorter access path, supporting low-latency reads and writes. GMB has extremely low latency (typically a few nanoseconds) and is physically located close to the computing unit, eliminating the need for complex interface protocols. GMB is suitable for operations that require fast responses (such as temporary storage of intermediate values in loops). The low latency of GMB can better support the non-continuous storage of temporary data generated during the determination of the maximum value in the matrix row direction, thereby reducing the read and write of HBM during the determination of the maximum value in the matrix row direction.
[0052] Therefore, in an embodiment of the present invention, a matrix layout space is applied for a target matrix in the GMB of the artificial intelligence chip as the matrix layout space corresponding to the target matrix, which is used to store temporary data generated in the process of calculating the maximum value of the elements in each row of the above target matrix. The number of times the high-bandwidth memory is read and written during the matrix row direction calculation process can be reduced from multiple times to one, thereby improving the computational efficiency of determining the maximum value in the matrix row direction.
[0053] As an optional embodiment, applying for a matrix layout space corresponding to the target matrix in the general matrix main cache of the artificial intelligence chip includes: applying for a matrix layout space in the general matrix main cache of the artificial intelligence chip, whose row dimension is the same as the row dimension of the target matrix and whose column dimension is the same as the number of threads in the thread group.
[0054] After initializing each element in the matrix layout space to negative infinity, the matrix layout space is determined to be the matrix layout space corresponding to the target matrix.
[0055] Figure 4 It is a schematic diagram of the principle of the method for determining the maximum value in the matrix row direction provided by the present invention. Figure 4 The green matrix in the figure is the matrix layout space corresponding to the target matrix in the GMB of the artificial intelligence chip. Figure 4The orange matrix in the figure is the vector layout space corresponding to the target matrix in the HBM of the artificial intelligence chip. Figure 4 The blue area in represents the target matrix.
[0056] like Figure 4 As shown in the figure, when a target computing task for computing the maximum value of the target matrix row direction is triggered, a matrix layout space with a dimension of (H, N) can be requested for the target matrix in the GMB of the AI chip based on the dimension of the target matrix. Where N represents the number of threads in the thread group of the AI chip. Figure 4 N=32 in .
[0057] After applying for a matrix layout space of dimension (H, N) for the target matrix in the GMB of the artificial intelligence chip, each element in the matrix layout space of dimension (H, N) can be initialized to negative infinity, and then the matrix layout space can be determined as the matrix layout space corresponding to the target matrix.
[0058] As an optional embodiment, applying for a matrix layout space corresponding to the target matrix in the general matrix main cache of the artificial intelligence chip includes: applying for a matrix layout space in the general matrix main cache of the artificial intelligence chip whose row dimensions and column dimensions are the same as the row dimensions and column dimensions of the target matrix.
[0059] After initializing each element in the matrix layout space to negative infinity, the matrix layout space is determined to be the matrix layout space corresponding to the target matrix.
[0060] Specifically, when a target calculation task for calculating the maximum value in the row direction of a target matrix is triggered, a matrix layout space with a dimension of (H, W) can be applied for the target matrix in the GMB of the artificial intelligence chip based on the dimension of the target matrix.
[0061] After applying for a matrix layout space of dimension (H, W) for the target matrix in the GMB of the artificial intelligence chip, each element in the matrix layout space of dimension (H, W) can be initialized to negative infinity, and then the matrix layout space can be determined as the matrix layout space corresponding to the target matrix.
[0062] As an optional embodiment, applying for a matrix layout space corresponding to the target matrix in the general matrix main cache of the artificial intelligence chip includes: applying for a matrix layout space in the general matrix main cache of the artificial intelligence chip whose row dimension is the same as the row dimension of the target matrix but the column dimension is 1.
[0063] After initializing each element in the matrix layout space to negative infinity, the matrix layout space is determined to be the matrix layout space corresponding to the target matrix.
[0064] Specifically, when a target calculation task for calculating the maximum value in the row direction of a target matrix is triggered, a matrix layout space with a dimension of (H, 1) can be applied for in the GMB of the artificial intelligence chip based on the dimension of the target matrix.
[0065] After applying for a matrix layout space of dimension (H, W) for the target matrix in the GMB of the artificial intelligence chip, each element in the matrix layout space of dimension (H, 1) can be initialized to negative infinity, and then the matrix layout space can be determined as the matrix layout space corresponding to the target matrix.
[0066] As an optional embodiment, applying for a vector layout space corresponding to the target matrix in the high-bandwidth memory of the artificial intelligence chip includes: applying for a vector layout space in the high-bandwidth memory of the artificial intelligence chip whose row dimension is the same as the row dimension of the target matrix and whose column dimension is 1 as the vector layout space corresponding to the target matrix.
[0067] Specifically, when a target computing task for calculating the maximum value in the row direction of a target matrix is triggered, a vector layout space with a dimension of (H, 1) can be applied for in the HBM of the artificial intelligence chip based on the dimension of the target matrix.
[0068] After applying for a vector layout space with a dimension of (H, 1) for the target matrix in the HBM of the artificial intelligence chip, the vector layout space with a dimension of (H, 1) can be determined as the vector layout space corresponding to the target matrix.
[0069] Step 302: Call multiple thread groups to calculate the maximum value of each row of the target matrix. The thread group calculates the target matrix In the process of finding the maximum value of the elements in the row, Each set of intermediate results calculated by the thread group is overwritten and written into the matrix layout space corresponding to the target matrix. For each element in the row, , Indicates the total number of rows in the target matrix.
[0070] Specifically, after applying for the matrix layout space corresponding to the target matrix in the general matrix main cache of the artificial intelligence chip and the vector layout space corresponding to the target matrix in the high-bandwidth memory of the artificial intelligence chip, the thread groups of multiple artificial intelligence chips can be called, and each thread group calculates the maximum value of the elements in each row of the target matrix.
[0071] In the embodiment of the present invention, Identify each row in the target matrix, , Indicates the total number of rows of the target matrix. In this embodiment of the present invention, the target matrix can be calculated separately. The thread group with the maximum value of the elements in the row is determined as the first Thread groups.
[0072] In the embodiment of the present invention, when calling the first The thread group calculates the target matrix In the process of finding the maximum value of the elements in the row, Each set of intermediate results calculated by the thread group is overwritten and written into the matrix layout space corresponding to the target matrix. Each element in the row can store the temporary data generated in the process of determining the maximum value in the matrix row direction into the GMB of the artificial intelligence chip, thereby reducing the number of times the high-bandwidth memory is read and written during the matrix row direction calculation process.
[0073] As an optional embodiment, in calling the first The thread group calculates the target matrix In the process of finding the maximum value of the elements in the row, Each set of intermediate results calculated by the thread group is overwritten and written into the matrix layout space corresponding to the target matrix. Each element in the row includes: calling Each thread in the thread group reads the target matrix No. Each element in the element group, A positive integer that starts at 0. The maximum value is based on The number of threads in the thread group is determined, and the target matrix Each element in the row is based on the The number of threads in a thread group is divided into multiple element groups in order from front to back.
[0074] Specifically, in the embodiment of the present invention, the The number of threads in the thread group, in order from front to back, the target matrix The elements in a row are divided into groups of elements.
[0075] In the embodiment of the present invention, it can be used to represent Indicates the The number of threads in a thread group. Typically, the number of threads in a thread group is 32.
[0076] In the embodiment of the present invention, Identify the target matrix The value of the element in the row, The maximum value of the target matrix is The number of element groups in the row The target matrix The number of element groups in the row for Round up.
[0077] It is understandable that in In the case of No. The number of elements in an element group is , for example, the target matrix The first element group in the row includes element T0, element T1, element T2, ..., element T30 and element T31; In the case of No. The number of elements in an element group can be , the target matrix No. The number of elements in an element group can also be less than .
[0078] It should be noted that the number of threads in any two thread groups in the artificial intelligence chip in the embodiment of the present invention is equal.
[0079] Get the target matrix No. The maximum value among the elements in a group of elements.
[0080] Specifically, call Each thread in the thread group reads the target matrix No. After each element in the element group, the target matrix can be obtained by comparison. No. The maximum value among all elements in the element group is recorded as .
[0081] Based on the target matrix No. The maximum value among the elements in the element group, get the The first thread group calculates An intermediate result.
[0082] Specifically, get the target matrix No. The maximum value among all elements in the element group Afterwards, the first The first thread group calculates An intermediate result.
[0083] As an optional embodiment, based on the target matrix No. The maximum value among the elements in the element group, get the The first thread group calculates Intermediate results include: reading the first Each element in the row. It can be understood that since the target matrix corresponds to the matrix layout space The initial value of each element in the row is negative infinity. In subsequent calculations, the first The first thread group calculates The intermediate result is written over the matrix layout space corresponding to the target matrix. Each element of the row. Therefore, the target matrix corresponds to the matrix layout space Each element in the row is the same. In the embodiment of the present invention, the first element in the matrix layout space corresponding to the target matrix can be Each element in the row is recorded as .
[0084] In the embodiment of the present invention, the first matrix in the matrix layout space corresponding to the target matrix can be directly read from the GMB of the artificial intelligence chip. Each element in the row can be processed without triggering the broadcast mechanism, which can reduce the hardware computing resource occupation of the artificial intelligence chip and improve the computational efficiency of matrix row-wise calculations.
[0085] In the embodiment of the present invention, the first matrix in the matrix layout space corresponding to the target matrix is read from the GMB of the artificial intelligence chip. Each element in a row can be combined into a load.
[0086] Compare the target matrix to the matrix layout space corresponding to the Each element in the row and the target matrix No. The maximum value among the elements in a group of elements. Specifically, obtain the first matrix in the matrix layout space corresponding to the target matrix Each element in the row Afterwards, we can compare the target matrix with the matrix layout space corresponding to the first Each element in the row and the target matrix No. The maximum value among all elements in the element group ,Sure and The larger of .
[0087] The matrix layout space corresponding to the target matrix Each element in the row and the target matrix No. The largest of the maximum values of the elements in the element group is determined as the The first thread group calculates An intermediate result.
[0088] Specifically, determine and After the larger of and The larger one is determined as The first thread group calculates An intermediate result.
[0089] The first The first thread group calculates The intermediate result is written over the matrix layout space corresponding to the target matrix. Each element of the row. Specifically, obtain the The first thread group calculates After the intermediate result, the The first thread group calculates The intermediate result is written over the matrix layout space corresponding to the target matrix. Each element in the row , thus achieving Updates.
[0090] In the embodiment of the present invention, The first thread group calculates The intermediate result is written over the matrix layout space corresponding to the target matrix. Each element in the row Can be combined into one store.
[0091] In determining When the maximum value is reached, determine the The thread group completes the target matrix The maximum value of the elements in the row is calculated in If the maximum value is not reached, Increase by 1 and return to the execution call Each thread in the thread group reads the target matrix No. Steps for each element in the element group until the Reached maximum value.
[0092] Specifically, the The first thread group calculates The intermediate result is written over the matrix layout space corresponding to the target matrix. Each element in the row Afterwards, you can judge Whether the maximum value has been reached.
[0093] In determining When the maximum value is reached, determine the The thread group completes the target matrix Computes the maximum value of elements in a row.
[0094] In determining If the maximum value is not reached, Increase by 1 and return to the execution call Each thread in the thread group reads the target matrix No. Steps for each element in the element group until the Reached maximum value.
[0095] Step 303: After determining The thread group completes the target matrix In the case of calculating the maximum value of the elements in a row, the matrix layout space corresponding to the target matrix is Any element in the row is overwritten in the vector layout space corresponding to the target matrix. row, get the target matrix The maximum value of the elements in the row.
[0096] Specifically, in determining The thread group completes the target matrix In the case of calculating the maximum value of the elements in a row, the matrix layout space corresponding to the target matrix can be Any element in the row covers the first element in the vector layout space corresponding to the target matrix in the HBM written into the artificial intelligence chip. OK.
[0097] In the embodiment of the present invention, the vector layout space corresponding to the target matrix can be The elements in a row are denoted by The first vector layout space corresponding to the target matrix in the HBM of the artificial intelligence chip Elements in a row , which is the target matrix The maximum value of the elements in the row.
[0098] The embodiment of the present invention applies for a matrix layout space corresponding to the target matrix in the general matrix main cache of the artificial intelligence chip and applies for a vector layout space corresponding to the target matrix in the high bandwidth memory of the artificial intelligence chip when a target calculation task for calculating the maximum value in the row direction of the target matrix is triggered, and then calls multiple thread groups to calculate the maximum value of the elements in each row of the target matrix respectively. The thread group calculates the target matrix In the process of finding the maximum value of the elements in the row, Each set of intermediate results calculated by the thread group is overwritten and written into the matrix layout space corresponding to the target matrix. For each element in the row, The thread group completes the target matrix In the case of calculating the maximum value of the elements in a row, the matrix layout space corresponding to the target matrix is Any element in the row is overwritten in the vector layout space corresponding to the target matrix. row, get the target matrix The maximum value of the elements in the row can avoid triggering the broadcast mechanism in the process of determining the maximum value in the matrix row direction, thereby reducing the hardware computing resource occupation of the artificial intelligence chip and improving the computational efficiency of determining the maximum value in the matrix row direction. It can utilize the low latency characteristics of the general matrix main cache of the artificial intelligence chip to store the temporary data generated in the process of determining the maximum value in the matrix row direction to the general matrix main cache of the artificial intelligence chip, which can significantly reduce the number of times high-bandwidth memory is read and written during the calculation process in the matrix row direction, thereby further improving the computational efficiency of determining the maximum value in the matrix row direction.
[0099] Taking the Attention operator in the target detection model as an example, the maximum value in the matrix row direction is determined based on the matrix row maximum value determination method provided by the present invention. The above-mentioned Attention operator can achieve a computational efficiency improvement of up to 10%.
[0100] Figure 5 This is a schematic diagram of the structure of the device for determining the maximum value in the matrix row direction provided by the present invention. Figure 5 The matrix row direction maximum value determination device provided by the present invention is described. The matrix row direction maximum value determination device described below and the matrix row direction maximum value determination method provided by the present invention described above can be referred to each other. Figure 5 As shown, the device includes: a space application module 501, a numerical calculation module 502 and a result output module 503.
[0101] Space request module 501 is used to request matrix layout space corresponding to the target matrix in the general matrix main cache of the AI chip and vector layout space corresponding to the target matrix in the high-bandwidth memory of the AI chip when a target computing task for computing the maximum value of the target matrix row direction is triggered. The numerical calculation module 502 is used to call multiple thread groups to calculate the maximum value of each row of the target matrix. The thread group calculates the target matrix In the process of finding the maximum value of the elements in the row, Each set of intermediate results calculated by the thread group is overwritten and written into the matrix layout space corresponding to the target matrix. For each element in the row, , Indicates the total number of rows in the target matrix. The result output module 503 is used to determine the The thread group completes the target matrix In the case of calculating the maximum value of the elements in a row, the matrix layout space corresponding to the target matrix is Any element in the row is overwritten in the vector layout space corresponding to the target matrix. row, get the target matrix The maximum value of the elements in the row.
[0102] The matrix row direction maximum value determination device in the embodiment of the present invention, when triggering a target calculation task for calculating the maximum value of the target matrix row direction, applies for a matrix layout space corresponding to the target matrix in the general matrix main cache of the artificial intelligence chip, applies for a vector layout space corresponding to the target matrix in the high bandwidth memory of the artificial intelligence chip, and then calls multiple thread groups to respectively calculate the maximum value of the elements in each row of the target matrix, and then calls the first thread group in each thread group. The thread group calculates the target matrix In the process of finding the maximum value of the elements in the row, Each set of intermediate results calculated by the thread group is overwritten and written into the matrix layout space corresponding to the target matrix. For each element in the row, The thread group completes the target matrix In the case of calculating the maximum value of the elements in a row, the matrix layout space corresponding to the target matrix is Any element in the row is overwritten in the vector layout space corresponding to the target matrix. row, get the target matrix The maximum value of the elements in the row can avoid triggering the broadcast mechanism in the process of determining the maximum value in the matrix row direction, thereby reducing the hardware computing resource occupation of the artificial intelligence chip and improving the computational efficiency of determining the maximum value in the matrix row direction. It can utilize the low latency characteristics of the general matrix main cache of the artificial intelligence chip to store the temporary data generated in the process of determining the maximum value in the matrix row direction to the general matrix main cache of the artificial intelligence chip, which can significantly reduce the number of times high-bandwidth memory is read and written during the calculation process in the matrix row direction, thereby further improving the computational efficiency of determining the maximum value in the matrix row direction.
[0103] As an optional embodiment, the space application module 501 is specifically used to apply for a matrix layout space in the general matrix main cache of the artificial intelligence chip, whose row dimension is the same as the row dimension of the target matrix and the column dimension is the same as the number of threads in the thread group; after initializing each element in the matrix layout space to negative infinity, the matrix layout space is determined as the matrix layout space corresponding to the target matrix.
[0104] As an optional embodiment, the space application module 501 is specifically used to apply for a matrix layout space in the general matrix main cache of the artificial intelligence chip, whose row and column dimensions are the same as the row and column dimensions of the target matrix; after initializing each element in the matrix layout space to negative infinity, the matrix layout space is determined as the matrix layout space corresponding to the target matrix.
[0105] As an optional embodiment, the space application module 501 is specifically used to apply for a matrix layout space in the general matrix main cache of the artificial intelligence chip, whose row dimension is the same as the row dimension of the target matrix but the column dimension is 1; after initializing each element in the matrix layout space to negative infinity, the matrix layout space is determined as the matrix layout space corresponding to the target matrix.
[0106] As an optional embodiment, the space application module 501 is specifically used to apply for a vector layout space in the high-bandwidth memory of the artificial intelligence chip, whose row dimension is the same as the row dimension of the target matrix and the column dimension is 1; after initializing each element in the vector layout space to negative infinity, the vector layout space is determined as the vector layout space corresponding to the target matrix.
[0107] As an optional embodiment, the numerical calculation module 502 is specifically used to call the Each thread in the thread group reads the target matrix No. Each element in the element group, A positive integer that starts at 0. The maximum value is based on The number of threads in the thread group is determined, and the target matrix Each element in the row is based on the The number of threads in a thread group is divided into multiple element groups in order from front to back; get the target matrix No. The maximum value of each element in the element group; based on the target matrix No. The maximum value among the elements in the element group, get the The first thread group calculates an intermediate result; The first thread group calculates The intermediate result is written over the matrix layout space corresponding to the target matrix. Each element of the row; in determining When the maximum value is reached, determine the The thread group completes the target matrix The maximum value of the elements in the row is calculated in If the maximum value is not reached, Increase by 1 and return to the execution call Each thread in the thread group reads the target matrix No. Steps for each element in the element group until the Reached maximum value.
[0108] As an optional embodiment, the numerical calculation module 502 is further specifically configured to read the first Each element in the row; compare the target matrix with the corresponding matrix layout space Each element in the row and the target matrix No. The maximum value of each element in the element group; the matrix layout space corresponding to the target matrix Each element in the row and the target matrix No. The largest of the maximum values of the elements in the element group is determined as the The first thread group calculates An intermediate result.
[0109] Figure 6 An example of a physical structure diagram of an electronic device is shown below. Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a method for determining the maximum value in the row direction of a matrix. The method includes: when a target calculation task for calculating the maximum value in the row direction of a target matrix is triggered, applying for a matrix layout space corresponding to the target matrix in the general matrix main cache of the artificial intelligence chip, and applying for a vector layout space corresponding to the target matrix in the high-bandwidth memory of the artificial intelligence chip; calling multiple thread groups to respectively calculate the maximum value of the elements in each row of the target matrix, and in calling the first thread group in each thread group, the maximum value of the elements in each row of the target matrix is respectively calculated. The thread group calculates the target matrix In the process of finding the maximum value of the elements in the row, Each set of intermediate results calculated by the thread group is overwritten and written into the matrix layout space corresponding to the target matrix. For each element in the row, , Indicates the total number of rows of the target matrix; The thread group completes the target matrix In the case of calculating the maximum value of the elements in a row, the matrix layout space corresponding to the target matrix is Any element in the row is overwritten in the vector layout space corresponding to the target matrix. row, get the target matrix The maximum value of the elements in the row.
[0110] Furthermore, the logic instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0111] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the matrix row direction maximum value determination method provided by the above methods, the method including: when triggering a target calculation task for calculating the maximum value of the target matrix row direction, applying for a matrix layout space corresponding to the target matrix in the general matrix main cache of the artificial intelligence chip, and applying for a vector layout space corresponding to the target matrix in the high-bandwidth memory of the artificial intelligence chip; calling multiple thread groups to respectively calculate the maximum value of the elements in each row of the target matrix, and in calling the first thread group in each thread group, The thread group calculates the target matrix In the process of finding the maximum value of the elements in the row, Each set of intermediate results calculated by the thread group is overwritten and written into the matrix layout space corresponding to the target matrix. For each element in the row, , Indicates the total number of rows of the target matrix; The thread group completes the target matrix In the case of calculating the maximum value of the elements in a row, the matrix layout space corresponding to the target matrix is Any element in the row is overwritten in the vector layout space corresponding to the target matrix. row, get the target matrix The maximum value of the elements in the row.
[0112] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the matrix row direction maximum value determination method provided by the above methods, the method comprising: in the case of triggering a target calculation task for calculating the maximum value of a target matrix row direction, applying for a matrix layout space corresponding to the target matrix in the general matrix main cache of the artificial intelligence chip, and applying for a vector layout space corresponding to the target matrix in the high-bandwidth memory of the artificial intelligence chip; calling multiple thread groups to respectively calculate the maximum value of elements in each row of the target matrix, and in calling the first thread group in each thread group, The thread group calculates the target matrix In the process of finding the maximum value of the elements in the row, Each set of intermediate results calculated by the thread group is overwritten and written into the matrix layout space corresponding to the target matrix. For each element in the row, , Indicates the total number of rows of the target matrix; The thread group completes the target matrix In the case of calculating the maximum value of the elements in a row, the matrix layout space corresponding to the target matrix is Any element in the row is overwritten in the vector layout space corresponding to the target matrix. row, get the target matrix The maximum value of the elements in the row.
[0113] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.
[0114] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.
[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.
Claims
1. A method for determining the maximum value in a matrix row direction, characterized in that: include: When a target computing task for computing the maximum value of a target matrix in a row direction is triggered, a matrix layout space corresponding to the target matrix is requested in a general matrix main cache of the artificial intelligence chip, and a vector layout space corresponding to the target matrix is requested in a high-bandwidth memory of the artificial intelligence chip; Call multiple thread groups to calculate the maximum value of each row of the target matrix, and then call the first The thread group calculates the target matrix In the process of finding the maximum value of the elements in the row, Each set of intermediate results calculated by the thread group is overwritten and written into the matrix layout space corresponding to the target matrix. For each element in the row, , Represents the total number of rows of the target matrix; In determining the The thread group completes the target matrix In the case of calculating the maximum value of the elements in a row, the matrix layout space corresponding to the target matrix is Any element in the row overwrites the first element in the vector layout space corresponding to the target matrix. row, obtain the target matrix The maximum value of the elements in the row.
2. The method for determining the maximum value in a matrix row direction according to claim 1, wherein: In calling the first The thread group calculates the target matrix In the process of finding the maximum value of the elements in the row, Each set of intermediate results calculated by the thread group is overwritten and written into the matrix layout space corresponding to the target matrix. Each element in the row includes: Call the Each thread in the thread group reads the target matrix No. Each element in the element group, A positive integer that starts at 0. The maximum value is based on the The number of threads in the thread group is determined by the target matrix Each element in the row is based on the The number of threads in a thread group is divided into multiple element groups in order from front to back; Get the target matrix No. The maximum value among all elements in the element group; Based on the target matrix No. The maximum value of each element in the element group, get the The first thread group calculates intermediate results; The said The first thread group calculates The intermediate result is overwritten and written into the matrix layout space corresponding to the target matrix. Each element of the row; In determining When the maximum value is reached, the The thread group completes the target matrix The maximum value of the elements in the row is calculated in If the maximum value is not reached, Increase by 1, return to execute the call Each thread in the thread group reads the target matrix No. Steps for each element in the element group until the Reached maximum value.
3. The method for determining the maximum value in a matrix row direction according to claim 2, wherein: The target matrix No. The maximum value of each element in the element group, get the The first thread group calculates Intermediate results include: Read the matrix layout space corresponding to the target matrix Each element in the row; Compare the matrix layout space corresponding to the target matrix Each element in the row and the target matrix No. The maximum value among all elements in the element group; The matrix layout space corresponding to the target matrix Each element in the row and the target matrix No. The largest of the maximum values of the elements in the element group is determined as the The first thread group calculates An intermediate result.
4. The method for determining the maximum value in a matrix row direction according to claim 1, wherein: The applying for a matrix layout space corresponding to the target matrix in the general matrix main cache of the artificial intelligence chip includes: Applying for a matrix layout space in the general matrix main cache of the artificial intelligence chip, the matrix layout space having the same row dimension as the target matrix and the same column dimension as the number of threads in the thread group; After initializing each element in the matrix layout space to negative infinity, the matrix layout space is determined as the matrix layout space corresponding to the target matrix.
5. The method for determining the maximum value in a matrix row direction according to claim 1, wherein: The applying for a matrix layout space corresponding to the target matrix in the general matrix main cache of the artificial intelligence chip includes: Applying for a matrix layout space in the general matrix main cache of the artificial intelligence chip, the matrix layout space having the same row and column dimensions as the target matrix; After initializing each element in the matrix layout space to negative infinity, the matrix layout space is determined as the matrix layout space corresponding to the target matrix.
6. The method for determining the maximum value in a matrix row direction according to claim 1, wherein: The applying for a matrix layout space corresponding to the target matrix in the general matrix main cache of the artificial intelligence chip includes: Applying for a matrix layout space in the general matrix main cache of the artificial intelligence chip, the matrix layout space having the same row dimension as the target matrix but a column dimension of 1; After initializing each element in the matrix layout space to negative infinity, the matrix layout space is determined as the matrix layout space corresponding to the target matrix.
7. The method for determining the maximum value in a matrix row direction according to any one of claims 1 to 5, characterized in that: The applying for a vector layout space corresponding to the target matrix in the high-bandwidth memory of the artificial intelligence chip includes: Apply for a vector layout space in the high-bandwidth memory of the artificial intelligence chip, whose row dimension is the same as the row dimension of the target matrix and whose column dimension is 1, as the vector layout space corresponding to the target matrix.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for determining the maximum value in the matrix row direction according to any one of claims 1 to 7 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for determining the maximum value in a matrix row direction according to any one of claims 1 to 7 is implemented.
10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for determining the maximum value in a matrix row direction according to any one of claims 1 to 7 is implemented.