Operator execution method and device, equipment, storage medium and program product
By employing a general tensor computation method in artificial intelligence chips, the target tensor is expanded to a second dimension and the data is rearranged, solving the problem of needing two computation schemes in existing technologies, and achieving efficient adaptation and cost reduction in different memory scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI BIREN TECH CO LTD
- Filing Date
- 2026-01-14
- Publication Date
- 2026-04-28
AI Technical Summary
Existing technologies require two sets of computation schemes when processing four-dimensional and five-dimensional tensors, resulting in high development and maintenance costs and an inability to adapt to the business scenarios of different memory types.
A general tensor computation method is adopted to expand the target tensor to a second dimension and rearrange the data in the cache. By expanding and rearranging, tensors of different dimensions can be adapted to various business scenarios, avoiding separate development for different memory requirements.
It reduces development and maintenance costs, can adapt to various business scenarios with different memory types, takes into account tensor hyperdimensional cases, and improves computational efficiency.
Smart Images
Figure CN121934889A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence chip technology, and in particular to an operator execution method, apparatus, device, storage medium, and program product. Background Technology
[0002] Currently, reordering technology rearranges and moves tensor data based on the physical layout and shape of the input tensor (src tensor) and output tensor (dst tensor).
[0003] In related technologies, for example, in the scenario of activation layout for large model training, the business scenario is divided into four-dimensional tensors (Activation 4D) and five-dimensional tensors (Activation 5D). When rearranging and moving the above four-dimensional tensors or five-dimensional tensors, two sets of computing schemes are usually used to process the above two types of tensors to meet the actual business needs. Developing and maintaining two sets of computing schemes consumes a lot of time and resources. Summary of the Invention
[0004] This application provides an operator execution method, apparatus, device, storage medium, and program product for using a general tensor calculation method when rearranging and transporting four-dimensional or five-dimensional tensors of artificial intelligence chips. This method ensures that the second dimension does not change with the business scenario, and can transform tensors of different dimensions into the second dimension. It covers multiple business scenarios in different memories, takes into account the case of tensors with higher dimensions, and eliminates the need to develop separate conversion calculations between various types of tensors for different memory requirements, thus reducing development and maintenance costs.
[0005] On one hand, embodiments of this application provide an operator execution method, the method comprising: The first dimension of the target tensor output by the first operator is obtained and stored in the first buffer. The target tensor has the same arrangement type but different dimensions in different business scenarios. When the size of the target tensor in any specified dimension exceeds the upper limit of the size supported by the chip for the arrangement type in the specified dimension, the target tensor is expanded to a second dimension in the first buffer according to the rule that the size of the target tensor in multiple specified dimensions is not greater than the upper limit of the size corresponding to each specified dimension. The second dimension is the same value in different business scenarios. Based on the target tensor of the second dimension, the data is rearranged in the first buffer to obtain the rearranged target tensor; Based on the dimensions of the first-dimensional target tensor in various dimensions, the dimensions of the rearranged target tensor in various dimensions, and the starting coordinates of the first-dimensional target tensor in the first buffer, the write coordinates of the rearranged target tensor in the second buffer are determined. A tensor retrieval request is sent to the first buffer, and the target tensor returned by the tensor retrieval request is written to the second buffer according to the write coordinates; the target tensor in the second buffer is used for calculation by the second operator.
[0006] Optionally, the multiple specified dimensions include C dimension, H dimension and W dimension, wherein C dimension represents the channel dimension of the target tensor, H dimension represents the height dimension of the target tensor and W dimension represents the width dimension of the target tensor; Expanding the target tensor to a second dimension according to the rule that the size of the target tensor in multiple specified dimensions is not greater than the upper limit of the size corresponding to each specified dimension includes: For each specified dimension among C, H, and W, perform the following steps: If the size of the target tensor in the specified dimension exceeds the upper limit of the chip's supported size in the specified dimension, then the data of the target tensor in the specified dimension is split according to the size in the specified dimension and the upper limit of the size; the size of the target tensor in the specified dimension is updated to the size of the split target tensor in the specified dimension; and the number of data parts of the split target tensor in the specified dimension is used as the size of the target tensor in a new dimension corresponding to the specified dimension. If the size of the target tensor in the specified dimension does not exceed the upper limit of the size supported by the chip in the specified dimension, then the number of data parts of the target tensor in the specified dimension is taken as the size of the target tensor in a new dimension corresponding to the specified dimension.
[0007] Optionally, the target tensor of the first dimension further includes a D dimension and an N dimension, wherein the D dimension represents the depth dimension of the target tensor, and the N dimension represents the batch dimension of the target tensor, and further includes: When the size of the target tensor in the D dimension exceeds the upper limit of the size supported by the chip in the D dimension, the data of the target tensor in the D dimension is adjusted to the N dimension.
[0008] Optionally, data rearrangement is performed in the first buffer based on the target tensor of the second dimension, including: In the first buffer, the data of the second-dimensional target tensor in the new dimension is mapped to the N-dimensional dimension to obtain a third-dimensional target tensor, wherein the third-dimensional target tensor differs from the first-dimensional target tensor in size in at least one dimension; and Data rearrangement is performed on the target tensor of the third dimension.
[0009] Optionally, before performing data rearrangement on the target tensor of the third dimension, the method further includes: When the size of the target tensor in the third dimension exceeds the upper limit of the chip's size in the N dimension, the target tensor is split in the N dimension to obtain multiple sub-tensors in the third dimension.
[0010] Optionally, the first dimension of the target tensor in different business scenarios is four-dimensional or five-dimensional, and it further includes: If the first dimension is four-dimensional, then the third dimension is four-dimensional; If the first dimension is five-dimensional, then the third dimension is four-dimensional.
[0011] On one hand, embodiments of this application provide an operator execution apparatus, the apparatus comprising: The tensor acquisition module is configured to acquire the target tensor of the first dimension output by the first operator to the first buffer. The target tensor has the same arrangement type but different dimensions in different business scenarios. The tensor expansion module is configured to expand the target tensor to a second dimension in the first buffer according to the rule that the size of the target tensor in multiple specified dimensions is not greater than the upper limit of the size corresponding to each specified dimension when the size of the target tensor in any specified dimension exceeds the upper limit of the size supported by the chip for the arrangement type in the specified dimension. The second dimension is the same value in different business scenarios. The tensor rearrangement module is configured to rearrange data in the first buffer to obtain the rearranged target tensor. The coordinate transformation module is configured to determine the write coordinates of the rearranged target tensor in the second cache based on the size of the first-dimensional target tensor in various dimensions, the size of the rearranged target tensor in various dimensions, and the starting coordinates of the first-dimensional target tensor in the first cache. The tensor storage module is configured to send a tensor retrieval request to the first buffer, and write the target tensor returned by the tensor retrieval request into the second buffer according to the write coordinates; the target tensor in the second buffer is used for calculation by the second operator.
[0012] Optionally, the multiple specified dimensions include C dimension, H dimension and W dimension, wherein C dimension represents the channel dimension of the target tensor, H dimension represents the height dimension of the target tensor and W dimension represents the width dimension of the target tensor; According to the rule that the size of the target tensor in multiple specified dimensions is no greater than the upper limit of the size corresponding to each specified dimension, the target tensor is expanded to a second dimension. The tensor expansion module is specifically configured as follows: For each specified dimension among C, H, and W, perform the following steps: If the size of the target tensor in the specified dimension exceeds the upper limit of the chip's supported size in the specified dimension, then the data of the target tensor in the specified dimension is split according to the size in the specified dimension and the upper limit of the size; the size of the target tensor in the specified dimension is updated to the size of the split target tensor in the specified dimension; and the number of data parts of the split target tensor in the specified dimension is used as the size of the target tensor in a new dimension corresponding to the specified dimension. If the size of the target tensor in the specified dimension does not exceed the upper limit of the size supported by the chip in the specified dimension, then the number of data parts of the target tensor in the specified dimension is taken as the size of the target tensor in a new dimension corresponding to the specified dimension.
[0013] Optionally, the target tensor of the first dimension further includes a D dimension and an N dimension, wherein the D dimension represents the depth dimension of the target tensor, and the N dimension represents the batch dimension of the target tensor. The tensor expansion module is further configured to: When the size of the target tensor in the D dimension exceeds the upper limit of the size supported by the chip in the D dimension, the data of the target tensor in the D dimension is adjusted to the N dimension.
[0014] Optionally, data rearrangement is performed in the first buffer based on the target tensor of the second dimension, and the tensor rearrangement module is specifically configured as follows: In the first buffer, the data of the second-dimensional target tensor in the new dimension is mapped to the N-dimensional dimension to obtain a third-dimensional target tensor, wherein the third-dimensional target tensor differs from the first-dimensional target tensor in size in at least one dimension; and Data rearrangement is performed on the target tensor of the third dimension.
[0015] Optionally, before performing data rearrangement on the target tensor of the third dimension, the tensor rearrangement module is further configured to: When the size of the target tensor in the third dimension exceeds the upper limit of the chip's size in the N dimension, the target tensor is split in the N dimension to obtain multiple sub-tensors in the third dimension.
[0016] Optionally, the first dimension of the target tensor in different business scenarios is four-dimensional or five-dimensional, and it further includes: If the first dimension is four-dimensional, then the third dimension is four-dimensional; If the first dimension is five-dimensional, then the third dimension is four-dimensional.
[0017] On one hand, embodiments of this application provide a computer device, including a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip. When the artificial intelligence chip executes the computer program, it implements the steps of the above-described operator execution method.
[0018] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the above-described operator execution method.
[0019] On one hand, embodiments of this application provide a computer program product, the computer program product including a computer program stored on a computer-readable storage medium, the computer program including program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described operator execution method.
[0020] In this embodiment, the target tensor of the first dimension is expanded to the second dimension, and the data is rearranged according to the target tensor of the second dimension. A general tensor calculation method is adopted so that the second dimension does not change with the change of business scenario. It can transform tensors of different dimensions into the second dimension, covering multiple business scenarios in different memory, taking into account the case of tensor hyperdimensionality, and eliminating the need to develop separate conversion calculations between various types of tensors with different memory requirements, thus reducing development and maintenance costs. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 This is a schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of this application; Figure 2 A flowchart illustrating an operator execution method provided in an embodiment of this application; Figure 3 A schematic diagram illustrating the process of expanding the target tensor to a second dimension, provided for an embodiment of this application; Figure 4 A schematic diagram illustrating the process of data rearrangement based on a target tensor of a second dimension, provided for an embodiment of this application; Figure 5 This is a schematic diagram of the structure of the operator execution device 600 provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0023] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0024] refer to Figure 1 This is a structural diagram of an artificial intelligence chip applicable to the embodiments of this application. The artificial intelligence chip 100 includes at least: a video memory 101 and multiple programmable multiprocessors 102. The programmable multiprocessors 102 can be a streaming processing cluster (SPC). Each programmable multiprocessor 102 includes multiple execution units 103.
[0025] Each execution unit 103 includes: on-chip cache 104, register 105, and thread group 106. The thread group 106 is also called thread bundle, which includes multiple threads.
[0026] Video memory 101 can be high-bandwidth memory (HBM) or other types of memory. On-chip cache 104 is a temporary memory with a smaller capacity than video memory 101, but a faster data exchange speed.
[0027] Compared to the on-chip cache 104, the register 105 has a smaller capacity but a faster data exchange speed.
[0028] In addition to the structure described above, the artificial intelligence chip 100 in this application may also include other structures, and this application does not specifically limit such structures.
[0029] Artificial intelligence chips 100 can be: Graphics Processing Unit (GPU), General-purpose computing on graphics processing units (GPGPU), Domain Specific Architecture (DSA), etc.
[0030] This application is based on Figure 1 The diagram of the AI chip architecture shown illustrates an operator execution method applicable to various scenarios, such as image processing, speech processing, and text processing. The physical meaning of the target tensor can differ across these scenarios.
[0031] For example, in a text processing scenario, the target tensor can be text data used in tasks such as text generation and text recognition.
[0032] For example, in a speech processing scenario, the target tensor can be the speech data used in tasks such as speech enhancement, speech recognition, and speech synthesis.
[0033] For example, in image processing scenarios, the target tensor can be image data used in tasks such as image preprocessing, image segmentation, and object detection.
[0034] For example, in video processing scenarios, the target tensor can be video data used in tasks such as video preprocessing, video segmentation, and object detection.
[0035] It should be noted that in this application, "first" and "second" are used for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features indicated.
[0036] The following is based on Figure 1 The diagram shown illustrates the architecture of an AI chip and details the flow of an operator execution method. (See also...) Figure 2 This method is executed by a computer device that contains an artificial intelligence chip, which can be... Figure 1 The artificial intelligence chip 100 shown herein includes the following steps: Step 201: Obtain the target tensor of the first dimension output by the first operator into the first buffer. The target tensor has the same arrangement type but different dimensions in different business scenarios.
[0037] For example, the layout type of the target tensor in different business scenarios can be ActivationLayout. The target tensor can be Activation 4D or Activation 5D, with dimensions of 4 and 5 respectively. For example, a 4-dimensional tensor is denoted as [N, C, H, W], and a 5-dimensional tensor is denoted as [N, C, D, H, W].
[0038] Step 202: When the size of the target tensor in any specified dimension exceeds the upper limit of the size supported by the chip for the arrangement type in the specified dimension, the target tensor is expanded to the second dimension in the first buffer according to the rule that the size of the target tensor in multiple specified dimensions is not greater than the upper limit of the size corresponding to each specified dimension. The second dimension is the same value in different business scenarios.
[0039] In some embodiments, the multiple specified dimensions include C dimension, H dimension and W dimension, where C dimension represents the channel dimension of the target tensor, H dimension represents the height dimension of the target tensor, and W dimension represents the width dimension of the target tensor. The number of two dimensions is the same in different business scenarios, which is 7 dimensions.
[0040] Specifically, following the rule that the size of the target tensor in multiple specified dimensions is no greater than the upper limit of the size corresponding to each specified dimension, the target tensor is expanded to a second dimension. See [link to relevant documentation]. Figure 3 This includes the following steps: Step 301: For each specified dimension among C, H, and W, perform the following steps: Step 302: If the size of the target tensor in the specified dimension exceeds the upper limit of the size supported by the chip in the specified dimension, then the data of the target tensor in the specified dimension is split according to the size in the specified dimension and the upper limit of the size; the size of the target tensor in the specified dimension is updated to the size of the split target tensor in the specified dimension; and the number of data parts of the split target tensor in the specified dimension is used as the size of the target tensor in a new dimension.
[0041] For example, the target tensor in the first dimension is a 4-dimensional tensor, denoted as [N, C, H, W], with specific shape dimensions of [100, 100, 100, 100]. The chip supports size limits of 50, 50, and 50 in the specified dimensions C, H, and W, respectively. Therefore, the target tensor's dimensions in the specified dimensions C, H, and W all exceed the chip's size limits for the activation layout in the specified dimensions C, H, and W. Taking C as an example, for each specified dimension (C, H, and W), the data in dimension C is split, resulting in 50 data points. 2. After splitting, the size of the C dimension is 50, and the number of data parts in the C dimension is 2. By analogy, the target tensor of the second dimension is obtained, denoted as [N, subC, subH, subW, baseC, baseH, baseW], with a specific shape size of [100, 2, 2, 2, 50, 50, 50].
[0042] Step 303: If the size of the target tensor in the specified dimension does not exceed the upper limit of the size supported by the chip in the specified dimension, then the number of data copies of the target tensor in the specified dimension is used as the size of the target tensor in a new dimension corresponding to the specified dimension.
[0043] Alternatively, in other examples, the target tensor in the first dimension is a 4-dimensional tensor, denoted as [N, C, H, W], with specific shape dimensions of [100, 50, 50, 100]. The chip supports upper size limits of 50, 50, and 50 in the specified dimensions C, H, and W, respectively. Therefore, if the target tensor's size in the specified dimension W exceeds the chip's upper size limit for the activation layout in the specified dimension W, the processing for dimension W is the same as step 302 above. If the target tensor's size in the specified dimensions C and H does not exceed the chip's upper size limit for the activation layout in the specified dimensions C and H, for dimensions C and H, taking dimension H as an example, dimension H can be denoted as 50. 1. The number of data in the H dimension is 1. The processing of the W dimension is the same as above. And so on, the specific shape size of the target tensor of the second dimension is [100, 1, 1, 2, 50, 50, 50].
[0044] It should be noted that there may be cases where the size of a specified dimension is not an integer multiple of the upper limit. In such cases, this embodiment calculates the amount of data that needs to be padded with zeros during tensor expansion. The final target tensor of the second dimension satisfies the condition that padding no longer increases and baseH is as large as possible. For example, if the upper limit of the H dimension is 1024, and H=4000 in the target tensor is already out of dimensionality, zeros are padded to make H=4096. Then, the tensor is split, resulting in H=4. If the value is 1024, then the target tensor after splitting will have baseH=1024 and subH=4.
[0045] When splitting (or folding) a target tensor, there is a lower bound on the size of the target tensor in a specified dimension after splitting, such as baseH being greater than or equal to 16. In some other scenarios, when the maximum size baseH cannot be met and the padding does not decrease, the lower bound of baseH is used to split the target tensor.
[0046] In some embodiments, the dimensions of the target tensor of the first dimension also include D dimension and N dimension, wherein D dimension represents the depth dimension of the target tensor and N dimension represents the batch dimension of the target tensor. When the size of the target tensor in the D dimension exceeds the upper limit of the size supported by the chip in the D dimension, the data of the target tensor in the D dimension is adjusted to the N dimension.
[0047] For example, the target tensor in the first dimension is a 5-dimensional tensor, denoted as [N, C, D, H, W], with specific shape dimensions of [100, 100, 100, 100, 100]. The chip supports a maximum size of 50 in the specified dimension D. Therefore, the size of the target tensor in the specified dimension D exceeds the maximum size supported by the chip for the activation layout in the specified dimension D. The data of the target tensor in dimension D is adjusted to dimension N, resulting in a 4-dimensional target tensor, denoted as [N...]. [D, C, H, W], with a specific shape size of
[100] . [100, 100, 100, 100] Then, the adjusted 4-dimensional tensor is expanded to obtain a 7-dimensional tensor. The specific expansion steps are the same as those mentioned above and will not be repeated here.
[0048] It should be added that when the target tensor of the first dimension is a 5-dimensional tensor, this application can directly adjust the data of the target tensor in the D dimension to the N dimension without considering whether the size in the D dimension exceeds the size limit. At the same time, it takes into account both cases of tensor exceeding the dimension and cases of tensor not exceeding the dimension. Through a general calculation method, a 4-dimensional tensor or a 5-dimensional tensor can be expanded and transformed to obtain a 7-dimensional tensor without the need for separate development for 4-dimensional tensors or 5-dimensional tensors, thus reducing development and maintenance costs.
[0049] Step 203: Based on the target tensor of the second dimension, rearrange the data in the first buffer to obtain the rearranged target tensor.
[0050] Specifically, data rearrangement is performed in the first buffer based on the target tensor of the second dimension, see [link to relevant documentation]. Figure 4 This includes the following steps: Step 401: In the first buffer, the data of the second-dimensional target tensor in the new dimension is mapped to the N-dimensional dimension to obtain the third-dimensional target tensor. The third-dimensional target tensor has a different size from the first-dimensional target tensor in at least one dimension.
[0051] Step 402: Perform data rearrangement on the target tensor of the third dimension.
[0052] For example, the first-dimensional target tensor is a 4-dimensional tensor, denoted as [N, C, H, W], with a specific shape of [100, 100, 100, 100]. After tensor expansion, the resulting second-dimensional target tensor is denoted as [N, subC, subH, subW, baseC, baseH, baseW], with a specific shape of [100, 2, 2, 2, 50, 50, 50]. Mapping the data in the new dimensions subC, subH, and subW onto the N dimension yields the third-dimensional target tensor, denoted as [N...]. subC subH [subW, baseC, baseH, baseW], with a specific shape size of
[100] . 2 2 [2, 50, 50, 50], that is, the target tensor of the third dimension is [800, 50, 50, 50].
[0053] Since the N-dimensional (batch axis) is executed cyclically when the hardware generates assembly code, the new dimensions subC, subH, and subW can be combined and multiplied into the N-dimensional.
[0054] In some embodiments, before data rearrangement of the target tensor of the third dimension, if the size of the target tensor of the third dimension in the N dimension exceeds the upper limit of the size supported by the chip in the N dimension, the target tensor is split in the N dimension to obtain multiple sub-tensors of the third dimension.
[0055] For example, the target tensor of the third dimension is [800, 50, 50, 50]. The maximum size supported in the N dimension is 100. The size of the target tensor of the third dimension, 800, in the N dimension is greater than 100. If the target tensor is split in the N dimension, the size of the sub-tensor of the third dimension is 100, resulting in 8 sub-tensors of the third dimension, all of which are [100, 50, 50, 50].
[0056] Step 204: Based on the dimensions of the first-dimensional target tensor in various dimensions, the dimensions of the rearranged target tensor in various dimensions, and the starting coordinates of the first-dimensional target tensor in the first buffer, determine the write coordinates of the rearranged target tensor in the second buffer.
[0057] In some embodiments, during tensor coordinate transformation, the dimensions (i.e., shape) of the target tensor in various dimensions and its starting coordinates in the first buffer are first determined. For example, if the first-dimensional target tensor src1 is [8, 4, 6], the rearranged target tensor dst1 is [4, 6, 8], the starting coordinates of the first-dimensional target tensor in the first buffer are [n1, h1, w1], and the write coordinates of the rearranged target tensor are [n2, h2, w2]. The coordinate transformation is performed using the following formula:
[0058] According to the above formula, all data points in the tensor Src1 before coordinate transformation correspond to all data points in the tensor dst1 after coordinate transformation.
[0059] Step 205: Send a tensor retrieval request to the first buffer, and write the target tensor returned by the tensor retrieval request into the second buffer according to the write coordinates; the target tensor in the second buffer is used for calculation by the second operator.
[0060] In some embodiments, regardless of whether the first dimension of the target tensor is four-dimensional or five-dimensional in different business scenarios, the third dimension is always four-dimensional.
[0061] In some embodiments, in image processing applications, the target tensor is image data used in tasks such as image preprocessing, image segmentation, and object detection, and the image data includes specific features of the image; taking a target tensor of the first dimension as four-dimensional image tensor data as an example, the operator execution method of this application specifically includes the following steps: The first step involves the AI chip initiating a thread group within its execution unit to retrieve the four-dimensional image tensor data output by the first operator and store it in the first buffer. This first buffer is located in High Bandwidth Memory (HBM).
[0062] The arrangement type of four-dimensional image tensor data is the same in different business scenarios, namely ActivationLayout. However, since different business scenarios require different tensor dimensions, for example, the hardware memory of Model A requires the target tensor to be four-dimensional with an upper limit of 100, or the hardware memory of Model B requires the target tensor to be four-dimensional with an upper limit of 50, or the hardware memory of Model C requires the target tensor to be three-dimensional with an upper limit of 50, etc. Therefore, the artificial intelligence chip will transform the acquired four-dimensional image tensor data in the first buffer to meet the needs of different business scenarios, so as to facilitate different models to better understand the image features in the tensor. The specific steps are as follows in step two.
[0063] The second step is that when the size of the four-dimensional image tensor data in any specified dimension exceeds the chip's upper limit of the size supported by the arrangement type in the specified dimension, the artificial intelligence chip expands the four-dimensional image tensor data into seven-dimensional image tensor data in the first cache through the thread group in the execution unit according to the rule that the size of the target tensor in multiple specified dimensions is not greater than the upper limit of the size corresponding to each specified dimension; wherein, the seven-dimensional image tensor data is fixed in different business scenarios.
[0064] For example, let's denote the four-dimensional image tensor data as [N, C, H, W], with specific shape dimensions of [100, 100, 100, 100]. The chip supports upper size limits of 50, 50, and 50 for the specified dimensions C, H, and W, respectively. Therefore, the dimensions of the current four-dimensional image tensor data in the specified dimensions C, H, and W all exceed the chip's upper size limits for the Activation layout in the specified dimensions C, H, and W. Taking dimension C as an example, for each specified dimension (C, H, and W), the data in dimension C is split, resulting in 50 data points. 2. After splitting, the size in dimension C is 50, and the number of data parts in dimension C is 2. By analogy, we obtain the seven-dimensional image tensor data, denoted as [N, subC, subH, subW, baseC, baseH, baseW], with specific shape dimensions of [100, 2, 2, 2, 50, 50, 50].
[0065] The third step is to rearrange the data in the first buffer based on the seven-dimensional image tensor data to obtain the rearranged target tensor.
[0066] It should be noted that the AI chip also maps the data in the new dimensions subC, subH, and subW to the N dimension, obtaining four-dimensional image tensor data, denoted as [N]. subC subH [subW, baseC, baseH, baseW], with a specific shape size of
[100] . 2 2 2, 50, 50, 50, meaning the four-dimensional image tensor data is [800, 50, 50, 50].
[0067] The fourth step involves determining the write coordinates of the rearranged four-dimensional image tensor data [800, 50, 50, 50] in the second buffer based on the dimensions of the four-dimensional image tensor data [100, 100, 100, 100] in various dimensions, the starting coordinates of the four-dimensional image tensor data [100, 100, 100, 100] in the first buffer, and the dimensions of the rearranged four-dimensional image tensor data [800, 50, 50, 50] in various dimensions. The specific details are omitted here. The second buffer is located in High Bandwidth Memory (HBM).
[0068] Fifth, the AI chip initiates a thread group in its execution unit, sends a tensor acquisition request to the first buffer, and writes the four-dimensional image tensor data [800, 50, 50, 50] returned by the tensor acquisition request into the second buffer according to the write coordinates. This four-dimensional image tensor data [800, 50, 50, 50] is used for calculation by the second operator. It should be noted that in different application scenarios, such as speech processing and text processing, the specific processing flow of the target tensor is similar to the processing flow of the four-dimensional image tensor data described above, and will not be repeated here.
[0069] In summary, in this embodiment, the target tensor of the first dimension is expanded to a second dimension, and the data is rearranged based on the target tensor of the second dimension. A general tensor calculation method is adopted so that the second dimension does not change with the business scenario. It can transform tensors of different dimensions into the second dimension, covering multiple business scenarios in different memory of different models, taking into account the case of tensor hyperdimensionality, and eliminating the need to develop separate conversion calculations between various types of tensors with different memory requirements, thus reducing development and maintenance costs.
[0070] Based on the same technical concept, this application provides a schematic diagram of the structure of an operator execution device, such as... Figure 5 As shown, the operator execution device 500 includes: Tensor acquisition module 501 is configured to acquire a target tensor of the first dimension output by the first operator to a first buffer. The target tensor has the same arrangement type but different dimensions in different business scenarios. Tensor expansion module 502 is configured to expand the target tensor to a second dimension in the first buffer according to the rule that the size of the target tensor in multiple specified dimensions is not greater than the size limit corresponding to each specified dimension when the size of the target tensor in any specified dimension exceeds the upper limit of the size supported by the chip for the arrangement type in the specified dimension. The second dimension is the same value in different business scenarios. Tensor rearrangement module 503 is configured to rearrange data in the first buffer to obtain the rearranged target tensor. The coordinate transformation module 504 is configured to determine the write coordinates of the rearranged target tensor in the second buffer based on the size of the first-dimensional target tensor in various dimensions, the size of the rearranged target tensor in various dimensions, and the starting coordinates of the first-dimensional target tensor in the first buffer. Tensor storage module 505 is configured to send a tensor acquisition request to the first buffer, and write the target tensor returned by the tensor acquisition request into the second buffer according to the write coordinates; the target tensor in the second buffer is used for calculation by the second operator.
[0071] Optionally, the multiple specified dimensions include C dimension, H dimension and W dimension, wherein C dimension represents the channel dimension of the target tensor, H dimension represents the height dimension of the target tensor and W dimension represents the width dimension of the target tensor; According to the rule that the size of the target tensor in multiple specified dimensions is no greater than the upper limit of the size corresponding to each specified dimension, the target tensor is expanded to a second dimension. The tensor expansion module is specifically configured as follows: For each specified dimension among C, H, and W, perform the following steps: If the size of the target tensor in the specified dimension exceeds the upper limit of the chip's supported size in the specified dimension, then the data of the target tensor in the specified dimension is split according to the size in the specified dimension and the upper limit of the size; the size of the target tensor in the specified dimension is updated to the size of the split target tensor in the specified dimension; and the number of data parts of the split target tensor in the specified dimension is used as the size of the target tensor in a new dimension corresponding to the specified dimension. If the size of the target tensor in the specified dimension does not exceed the upper limit of the size supported by the chip in the specified dimension, then the number of data parts of the target tensor in the specified dimension is taken as the size of the target tensor in a new dimension corresponding to the specified dimension.
[0072] Optionally, the target tensor of the first dimension further includes a D dimension and an N dimension, wherein the D dimension represents the depth dimension of the target tensor, and the N dimension represents the batch dimension of the target tensor. The tensor expansion module is further configured to: When the size of the target tensor in the D dimension exceeds the upper limit of the size supported by the chip in the D dimension, the data of the target tensor in the D dimension is adjusted to the N dimension.
[0073] Optionally, data rearrangement is performed in the first buffer based on the target tensor of the second dimension, and the tensor rearrangement module is specifically configured as follows: In the first buffer, the data of the second-dimensional target tensor in the new dimension is mapped to the N-dimensional dimension to obtain a third-dimensional target tensor, wherein the third-dimensional target tensor differs from the first-dimensional target tensor in size in at least one dimension; and Data rearrangement is performed on the target tensor of the third dimension.
[0074] Optionally, before performing data rearrangement on the target tensor of the third dimension, the tensor rearrangement module is further configured to: When the size of the target tensor in the third dimension exceeds the upper limit of the chip's size in the N dimension, the target tensor is split in the N dimension to obtain multiple sub-tensors in the third dimension.
[0075] Optionally, the first dimension of the target tensor in different business scenarios is four-dimensional or five-dimensional, and it further includes: If the first dimension is four-dimensional, then the third dimension is four-dimensional; If the first dimension is five-dimensional, then the third dimension is four-dimensional.
[0076] In this embodiment, the target tensor of the first dimension is expanded to the second dimension, and the data is rearranged according to the target tensor of the second dimension. A general tensor calculation method is adopted, and the second dimension does not change with the business scenario. It can transform tensors of different dimensions into the second dimension, covering a variety of business scenarios. Moreover, there is no need to develop separate conversion calculations between various types of tensors, which reduces development and maintenance costs.
[0077] Based on the same technical concept, embodiments of this application provide a computer device, such as... Figure 6 As shown, it includes at least one artificial intelligence chip 100 and a memory 601 connected to at least one artificial intelligence chip 100. In this embodiment, the specific connection medium between the artificial intelligence chip 100 and the memory 601 is not limited. Figure 6 Taking the connection between the AI chip 100 and the memory 601 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.
[0078] In this embodiment of the application, the memory 601 stores instructions that can be executed by at least one artificial intelligence chip 100. By executing the instructions stored in the memory 601, at least one artificial intelligence chip 100 can perform the steps of the above-described operator execution method.
[0079] The artificial intelligence chip 100 serves as the control center of the computer device, connecting to various parts of the device via various interfaces and lines. It performs tensor rearrangement by running or executing instructions stored in the memory 601 and accessing data stored in the memory 601. Optionally, the artificial intelligence chip 100 may include one or more processing units. It may integrate an application processor and a modem processor, where the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 601 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.
[0080] The artificial intelligence chip 100 can be a general-purpose processor, such as a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.
[0081] Memory 601, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 601 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory 601 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer device, but is not limited thereto. Memory 601 in the embodiments of this application may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.
[0082] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the above-described operator execution method.
[0083] Based on the same inventive concept, this application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described operator execution method.
[0084] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0085] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer apparatus or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer device or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0087] These computer program instructions may also be loaded onto a computer device or other programmable data processing equipment to cause a series of operational steps to be performed on the computer device or other programmable equipment to produce a process implemented by the computer device, thereby providing instructions that execute on the computer device or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0088] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.
[0089] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. An operator execution method, characterized in that, The method includes: The first dimension of the target tensor output by the first operator is obtained and stored in the first buffer. The target tensor has the same arrangement type but different dimensions in different business scenarios. When the size of the target tensor in any specified dimension exceeds the upper limit of the size supported by the chip for the arrangement type in the specified dimension, the target tensor is expanded to a second dimension in the first buffer according to the rule that the size of the target tensor in multiple specified dimensions is not greater than the upper limit of the size corresponding to each specified dimension. The second dimension is the same value in different business scenarios. Based on the target tensor of the second dimension, the data is rearranged in the first buffer to obtain the rearranged target tensor; Based on the dimensions of the first-dimensional target tensor in various dimensions, the dimensions of the rearranged target tensor in various dimensions, and the starting coordinates of the first-dimensional target tensor in the first buffer, the write coordinates of the rearranged target tensor in the second buffer are determined. A tensor retrieval request is sent to the first buffer, and the target tensor returned by the tensor retrieval request is written to the second buffer according to the write coordinates; the target tensor in the second buffer is used for calculation by the second operator.
2. The method as described in claim 1, characterized in that, The specified dimensions include C dimension, H dimension and W dimension, where C dimension represents the channel dimension of the target tensor, H dimension represents the height dimension of the target tensor and W dimension represents the width dimension of the target tensor. Expanding the target tensor to a second dimension according to the rule that the size of the target tensor in multiple specified dimensions is not greater than the upper limit of the size corresponding to each specified dimension includes: For each specified dimension among C, H, and W, perform the following steps: If the size of the target tensor in the specified dimension exceeds the upper limit of the chip's supported size in the specified dimension, then the data of the target tensor in the specified dimension is split according to the size in the specified dimension and the upper limit of the size; the size of the target tensor in the specified dimension is updated to the size of the split target tensor in the specified dimension; and the number of data parts of the split target tensor in the specified dimension is used as the size of the target tensor in a new dimension corresponding to the specified dimension. If the size of the target tensor in the specified dimension does not exceed the upper limit of the size supported by the chip in the specified dimension, then the number of data parts of the target tensor in the specified dimension is taken as the size of the target tensor in a new dimension corresponding to the specified dimension.
3. The method as described in claim 2, characterized in that, The first dimension of the target tensor also includes a D dimension and an N dimension, wherein the D dimension represents the depth dimension of the target tensor, and the N dimension represents the batch dimension of the target tensor, and further includes: When the size of the target tensor in the D dimension exceeds the upper limit of the size supported by the chip in the D dimension, the data of the target tensor in the D dimension is adjusted to the N dimension.
4. The method as described in claim 2 or 3, characterized in that, Data rearrangement is performed in the first buffer based on the target tensor of the second dimension, including: In the first buffer, the data of the second-dimensional target tensor in the new dimension is mapped to the N-dimensional dimension to obtain a third-dimensional target tensor, wherein the third-dimensional target tensor differs from the first-dimensional target tensor in size in at least one dimension; and Data rearrangement is performed on the target tensor of the third dimension.
5. The method as described in claim 4, characterized in that, Before performing data rearrangement on the target tensor of the third dimension, the method further includes: When the size of the target tensor in the third dimension exceeds the upper limit of the chip's size in the N dimension, the target tensor is split in the N dimension to obtain multiple sub-tensors in the third dimension.
6. The method as described in claim 4, characterized in that, The first dimension of the target tensor in different business scenarios is four-dimensional or five-dimensional, and it also includes: If the first dimension is four-dimensional, then the third dimension is four-dimensional; If the first dimension is five-dimensional, then the third dimension is four-dimensional.
7. An operator execution device, characterized in that, The device includes: The tensor acquisition module is configured to acquire the target tensor of the first dimension output by the first operator to the first buffer. The target tensor has the same arrangement type but different dimensions in different business scenarios. The tensor expansion module is configured to expand the target tensor to a second dimension in the first buffer according to the rule that the size of the target tensor in multiple specified dimensions is not greater than the upper limit of the size corresponding to each specified dimension when the size of the target tensor in any specified dimension exceeds the upper limit of the size supported by the chip for the arrangement type in the specified dimension. The second dimension is the same value in different business scenarios. The tensor rearrangement module is configured to rearrange data in the first buffer to obtain the rearranged target tensor. The coordinate transformation module is configured to determine the write coordinates of the rearranged target tensor in the second cache based on the size of the first-dimensional target tensor in various dimensions, the size of the rearranged target tensor in various dimensions, and the starting coordinates of the first-dimensional target tensor in the first cache. The tensor storage module is configured to send a tensor retrieval request to the first buffer, and write the target tensor returned by the tensor retrieval request into the second buffer according to the write coordinates; the target tensor in the second buffer is used for calculation by the second operator.
8. A computer device, comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, characterized in that, When the artificial intelligence chip executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, It stores a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that, The computer program product includes a computer program stored on a computer-readable storage medium, the computer program including program instructions that, when executed by a computer device, cause the computer device to perform the steps of the method as described in any one of claims 1-6.