A method, device, medium and product for cleaning a cross-boundary region

Abstract

The application provides an out-of-bound region cleaning method, device, medium and product, which are applied to the field of artificial intelligence chip technology. In the method, a target storage region is allocated for a target tensor, the logical valid byte number of the target tensor is obtained based on the element number contained in the target tensor and the byte number corresponding to each element; when the physical storage byte number of the target storage region is greater than the logical valid byte number, a target write path of the target tensor is identified; when the target write path is writing through a target operator and the target operator belongs to an operator type of a newly created output tensor, the target out-of-bound region in the target storage region is cleaned before an operator reading performed after the target storage region is written by the target operator, and the target tensor is written into a target valid region in the target storage region through the target operator, so that out-of-bound judgment or cleaning is avoided when a subsequently executed operator reads the target tensor, the overhead of additional operations is reduced, and the overall performance of the chip is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence chip technology, and in particular to a method, device, medium, and product for clearing out-of-bounds areas. Background Technology

[0002] In deep learning frameworks, tensors have a logical shape that is visible to the user. When artificial intelligence chips (such as graphics processing units, or GPUs) perform calculations on tensors, the hardware storage area allocated to the tensor often needs to be aligned with the hardware granularity. Therefore, the hardware storage area allocated to the tensor may be larger than the logical shape of the tensor. In this case, the hardware storage area not only includes the logically valid area that stores the tensor, but also the out-of-bounds (OOB) area outside the logically valid area.

[0003] When non-zero data remains in the bounded region, the hardware unit that performs subsequent calculations on the tensor (such as the tensor calculation unit (TCore) in an artificial intelligence chip) may substitute the non-zero data in the bounded region into the intermediate calculation, resulting in incorrect calculation results. Therefore, it is necessary to clean up the bounded region.

[0004] In related technologies, the tensor computation unit performs out-of-bounds judgment or cleanup of out-of-bounds areas in the hardware storage area before each tensor computation. This results in a large overhead for the tensor computation unit, which in turn affects the overall performance of the chip. Summary of the Invention

[0005] This invention provides a method, device, medium, and product for clearing out-of-bounds regions, which avoids the tensor computation unit from performing out-of-bounds judgment or clearing of out-of-bounds regions of the hardware storage area every time it reads tensor computation, thereby reducing the overhead of the tensor computation unit and improving the overall performance of the chip.

[0006] On the one hand, embodiments of this application provide a method for clearing out-of-bounds areas, the method comprising: Allocate a target storage region for the target tensor; Based on the number of elements contained in the target tensor and the number of bytes corresponding to each element, identify the number of logically valid bytes of the target tensor; When the number of physical storage bytes in the target storage region is greater than the number of logical valid bytes, obtain the target write path of the target tensor; When the target write path is written through a target operator, and the target operator is an operator type for creating a new output tensor, before the target storage area is read by the operator executed after the target operator, the target out-of-bounds region in the target storage area is cleaned up, and the target tensor is written into the target valid region in the target storage area through the target operator.

[0007] On one hand, embodiments of this application provide a boundary crossing clearing device, the device comprising: The allocation module is used to allocate target storage areas for target tensors; The acquisition module is used to obtain the logically valid number of bytes of the target tensor based on the number of elements contained in the target tensor and the number of bytes corresponding to each element; The judgment module is used to identify the target write path of the target tensor when the number of physical storage bytes in the target storage area is greater than the number of logical valid bytes; The cleanup module is used to clean up the out-of-bounds region of the target storage area before the target storage area is read by the operator executed after the target operator when the target write path is written through the target operator and the target operator belongs to the operator type of creating a new output tensor, and then write the target tensor into the target valid region of the target storage area through the target operator.

[0008] Optionally, the cleaning module is specifically used for: The target out-of-bounds region can be cleaned up by calling a public out-of-bounds cleanup function; or, a preset cleanup value can be written to the target out-of-bounds region through the target operator.

[0009] Optionally, the cleaning module is further configured to: When the target write path is written through a target operator, and the target operator belongs to the operator type that modifies an existing tensor, the target tensor is written into the target valid region through the target operator.

[0010] Optionally, the cleaning module is further configured to: The target out-of-bounds area is cleared and detected by a detection operator, and the detection results are obtained. When the detection result indicates that the target out-of-bounds area contains a non-preset cleanup value, the target out-of-bounds area is cleaned up by a cleanup operator.

[0011] Optionally, the target operator, the detection operator, and the cleanup operator are executed sequentially in the same task flow.

[0012] Optionally, the cleaning module is further configured to: In test mode, if the output result of the target valid area is consistent with the preset result, and the target out-of-bounds area does not contain non-preset cleanup values, then the target operator test is determined to be passed. If the output result of the target valid region is inconsistent with the preset result, or if the target out-of-bounds region contains a non-preset cleanup value, then the target operator test is determined to be unsuccessful.

[0013] Optionally, the cleaning module is further configured to: When the target write path is written via a batch setting interface, and the target tensor's fill value is a preset cleanup value, the target out-of-bounds region and the target valid region are set to the preset cleanup value via the batch setting interface.

[0014] Optionally, the cleaning module is further configured to: When the target write path is written via the copy interface, the target tensor is copied from the source valid region in the source storage region to the target valid region via the copy interface; The copy interface is used to copy the padding value of the source out-of-bounds region in the source storage area to the target out-of-bounds region.

[0015] Optionally, the cleaning module is further configured to: In test mode, if the output result of the target valid area is consistent with the preset result, and the target out-of-bounds area does not contain non-preset cleanup values, then the copy interface test is determined to be passed. If the output result of the target valid area is inconsistent with the preset result, or if the target out-of-bounds area contains a non-preset cleanup value, then the copy interface test is determined to be unsuccessful.

[0016] On one hand, embodiments of this application provide a computer device, including: Memory, used to store program instructions; An artificial intelligence chip is used to run program instructions stored in the memory and execute the steps of the above-mentioned out-of-bounds area cleanup method according to the obtained program.

[0017] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer to perform the steps of the above-described out-of-bounds area cleanup method.

[0018] On one hand, embodiments of this application provide a computer program product, including a computer program stored on a computer-readable storage medium, the computer program including program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described out-of-bounds area cleanup method.

[0019] In this embodiment, a target storage region is allocated for the target tensor. Based on the number of elements contained in the target tensor and the number of bytes corresponding to each element, the logically valid number of bytes of the target tensor is obtained. When the physical storage bytes of the target storage region are greater than the logically valid number of bytes, it is determined that there is a target out-of-bounds region in the target storage region, and the target write path of the target tensor is identified. When the target write path is written through a target operator, and the target operator belongs to the operator type for creating a new output tensor, the target out-of-bounds region in the target storage region is cleaned up before the target storage region is read by the operator executed after the target operator. Then, the target tensor is written into the target valid region of the target storage region through the target operator. That is, during the process of writing the target tensor into the target valid region through the target operator, the target out-of-bounds region is cleaned up at the same time. This ensures the cleanliness of the target out-of-bounds region. Therefore, when the operator executed after the target operator reads the target tensor in the target storage region, the tensor calculation unit no longer needs to perform out-of-bounds judgment or cleanup of the out-of-bounds region, thereby effectively reducing the performance overhead caused by additional operations, while ensuring the accuracy of tensor calculation, and thus improving the overall performance of the chip. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of a chip architecture provided in an embodiment of this application; Figure 2 This is a schematic diagram of a method for clearing out-of-bounds areas provided in an embodiment of this application; Figure 3 A flowchart illustrating a method for clearing out-of-bounds areas provided in an embodiment of this application; Figure 4 A flowchart illustrating a testing method provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a boundary crossing clearing device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0023] refer to Figure 1 This is a structural diagram of an artificial intelligence chip applicable to the embodiments of this application. The artificial intelligence chip 100 includes at least: a video memory 101 and a plurality of execution units 102, wherein each execution unit 102 includes: an on-chip cache 103 and a tensor calculation unit 104.

[0024] Video memory 101 can be high-bandwidth memory (HBM) or other types of memory. On-chip cache 103 is a temporary memory with a smaller capacity than video memory 101, but a faster data exchange speed. On-chip cache 103 can be a general-purpose matrix main buffer (GMB) or other types of memory.

[0025] Tensor computation unit 104, as an accelerator in artificial intelligence chip 100, typically performs convolution operations or matrix multiplication operations, that is, tensor computation unit 104 is typically used to perform matrix multiplication operators, convolution operators, etc.

[0026] In this embodiment of the application, a target storage area can be allocated for the target tensor in the video memory 101, or the target storage area can be allocated for the target tensor in the on-chip cache 103.

[0027] Based on the number of elements contained in the target tensor and the number of bytes corresponding to each element, the logically valid number of bytes of the target tensor is obtained. When the number of physical storage bytes in the target storage area is greater than the number of logically valid bytes, it is determined that there is a target out-of-bounds region in the target storage area, and the target write path of the target tensor is identified.

[0028] When the target write path is through a target operator, and the target operator is of the type that creates a new output tensor, the out-of-bounds region in the target storage area is cleaned up before the target storage area is read by the operator executed after the target operator. Then, the target tensor is written into the target valid region of the target storage area through the target operator. That is, during the process of writing the target tensor into the target valid region through the target operator, the out-of-bounds region is simultaneously cleaned up, thus ensuring the cleanliness of the out-of-bounds region. In this way, when the tensor calculation unit 104 reads the target tensor from the target storage area for subsequent calculations, it no longer needs to perform out-of-bounds judgment or cleanup of the out-of-bounds region, effectively reducing the performance overhead caused by additional operations, while ensuring the accuracy of tensor calculation, thereby improving the overall performance of the chip.

[0029] In addition to the structure described above, the artificial intelligence chip 100 in this application may also include other structures, and this application does not specifically limit such structures.

[0030] Artificial intelligence chips 100 can be: Graphics Processing Unit (GPU), General-purpose computing on graphics processing units (GPGPU), Domain Specific Architecture (DSA), etc.

[0031] It should be noted that in this application, "first" and "second" are used for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features indicated.

[0032] The following is based on Figure 1 The diagram shown illustrates the architecture of an AI chip and details the process of a method for clearing out-of-bounds areas. (See also...) Figure 2 The method is performed by a computer device, wherein the computer device includes Figure 1 The artificial intelligence chip shown includes the following steps: Step 201: Allocate a target storage area for the target tensor.

[0033] The boundary violation clearing method of this application can be applied to various scenarios, such as image processing, speech processing, and text processing. The physical meaning of the target tensor may differ in different application scenarios.

[0034] For example, in a text processing scenario, the target tensor can be text data used in tasks such as text generation and text recognition.

[0035] For example, in a speech processing scenario, the target tensor can be the speech data used in tasks such as speech enhancement, speech recognition, and speech synthesis.

[0036] For example, in image processing scenarios, the target tensor can be image data used in tasks such as image preprocessing, image segmentation, and object detection.

[0037] In this embodiment, the target tensor can be the input tensor of the inference or training process, or it can be an intermediate result generated by the inference or training process. A target storage area is pre-allocated for storing the target tensor, so that the tensor computation unit can read the target tensor from the target storage area for computation.

[0038] In the embodiments of this application, the target storage area can be allocated in the video memory of the artificial intelligence chip, or in the on-chip cache of the artificial intelligence chip; of course, the target storage area can also be allocated in other memories of the artificial intelligence chip, and this application does not specifically limit this.

[0039] Step 202: Based on the number of elements contained in the target tensor and the number of bytes corresponding to each element, obtain the number of logically valid bytes of the target tensor.

[0040] Step 203: When the number of physical storage bytes in the target storage area is greater than the number of logical valid bytes, identify the target write path of the target tensor.

[0041] Specifically, when allocating a target storage region for a target tensor, the allocator preserves the physical shape of the target storage region. Even if the target tensor is reshaped or sliced, the physical shape of the target storage region remains unchanged. The allocator obtains the number of elements in the target tensor, the number of bytes corresponding to each element, and the number of physical storage bytes corresponding to the physical shape of the target storage region. Multiplying the number of elements in the target tensor by the number of bytes corresponding to each element determines the logically valid number of bytes in the target tensor.

[0042] It should be noted that the number of regions in the target storage region varies depending on the application scenario. For example, the number of regions in the target storage region differs for multi-die scenarios, Non-Uniform Memory Access (NUMA), Uniform Memory Access (UMA), and Split / Broadcast / Partial (SBP) scenarios. The specific number of regions is actually adjusted according to the hardware distribution.

[0043] When the number of physical storage bytes in the target storage area is greater than the number of logical valid bytes, it indicates that there is an out-of-bounds region in the target storage area. In this case, it is necessary to ensure the cleanliness of the out-of-bounds region. When the number of physical storage bytes in the target storage area is equal to the number of logical valid bytes, it indicates that there is no out-of-bounds region in the target storage area. In this case, the target tensor does not need to be checked or cleaned up during this processing.

[0044] After determining that an out-of-bounds region exists in the target storage area, the target write path for the target tensor is determined. The target write path includes: writing via the target operator, writing via the batch set interface, and writing via the copy interface.

[0045] Step 204: When the target write path is written through the target operator and the target operator belongs to the operator type of creating a new output tensor, before the target storage area is read by the operator executed after the target operator, the target out-of-bounds area in the target storage area is cleaned up, and the target tensor is written into the target valid area in the target storage area through the target operator.

[0046] Specifically, when allocating the target storage area, a mapping function from logical coordinates to physical coordinates is established based on the logical shape of the target tensor and the physical shape of the target storage area. Any physical coordinate within the target storage area is traversed, and a reverse mapping is used to determine whether the physical coordinate corresponds to a valid logical coordinate (i.e., whether it falls within the logical shape). If a corresponding valid logical coordinate exists, the physical location belongs to the target's valid area; if no corresponding valid logical coordinate exists, the location is determined to be outside the target's bounds area.

[0047] For example, in a two-dimensional row-major scenario, the logical shape of the target tensor is set to [3,5], and the physical shape of the target storage area is set to [3,8] (aligned by row).

[0048] Row-major order mapping: logical coordinates (r, l), requiring , This is mapped to physical coordinates (r, p=l).

[0049] Traverse the physical coordinates (r, p), when When the time is right, the logical column coordinates p=l can be obtained by reverse mapping, which means it is valid and therefore belongs to the target valid region.

[0050] when When the physical column coordinates exceed the upper limit of the logical column, they cannot be mapped to valid logical coordinates, and therefore belong to the target out-of-bounds area.

[0051] For example, in a two-dimensional column-major scenario, the logical shape of the target tensor is set to [3,5], and the physical shape of the target storage area is set to [6,5] (aligned by columns, with 6 rows allocated per column).

[0052] Column-major mapping: Logical coordinates (l,c) are mapped to physical coordinates (p=l,c).

[0053] Traverse the physical coordinates (p,c), when When the time is right, the logical row coordinates p=l can be obtained by reverse mapping, which means it is valid and therefore belongs to the target valid region.

[0054] when When the physical row coordinates exceed the upper limit of the logical row, they cannot be mapped to valid logical coordinates, and therefore belong to the target out-of-bounds area.

[0055] In multi-region storage scenarios, the same method described above is used to divide the out-of-bounds regions for each region, which will not be repeated here. In addition, the reshape operation and the slice operation only modify the metadata of the target tensor and do not change the out-of-bounds region division of the underlying storage.

[0056] The target write path refers to the method of writing the target tensor to the target storage area; the target operator represents the operator that writes the target tensor to the target storage area. Writing via the target operator means: running the target operator to write the target tensor to the target valid area in the target storage area.

[0057] The operator type for creating a new output tensor refers to an operator type that generates a new output tensor and stores it in a newly allocated storage area. When the target operator is the operator type for creating a new output tensor, it means that the target tensor is a new output tensor and the target storage area is a newly allocated storage area. At this time, the target storage area is uninitialized dirty memory, so it is necessary to clean up the target out-of-bounds region in the target storage area.

[0058] In the embodiments of this application, the operators for creating output tensors include, but are not limited to: element-wise operators, matrix multiplication operators, cumsum operators, reduce operators, topk operators (i.e., operators that take the first K values), layer normalization operators, reorder operators, and layout operators.

[0059] It should be noted that if the cleanup operation covers the target valid area, the target out-of-bounds area in the target storage area is cleaned first, and then the target tensor is written into the target valid area in the target storage area. If the cleanup operation does not cover the target valid area, the execution order of the cleanup operation on the target out-of-bounds area and the writing of the target tensor into the target valid area can be arbitrarily adjusted. In particular, both the cleanup operation and the write operation must be completed before the subsequent operator reads the target tensor from the target storage area.

[0060] In this embodiment, a target storage region is allocated for the target tensor. Based on the number of elements contained in the target tensor and the number of bytes corresponding to each element, the logically valid number of bytes of the target tensor is obtained. When the physical storage bytes of the target storage region are greater than the logically valid number of bytes, it is determined that there is a target out-of-bounds region in the target storage region, and the target write path of the target tensor is identified. When the target write path is written through a target operator, and the target operator belongs to the operator type for creating a new output tensor, the target out-of-bounds region in the target storage region is cleaned up before the operator executed after the target operator reads the target storage region. Then, the target tensor is written into the target valid region of the target storage region through the target operator. That is, during the process of writing the target tensor into the target valid region through the target operator, the target out-of-bounds region is cleaned up at the same time. This ensures the cleanliness of the target out-of-bounds region. Therefore, when the operator executed after the target operator reads the target tensor in the target storage region, the tensor calculation unit no longer needs to perform out-of-bounds judgment or cleanup of the out-of-bounds region, thereby effectively reducing the performance overhead caused by additional operations, while ensuring the accuracy of tensor calculation, and thus improving the overall performance of the chip.

[0061] In some embodiments, a public out-of-bounds cleanup function is invoked to clean up the target out-of-bounds region; or, a preset cleanup value is written to the target out-of-bounds region through a target operator.

[0062] Specifically, when the target boundary-crossing area is completely filled with the preset cleanup value, it indicates that the target boundary-crossing area is clean; when the target boundary-crossing area contains non-preset cleanup values, it indicates that the target boundary-crossing area is dirty. In practical applications, the preset cleanup value can be set to zero, and the non-preset cleanup value can be set to non-zero; the preset cleanup value and the non-preset cleanup value can also be set to other values. This application does not specifically limit this.

[0063] The out-of-bounds region can be cleared by writing a preset cleanup value to it using the target operator. Alternatively, during the execution of the target operator, the common out-of-bounds cleanup function (clear_Partial_OOB) can be called to clean up the out-of-bounds region. This function calculates the position of the last unaligned block according to different tensor layout types, such as vector, row-major matrix, column-major matrix, and activation vector, and writes the preset cleanup value to the out-of-bounds region.

[0064] In some cases, certain hardware uses a burst store approach when calling the public out-of-bounds cleanup function to clean up out-of-bounds regions. This burst store method writes the entire aligned memory block at once. If the target out-of-bounds region and the target valid region fall within the same aligned memory block, valid data in the target valid region will be overwritten. Therefore, this application sets the calling order of the public out-of-bounds cleanup function as follows: The calling order is as follows: First, if the public out-of-bounds cleanup function will overwrite the target valid area, the public out-of-bounds cleanup function will be called first to clean up the target storage area, and then the target tensor will be written into the target valid area of ​​the target storage area.

[0065] The second step in the call sequence is to first write the target tensor into the target valid region of the target storage area, configure the coordinate range of the target valid region inside the public out-of-bounds cleanup function, and then clean up the target out-of-bounds region based on the coordinate range.

[0066] In this embodiment, a common out-of-bounds cleanup function is called to clean up the out-of-bounds region, avoiding each target operator from directly cleaning up the out-of-bounds region by traversing the target storage area, thus reducing the waste of target operator resources; at the same time, the calling order of the common out-of-bounds cleanup function is set to ensure the reuse of the common out-of-bounds cleanup function and to prevent the common out-of-bounds cleanup function from destroying the target tensor in the target valid region.

[0067] In some embodiments, when the target write path is written through a target operator and the target operator belongs to the operator type that modifies an existing tensor, the target tensor is written to the target valid region through the target operator.

[0068] Specifically, when the target operator is a type of operator that modifies an existing tensor, it means that the target tensor is not generated for the first time by the target operator, but is a tensor that is partially updated from an existing tensor. The corresponding target storage area is the allocated storage area. Operators that modify existing tensors include, but are not limited to: index_put_, scatter_, add_, masked_fill_, and local cache write path and in-place update operators (op_ for short).

[0069] When the target operator is an operator that modifies an existing tensor, the target storage area is an allocated storage area. This allocated storage area has been cleaned up during the initial allocation. Therefore, the target operator only writes the target tensor into the target valid area and does not touch the target out-of-bounds area to keep the target out-of-bounds area clean.

[0070] In this embodiment, the target operator performs a cleanup operation on the out-of-bounds region of the target tensor while writing the target tensor to the target storage area. Compared with the traditional approach where each operator writes the target tensor and then starts the cleanup operator to perform the cleanup operation, this application can reduce the overhead of starting the cleanup operator and reduce the subsequent synchronization and scheduling overhead.

[0071] In some embodiments, when the target write path is written via a batch setting interface and the target tensor's fill value is a preset cleanup value, the target out-of-bounds region and the target valid region are set to the preset cleanup value via the batch setting interface.

[0072] Specifically, writing via the batch setting interface means setting each byte in the target storage area to the same value directly through the batch setting interface, thereby forming a target tensor in the target valid area of ​​the target storage area.

[0073] When the target tensor's fill value is the preset cleanup value, it means that the target out-of-bounds region and the target valid region need to be filled with the same value. Therefore, the target out-of-bounds region and the target valid region are set to the preset cleanup value through the batch setting interface, so as to realize the target tensor writing and out-of-bounds region cleanup in one go.

[0074] When the number of logically valid bytes of the target tensor is exactly equal to the number of physical storage bytes of the target storage area, it indicates that there is no out-of-bounds region in the target storage area. In this case, if the padding value of the target tensor is the same and is not a preset cleanup value, the target storage area is set to the padding value of the target tensor through the batch setting interface, thus writing the target tensor into the target storage area. Of course, the target storage area can also be set to the padding value of the target tensor through the padding operator or equivalent operator.

[0075] When there is a target out-of-bounds region in the target storage area, and the padding value of the target tensor is not the preset cleanup value, the target out-of-bounds region is set to zero by the padding operator or equivalent operator (i.e., the operator type for creating a new output tensor described above), and the padding value of the target tensor is written into the target valid area.

[0076] When the data type or layout of the target tensor is complex, and the batch setter interface is a low-level function that operates on contiguous memory in bytes, the batch setter interface cannot completely and correctly write the target tensor into the target valid region according to the data type or layout of the target tensor. In this case, the target tensor is written into the target valid region by using a fill operator or equivalent operator (i.e., the operator type for creating a new output tensor described above), and the target out-of-bounds region is set to a preset cleanup value.

[0077] In this embodiment, the batch setting interface itself does not have the function of clearing out-of-bounds areas. However, for special cases such as the target tensor's fill value being a preset cleanup value, a preset cleanup value can be written to the target out-of-bounds area at the same time as the target valid area is written, thereby clearing out-of-bounds areas.

[0078] In some embodiments, when the target write path is written via the copy interface, the target tensor is copied from the source valid region in the source storage region to the target valid region via the copy interface; the padding value of the source out-of-bounds region in the source storage region is copied to the target out-of-bounds region via the copy interface.

[0079] Specifically, writing via the copy interface refers to memory copying, which means that the copy interface copies the complete data from the source storage area to the target storage area.

[0080] The source out-of-bounds region in the source storage area has already undergone a cleanup operation, meaning that the source out-of-bounds region has been filled with a preset cleanup value. Therefore, when copying the target tensor of the source valid region to the target valid region and copying the fill value of the source out-of-bounds region to the target out-of-bounds region, the cleanliness of the target out-of-bounds region can also be guaranteed.

[0081] In some embodiments, a detection operator is used to perform cleanup detection on the target out-of-bounds area to obtain a detection result; when the detection result indicates that the target out-of-bounds area contains a non-preset cleanup value, the target out-of-bounds area is cleaned up using a cleanup operator.

[0082] Specifically, after the target tensor is written to the target valid region according to the target write path, a detection operator is used to clean up the target out-of-bounds region and obtain the detection result. If the detection result indicates that the target out-of-bounds region contains a non-preset cleanup value, it means that the target operator failed to clean up the target out-of-bounds region during the writing process, or that the target out-of-bounds region was polluted when writing the target tensor. Therefore, a cleanup operator is used to perform a fallback cleanup operation on the target out-of-bounds region.

[0083] In application mode, pre-checks can be set for hardware-sensitive operators such as tensor computation unit (Tcore), matrix multiplication operator (Matmul), and multilayer perceptron operator (FusedMlp) to prevent upstream target operators from being missed and thus affecting the correctness of the read and write results of these hardware-sensitive operators.

[0084] In some embodiments, the target operator, detection operator, and cleanup operator are executed sequentially in the same task flow.

[0085] Specifically, in asynchronous execution, concurrent task flow, or graph execution scenarios, the same task flow is used to explicitly establish the execution dependencies between the target operator, the detection operator, and the cleanup operator to ensure the data dependency safety of the target tensor.

[0086] If the target operator, detection operator, and cleanup operator are submitted to different task flows, then graph dependency ensures that detection occurs after the target tensor is written into the target valid region, and the cleanup operator performs cleanup operations on the out-of-bounds region of the target before the target tensor has been written into the target valid region and read by subsequent operations.

[0087] In some embodiments, in test mode, if the output result of the target valid region is consistent with the preset result and the target out-of-bounds region does not contain non-preset cleanup values, then the target operator test is determined to pass; if the output result of the target valid region is inconsistent with the preset result, or the target out-of-bounds region contains non-preset cleanup values, then the target operator test is determined to fail.

[0088] Specifically, in test mode, the preset result of the target operator is set in advance; after the target operator runs, if the output result of the target valid area is consistent with the preset result, and the target out-of-bounds area does not contain non-preset cleanup values, then the write operation and cleanup operation of the target operator can be executed normally, so the target operator test is confirmed to be passed.

[0089] If the output of the target valid region is inconsistent with the preset result, it indicates that an error occurred when the target operator wrote the target valid region. If the target out-of-bounds region contains non-preset cleanup values, it indicates that the cleanup of the target out-of-bounds region failed during the execution of the target operator, or that the originally clean target out-of-bounds region was contaminated. If either "the output of the target valid region is inconsistent with the preset result" or "the target out-of-bounds region contains non-preset cleanup values" exists, the target operator test is determined to be unsuccessful.

[0090] The test mode can be enabled by switching it on or off in continuous integration and high-quality access control scenarios, avoiding additional testing and checks in non-boundary areas and normal performance paths.

[0091] In this embodiment of the application, in the test mode, even if the target tensor of the target valid region is written correctly, but the target out-of-bounds region is not cleaned up, it is considered that the target operator has failed to execute. This realizes that the responsibility for cleaning up the out-of-bounds region is located to the target operator that performed the write operation, thereby improving the accuracy of the target operator test.

[0092] In some embodiments, in test mode, if the output result of the target valid area is consistent with the preset result and the target out-of-bounds area does not contain non-preset cleanup values, then the copy interface test is determined to pass; if the output result of the target valid area is inconsistent with the preset result, or the target out-of-bounds area contains non-preset cleanup values, then the copy interface test is determined to fail.

[0093] Specifically, after the copy interface runs, if the output result of the target valid area is consistent with the preset result, and the target out-of-bounds area does not contain non-preset cleanup values, then it is determined that the copy interface can perform the copy operation normally and does not pollute the target out-of-bounds area during the copy process, so the copy interface test is confirmed to be passed.

[0094] If the output of the target valid region is inconsistent with the preset result, it indicates that an error occurred during the copy interface's copying process from the source valid region. If the target out-of-bounds region contains non-preset cleanup values, it indicates that the copy interface failed to copy from the source out-of-bounds region, or that the preset cleanup values ​​in the originally clean source out-of-bounds region were contaminated. If either "the output of the target valid region is inconsistent with the preset result" or "the target out-of-bounds region contains non-preset cleanup values" exists, the copy interface test is deemed to have failed.

[0095] It should be noted that the same method as the test copy interface described above can be used to test the batch value setting interface, which will not be repeated here.

[0096] To better explain the embodiments of this application, the following describes the process of a boundary crossing clearing method provided by the embodiments of this application in conjunction with specific implementation scenarios, such as... Figure 3 As shown, the specific steps include: Step 301: Allocate the target storage area.

[0097] Specifically, the allocator requests an uninitialized target storage area in the video memory. The uninitialized target storage area retains residual dirty data and does not generate any valid data.

[0098] Step 302: Determine if the target crosses the boundary.

[0099] Specifically, determine whether there is a target out-of-bounds region in the target storage area. If there is a target out-of-bounds region, clean it up according to the responsibility branch for cleaning up the target out-of-bounds region under the different target write paths described in step 304.

[0100] Step 303: Write the target path and responsibility branch.

[0101] Specifically, the target write path includes: target operator write, batch value setting interface write, and copy interface write.

[0102] For writing to target operators, the target operators include: operator types for creating new output tensors and operator types for modifying existing tensors; the process of writing to the valid region and cleaning up out-of-bounds regions for these two operator types has been introduced earlier and will not be repeated here.

[0103] For batch setting interface writes, in the scenario of batch setting / no out-of-bounds region (i.e., the target tensor's fill value is the preset cleanup value / the target storage area does not have an out-of-bounds region), the target tensor is written to the target storage area through the batch setting interface. However, in the scenario of out-of-bounds region and batch setting (i.e., the target storage area contains a target out-of-bounds region, and the target tensor's fill value is not the preset cleanup value), the process switches to the operator type of the newly created output tensor under the target operator write path (e.g., fill operator or equivalent operator) to write the target tensor to the target storage area, while simultaneously cleaning up the target out-of-bounds region.

[0104] For write operations via the copy interface, in scenarios where data is copied from one GPU to another, the out-of-bounds region in the source storage area is kept clean; the data in the source storage area is copied to the target storage area via the copy interface.

[0105] In scenarios where the Central Processing Unit (CPU) copies data to the GPU, the logical data in the CPU is written to the GPU's memory area without out-of-bounds regions through the batch setting interface. Then, the logical data in the memory area without out-of-bounds regions is converted into a target tensor with the GPU-side arrangement through the rearrangement operator and written to the target valid region in the newly allocated target memory area. At the same time, the target out-of-bounds regions in the target memory area are cleaned up.

[0106] Step 304, Branch Merge: Outbound Area Data Check.

[0107] Specifically, when the test gate is open, i.e. in test mode, the target out-of-bounds area is cleaned up by the detection operator.

[0108] Step 305: Determine whether all out-of-bounds areas are preset cleanup values. If so, proceed to step 306; otherwise, proceed to step 307.

[0109] Step 306, check passed: proceed to subsequent operators.

[0110] Step 307, Failure Check: Record the responsibility information and perform a cleanup operation on the data in the out-of-bounds area.

[0111] It should be noted that in application mode (such as model inference or model training process), steps 304-307 do not need to be executed to ensure execution efficiency in application mode.

[0112] In this embodiment, the operator performs a cleanup operation on the out-of-bounds region during operation. The tensor computation unit does not need to perform out-of-bounds region judgment or cleanup operation every time it calculates the target tensor, which reduces the overhead of the tensor computation unit and thus improves the overall performance of the chip. Secondly, recording responsibility information can locate the problem of the target out-of-bounds region not being cleaned from the model level to the specific target operator level, and quickly solve the problem of the target out-of-bounds region not being cleaned through the fallback cleanup operation of the cleanup operator.

[0113] Please see Figure 4 This application provides a flowchart of a testing method, which includes the following steps: Step 401: Start the OOB test mode.

[0114] Step 402: Construct the OOB covering shape.

[0115] Specifically, in test mode, the shape of the target out-of-bounds region should cover various out-of-bounds region generation scenarios, including but not limited to last dimension misalignment, middle dimension misalignment, batch misalignment, UMA / NUMA, multiple regions, multiple bare dies, split / broadcast / partial (SBP) reduction, different data types, different tensor layouts, and OOB scenarios formed by write paths such as new output, in-place update, partial update, batch setting, copy and rearrangement.

[0116] Step 403: Allocate the target storage area.

[0117] The target storage area includes the target valid area and the target out-of-bounds area.

[0118] Step 404, non-zero data injection.

[0119] Specifically, non-zero values ​​are filled into the out-of-bounds regions of the target storage area.

[0120] Step 405: Select the target write path to be tested.

[0121] Specifically, the target write path includes: target operator write, batch value setting interface write, and copy interface write.

[0122] Step 406: Write data to the target storage area through the selected target write path.

[0123] Step 407: Determine whether the target valid area has passed the verification. If it has, proceed to step 408; otherwise, proceed to step 410.

[0124] Specifically, the check is whether the output of the target valid area is consistent with the preset result. If they are consistent, the check passes; otherwise, the check fails.

[0125] Step 408: Determine whether the target has passed the boundary check. If it has, proceed to step 409; otherwise, proceed to step 410.

[0126] If the target out-of-bounds area contains a non-preset cleanup value, the check fails; otherwise, the check passes.

[0127] Step 409, test passed.

[0128] Step 410, test failed.

[0129] In this embodiment, the ability of each operator or interface to maintain the cleanliness of the target out-of-bounds region under different target write paths is checked through the test mode, ensuring the cleanliness of the target out-of-bounds region under the application mode; at the same time, by enabling non-zero data injection and target out-of-bounds region checking by switching, multiple checks can be performed in the test mode to further ensure the normal operation of each operator or interface under different target write paths.

[0130] Based on the same technical concept, this application provides a schematic diagram of the structure of a boundary crossing clearing device, as shown below. Figure 5 As shown, the boundary crossing clearing device 500 includes: The allocation module 501 is used to allocate a target storage area for the target tensor. The acquisition module 502 is used to obtain the logically valid number of bytes of the target tensor based on the number of elements contained in the target tensor and the number of bytes corresponding to each element; The judgment module 503 is used to identify the target write path of the target tensor when the number of physical storage bytes of the target storage area is greater than the number of logical valid bytes; The cleanup module 504 is used to clean up the target out-of-bounds region in the target storage region before the target storage region is read by the operator executed after the target operator when the target write path is written through the target operator and the target operator belongs to the operator type of creating a new output tensor, and write the target tensor into the target valid region in the target storage region through the target operator.

[0131] Optionally, the cleaning module 504 is specifically used for: The target out-of-bounds region can be cleaned up by calling a public out-of-bounds cleanup function; or, a preset cleanup value can be written to the target out-of-bounds region through the target operator.

[0132] Optionally, the cleaning module 504 is further configured to: When the target write path is written through a target operator, and the target operator belongs to the operator type that modifies an existing tensor, the target tensor is written into the target valid region through the target operator.

[0133] Optionally, the cleaning module 504 is further configured to: The target out-of-bounds area is cleared and detected by a detection operator, and the detection results are obtained. When the detection result indicates that the target out-of-bounds area contains a non-preset cleanup value, the target out-of-bounds area is cleaned up by a cleanup operator.

[0134] Optionally, the target operator, the detection operator, and the cleanup operator are executed sequentially in the same task flow.

[0135] Optionally, the cleaning module 504 is further configured to: In test mode, if the output result of the target valid area is consistent with the preset result, and the target out-of-bounds area does not contain non-preset cleanup values, then the target operator test is determined to be passed. If the output result of the target valid region is inconsistent with the preset result, or if the target out-of-bounds region contains a non-preset cleanup value, then the target operator test is determined to be unsuccessful.

[0136] Optionally, the cleaning module 504 is further configured to: When the target write path is written via a batch setting interface, and the target tensor's fill value is a preset cleanup value, the target out-of-bounds region and the target valid region are set to the preset cleanup value via the batch setting interface.

[0137] Optionally, the cleaning module 504 is further configured to: When the target write path is written via the copy interface, the target tensor is copied from the source valid region in the source storage region to the target valid region via the copy interface; The copy interface is used to copy the padding value of the source out-of-bounds region in the source storage area to the target out-of-bounds region.

[0138] Optionally, the cleaning module 504 is further configured to: In test mode, if the output result of the target valid area is consistent with the preset result, and the target out-of-bounds area does not contain non-preset cleanup values, then the copy interface test is determined to be passed. If the output result of the target valid area is inconsistent with the preset result, or if the target out-of-bounds area contains a non-preset cleanup value, then the copy interface test is determined to be unsuccessful.

[0139] In this embodiment, a target storage region is allocated for the target tensor. Based on the number of elements contained in the target tensor and the number of bytes corresponding to each element, the logically valid number of bytes of the target tensor is obtained. When the physical storage bytes of the target storage region are greater than the logically valid number of bytes, it is determined that there is a target out-of-bounds region in the target storage region, and the target write path of the target tensor is identified. When the target write path is written through a target operator, and the target operator belongs to the operator type for creating a new output tensor, the target out-of-bounds region in the target storage region is cleaned up before the target storage region is read by the operator executed after the target operator. Then, the target tensor is written into the target valid region of the target storage region through the target operator. That is, during the process of writing the target tensor into the target valid region through the target operator, the target out-of-bounds region is cleaned up at the same time. This ensures the cleanliness of the target out-of-bounds region. Therefore, when the operator executed after the target operator reads the target tensor in the target storage region, the tensor calculation unit no longer needs to perform out-of-bounds judgment or cleanup of the out-of-bounds region, thereby effectively reducing the performance overhead caused by additional operations, while ensuring the accuracy of tensor calculation, and thus improving the overall performance of the chip.

[0140] Based on the same technical concept, embodiments of this application provide a computer device, such as... Figure 6 As shown, it includes at least one artificial intelligence chip 100 and a memory 601 connected to at least one artificial intelligence chip 100. In this embodiment, the specific connection medium between the artificial intelligence chip 100 and the memory 601 is not limited. Figure 6 Taking the connection between the AI ​​chip 100 and the memory 601 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0141] In this embodiment of the application, the memory 601 stores instructions that can be executed by at least one artificial intelligence chip 100. By executing the instructions stored in the memory 601, at least one artificial intelligence chip 100 can perform the steps of the above-mentioned out-of-bounds area clearing method.

[0142] The artificial intelligence chip 100 serves as the control center of the computer device. It connects to various parts of the computer device via various interfaces and lines, and performs boundary clearing by running or executing instructions stored in the memory 601 and accessing data stored in the memory 601. Optionally, the artificial intelligence chip 100 may include one or more processing units. The artificial intelligence chip 100 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 601 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.

[0143] The artificial intelligence chip 100 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0144] Memory 601, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 601 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory 601 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer device, but is not limited thereto. Memory 601 in the embodiments of this application may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0145] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the above-described out-of-bounds area cleanup method.

[0146] Based on the same inventive concept, this application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described out-of-bounds area clearing method.

[0147] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for clearing outbound areas, characterized in that, include: Allocate a target storage region for the target tensor; Based on the number of elements contained in the target tensor and the number of bytes corresponding to each element, the logical effective number of bytes of the target tensor is obtained; When the number of physical storage bytes in the target storage region is greater than the number of logical valid bytes, the target write path of the target tensor is identified. When the target write path is written through a target operator, and the target operator is an operator type for creating a new output tensor, before the target storage area is read by the operator executed after the target operator, the target out-of-bounds region in the target storage area is cleaned up, and the target tensor is written into the target valid region in the target storage area through the target operator.

2. The method as described in claim 1, characterized in that, The step of cleaning up the out-of-bounds region in the target storage area includes: The target out-of-bounds region can be cleaned up by calling a public out-of-bounds cleanup function; or, a preset cleanup value can be written to the target out-of-bounds region through the target operator.

3. The method as described in claim 1, characterized in that, Also includes: When the target write path is written through a target operator, and the target operator belongs to the operator type that modifies an existing tensor, the target tensor is written into the target valid region through the target operator.

4. The method as described in claim 2 or 3, characterized in that, After the target operator is executed, it also includes: The target out-of-bounds area is cleared and detected by a detection operator, and the detection results are obtained. When the detection result indicates that the target out-of-bounds area contains a non-preset cleanup value, the target out-of-bounds area is cleaned up by a cleanup operator.

5. The method as described in claim 4, characterized in that, The target operator, the detection operator, and the cleanup operator are executed sequentially in the same task flow.

6. The method as described in claim 1 or 3, characterized in that, After the target operator is executed, it also includes: In test mode, if the output result of the target valid area is consistent with the preset result, and the target out-of-bounds area does not contain non-preset cleanup values, then the target operator test is determined to be passed. If the output result of the target valid region is inconsistent with the preset result, or if the target out-of-bounds region contains a non-preset cleanup value, then the target operator test is determined to be unsuccessful.

7. The method as described in claim 1, characterized in that, Also includes: When the target write path is written via a batch setting interface, and the target tensor's fill value is a preset cleanup value, the target out-of-bounds region and the target valid region are set to the preset cleanup value via the batch setting interface.

8. The method as described in claim 1, characterized in that, Also includes: When the target write path is written via the copy interface, the target tensor is copied from the source valid region in the source storage region to the target valid region via the copy interface; The copy interface is used to copy the padding value of the source out-of-bounds region in the source storage area to the target out-of-bounds region.

9. The method as described in claim 8, characterized in that, After copying the padding value of the source out-of-bounds region in the source storage area to the target out-of-bounds region through the copy interface, the method further includes: In test mode, if the output result of the target valid area is consistent with the preset result, and the target out-of-bounds area does not contain non-preset cleanup values, then the copy interface test is determined to be passed. If the output result of the target valid area is inconsistent with the preset result, or if the target out-of-bounds area contains a non-preset cleanup value, then the copy interface test is determined to be unsuccessful.

10. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored on the memory and running on the artificial intelligence chip, characterized in that, When the artificial intelligence chip executes the computer program, it implements the steps of the method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, It stores a computer program that is executed by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the method as described in any one of claims 1 to 9.

12. A computer program product, characterized in that, The computer program product includes a computer program stored on a computer-readable storage medium, the computer program including program instructions that, when executed by a computer device, cause the computer device to perform the steps of the method as described in any one of claims 1 to 9.

Images