Optimization method and related equipment of attention operator based on Ascend AI processor

Abstract

The present invention relates to the field of data processing technology, and provides an optimization method and related equipment for an attention operator based on an Ascend AI processor. By transferring input data of a multi-scale deformable attention operator to an AI Core, the input data is directly used inside the AI ​​Core to determine the position and size of a target feature map and the coordinates of a target interpolation point, and the reference coordinates and corresponding reference interpolation indexes required for the target interpolation point are calculated, thereby avoiding the need to transfer a large amount of data to a CPU for processing. Through parallel computing and a local interpolation strategy, the target interpolation value and its weight of the target interpolation point are calculated on the AI ​​Core, and at the same time, the fifth input data related to the target interpolation weight is transferred to the AI ​​Core, and the output value calculation of the attention operator is completed inside the AI ​​Core, thereby reducing the number of data transmissions between the inside and outside of the processor and improving the overall processing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an optimization method for an attention operator based on an Ascend AI processor and related equipment. Background Art

[0002] In the field of computer vision and image processing, Multi-Scale Deformable Attention, as an efficient attention mechanism, has become a key technology to improve model performance by virtue of its ability to adaptively weight features at different scales and extract highly representative features. However, on Huawei's self-developed Ascend AI processor, a high-performance domestic AI hardware platform, directly implementing and calling this operator faces technical challenges. Summary of the invention

[0003] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0004] To this end, the first purpose of the present invention is to propose an optimization method for the attention operator based on the Ascend AI processor. Through the deep integration of the algorithm and the hardware architecture, the efficient operation of the multi-scale deformable attention operator on the Ascend AI processor is realized, thereby improving the model reasoning speed.

[0005] The second object of the present invention is to propose an optimization method for an attention operator based on an Ascend AI processor, comprising: obtaining first input data, second input data, third input data, fourth input data and fifth input data of the attention operator, and transferring the second input data, the third input data and the fourth input data to the AI ​​Core; determining the width and height of the target feature map according to the second input data, determining the starting position of the target feature map in the spliced ​​feature map according to the third input data, and determining the coordinates of the target interpolation point according to the fourth input data; wherein the spliced ​​feature map is obtained by splicing the input feature maps of all attention operators, the target feature map is included in the spliced ​​feature map, and the target interpolation point is included in the target feature map; obtaining at least four reference coordinates of the target interpolation point according to the coordinates of the target interpolation point; determining the indexes of at least four reference interpolations corresponding to the at least four reference coordinates according to the width and height of the target feature map, the starting position of the target feature map in the spliced ​​feature map, the coordinates of the target interpolation point and the at least four reference coordinates of the target interpolation point, and transferring the at least four reference interpolations from the first input data to the AI ​​Core according to the indexes.

[0006] According to an optimization method of an attention operator based on an Ascend AI processor of the present invention, the input data of the multi-scale deformable attention operator (the second, third, fourth input data and the part of the first input data required subsequently) is transported to the AI ​​Core (the core acceleration unit of the Ascend AI processor) through a data transport strategy to reduce the overhead and delay of data movement. The second, third, and fourth input data are directly used inside the AI ​​Core to determine the position, size, and coordinates of the target feature map and the target interpolation point, and the reference coordinates and corresponding reference interpolation indexes required for the target interpolation point are calculated, avoiding the need to move a large amount of data to the CPU for processing. Through parallel computing and local interpolation strategies, the target interpolation and its weight of the target interpolation point are directly calculated on the AI ​​Core, and the fifth input data related to the target interpolation weight is also transported to the AI ​​Core, and the calculation of the target interpolation weight is completed inside the AI ​​Core, reducing the intervention of the CPU and improving the overall efficiency. Finally, the output value calculation of the multi-scale deformable attention operator is completed inside the AI ​​Core, and the output value is moved out to the subsequent processing unit, reducing the number of data transmissions between the inside and outside of the processor, and improving the overall processing efficiency.

[0007] In addition, the data processing method according to the above embodiment of the present invention may also have the following additional technical features:

[0008] As an optional embodiment, after transferring at least four reference interpolation values ​​from the first input data to the AI ​​Core according to the index, the method further includes:

[0009] respectively calculating at least four reference interpolation weights corresponding to at least four reference interpolation values;

[0010] Determining a target interpolation value of a target interpolation point according to at least four reference interpolation values ​​and at least four reference interpolation weights;

[0011] The fifth input data is transferred to the AI ​​Core, and a target interpolation weight of the target interpolation point is determined according to the fifth input data;

[0012] Determine the output value of the attention operator according to the target interpolation and the target interpolation weight;

[0013] Move the output value out of AI Core.

[0014] As an optional embodiment, obtaining first input data, second input data, third input data, fourth input data, and fifth input data of the attention operator includes:

[0015] The first input data, the second input data, the third input data, the fourth input data and the fifth input data are stored in AI Core's external storage area; wherein, the first input data includes a concatenated feature map concatenated by L input feature maps, and the size is [bs, S, M, D]; the second input data includes the width and height of each input feature map in the L input feature maps, and the size is [L, 2]; the third input data includes the starting position of each input feature map in the concatenated feature map in the L input feature maps, and the size is [L]; the fourth input data includes the coordinates of the interpolation points sampled from the input feature map of the first input data, and the size is [bs, Lq, M, L, P, 2]; the fifth input data includes the attention weights corresponding to the coordinates of the interpolation points sampled on the input feature map of the first input data, and the size is [bs, Lq, M, L, P]; bs represents the number of samples used in a training process, S represents the length of the input feature map, M represents the number of attention heads in the self-attention mechanism, D represents the hidden dimension, L represents the number of input feature maps, Lq represents the ratio of the encoder feature map length to the decoder Query length, and P represents the number of target interpolation points.

[0016] As an optional embodiment, the second input data, the third input data, and the fourth input data are transferred to the AI ​​Core, including:

[0017] From the second input data located at the starting position 0 of the external storage area, move the second transfer data of size [L, 2] to the AI ​​Core;

[0018] From the third input data located at the starting position 0 of the external storage area, move the third transfer data of size [L] to the AI ​​Core;

[0019] From the fourth input data located at the starting position of coreIdx*M*L*P*2 in the external storage area, the fourth transfer data of size [M, L, P, 2] is moved to the AI ​​Core.

[0020] As an optional embodiment, determining the width and height of the target feature map according to the second input data includes:

[0021] The dimension of the second transfer data is transformed from [L, 2] to [2, L, P, M], and the dimensionally transformed second transfer data is split into the width and height of the target feature map according to the highest dimension; wherein the width of the target feature map is [L, P, M], and the height of the target feature map is [L, P, M].

[0022] As an optional embodiment, determining the position of the target feature map in the spliced ​​feature map according to the third input data further includes:

[0023] The dimension of the third transfer data is transformed from [L] to [L, P, M].

[0024] As an optional embodiment, determining the coordinates of the target interpolation point according to the fourth input data includes:

[0025] The dimension of the fourth transfer data is transformed from [M, L, P, 2] to [2, L, P, M], and the dimension-transformed fourth transfer data is split into a first coordinate and a second coordinate of a target interpolation point according to the highest dimension; wherein the size of the first coordinate of the target interpolation point is [L, P, M], and the size of the second coordinate of the target interpolation point is [L, P, M].

[0026] As an optional embodiment, at least four reference coordinates of the target interpolation point are obtained according to the coordinates of the target interpolation point, including:

[0027]

[0028] in, is the rounding function, is the first coordinate of the target interpolation point, is the second coordinate of the target interpolation point, is the first reference coordinate of the target interpolation point, is the second reference coordinate of the target interpolation point, is the third reference coordinate of the target interpolation point, The fourth reference coordinate of the target interpolation point.

[0029] As an optional embodiment, determining the indexes of at least four reference interpolations corresponding to the at least four reference coordinates according to the width and height of the target feature map, the position of the target feature map in the spliced ​​feature map, the coordinates of the target interpolation point, and at least four reference coordinates of the target interpolation point includes:

[0030]

[0031] in, is the height of the target feature map, is the width of the target feature map, is the index of the first reference coordinate and the second reference coordinate at the height of the target feature map, is the index of the third reference coordinate and the fourth reference coordinate at the height of the target feature map, is the index of the first reference coordinate and the third reference coordinate on the width of the target feature map, is the index of the second reference coordinate and the fourth reference coordinate on the width of the target feature map, is the second input data, is the index of AI Core, is the index of the attention head in the self-attention mechanism, is the index of the first reference interpolation corresponding to the first reference coordinate, is the index of the second reference interpolation corresponding to the second reference coordinate, is the index of the third reference interpolation corresponding to the third reference coordinate, is the index of the fourth reference interpolation value corresponding to the fourth reference coordinate.

[0032] As an optional embodiment, respectively calculating at least four reference interpolation weights corresponding to at least four reference interpolation values ​​includes:

[0033]

[0034] in, is the weight of the first reference interpolation, is the weight of the second reference interpolation, is the weight of the third reference interpolation, is the weight of the fourth reference interpolation, is the index of the target interpolation point at the height of the target feature map, The index of the target interpolation point on the width of the target feature map.

[0035] As an optional embodiment, determining a target interpolation value of a target interpolation point according to at least four reference interpolations and at least four reference interpolation weights includes:

[0036]

[0037] in, is the target interpolation point, is the first reference interpolation, is the second reference interpolation, is the third reference interpolation, Interpolate for the fourth reference.

[0038] As an optional embodiment, the fifth input data is transferred to the AI ​​Core, including:

[0039] From the fifth input data located at the starting position coreIdx*M*L*P in the external storage area, move the fifth transfer data of size [M,L,P] to the AI ​​Core.

[0040] The second object of the present invention is to propose an optimization system for an attention operator based on an Ascend AI processor, comprising:

[0041] An input module, configured to obtain first input data, second input data, third input data, fourth input data, and fifth input data of the attention operator, and transfer the second input data, the third input data, and the fourth input data to the AI ​​Core;

[0042] A data preparation module is configured to determine the width and height of a target feature map according to the second input data, determine the starting position of the target feature map in the concatenated feature map according to the third input data, and determine the coordinates of a target interpolation point according to the fourth input data; wherein the concatenated feature map is obtained by concatenating the input feature maps of all attention operators, the target feature map is included in the concatenated feature map, and the target interpolation point is included in the target feature map;

[0043] The interpolation calculation module is configured to obtain at least four reference coordinates of the target interpolation point according to the coordinates of the target interpolation point; determine the indexes of at least four reference interpolations corresponding to the at least four reference coordinates according to the width and height of the target feature map, the starting position of the target feature map in the spliced ​​feature map, the coordinates of the target interpolation point, and the at least four reference coordinates of the target interpolation point, and move the at least four reference interpolations from the first input data to the AI ​​Core according to the indexes.

[0044] According to an optimization system of an attention operator based on an Ascend AI processor of the present invention, the input data of the multi-scale deformable attention operator (the second, third, fourth input data and the part of the first input data required subsequently) is transported to the AI ​​Core (the core acceleration unit of the Ascend AI processor) through a data transport strategy to reduce the overhead and delay of data movement. The second, third, and fourth input data are directly used inside the AI ​​Core to determine the position, size, and coordinates of the target feature map and the target interpolation point, and the reference coordinates and corresponding reference interpolation indexes required for the target interpolation point are calculated, avoiding the need to move a large amount of data to the CPU for processing. Through parallel computing and local interpolation strategies, the target interpolation and its weight of the target interpolation point are directly calculated on the AI ​​Core, and the fifth input data related to the target interpolation weight is also transported to the AI ​​Core, and the calculation of the target interpolation weight is completed inside the AI ​​Core, reducing the intervention of the CPU and improving the overall efficiency. Finally, the output value calculation of the multi-scale deformable attention operator is completed inside the AI ​​Core, and the output value is moved out to the subsequent processing unit, reducing the number of data transmissions between the inside and outside of the processor, and improving the overall processing efficiency.

[0045] To achieve the above objectives, the third aspect of the present invention proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the optimization method of the attention operator based on the Ascend AI processor as described above is implemented.

[0046] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned optimization method of the attention operator based on the Ascend AI processor.

[0047] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A schematic diagram of the architecture of the Ascend AI processor provided in an embodiment of the present invention.

[0050] Figure 2 A schematic diagram of the hardware architecture of the AI ​​Core provided in an embodiment of the present invention.

[0051] Figure 3 Schematic diagram of an optimization system for an attention operator based on an Ascend AI processor provided in an embodiment of the present invention.

[0052] Figure 4 A schematic diagram of target interpolation points and reference interpolation points provided by an embodiment of the present invention.

[0053] Figure 5 A flowchart of an optimization method for an attention operator based on an Ascend AI processor provided in an embodiment of the present invention. Figure 6 A schematic diagram of a splicing feature map when L=4 provided in an embodiment of the present invention.

[0054] Figure 7 Another flowchart of the optimization method of the attention operator based on the Ascend AI processor provided in an embodiment of the present invention.

[0055] Figure 8 A more specific schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0057] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0058] As mentioned in the background technology section, Multi-Scale Deformable Attention is an attention mechanism for computer vision tasks. It has achieved good results in image processing and computer vision tasks and is widely used in various application fields. Its advantage is that it can adaptively weight features at different scales to extract more representative features. At the same time, because it only needs to apply local and sparse attention weights to the corresponding values, it can greatly accelerate the convergence of the model and improve the efficiency of reasoning.

[0059] Currently, due to the limitations of hardware architecture and operator optimization, the multi-scale deformable attention operator cannot be efficiently executed directly on the core acceleration unit (such as the AI ​​accelerator) of the Ascend AI processor, but needs to be decomposed into a complex sequence of operations centered on grid sampling and processed by the AI ​​CPU. This processing method not only increases the computational complexity, but also significantly reduces the inference speed of the model, restricting the performance advantages and application potential of the Ascend AI processor in computer vision tasks.

[0060] Therefore, there is an urgent need for a multi-scale deformable attention operator optimization method optimized for the Ascend AI processor, which aims to achieve efficient operation of the operator on the AI ​​accelerator through the deep integration of the algorithm and hardware architecture, thereby improving the model reasoning speed and meeting the growing needs of computer vision applications.

[0061] The technical solution of the present invention is further described in detail below through specific embodiments.

[0062] refer to Figure 1 , which is a schematic diagram of the architecture of the Ascend AI processor provided in an embodiment of the present invention.

[0063] The architecture diagram of the Ascend AI processor is mainly composed of AI Core, AI CPU, data vision preprocessing module, task scheduler, L2 / HBM / DDR memory area, etc.

[0064] Among them, AI Core is the computing engine of the Ascend AI processor, which is designed to handle large-scale matrix operations, vector operations, and tensor operations in deep learning. These operations are crucial in the forward propagation and backpropagation of neural networks. AI Core can provide extremely high computing efficiency through highly parallel computing units (such as matrix multiplication units) and optimized instruction sets, meeting the strict requirements for computing power and speed in AI applications.

[0065] The AI ​​CPU plays a supporting role in the AI ​​processor, and is mainly responsible for handling tasks with complex logic and dense branches, or specific computationally intensive tasks that the AI ​​Core cannot handle directly (such as complex number operations). In addition, the AI ​​CPU is also responsible for handling the management of control flow and data flow to ensure the stability and efficiency of the overall operation of the AI ​​processor. Compared with the AI ​​Core, the AI ​​CPU is more flexible in processing non-matrix, non-vector, and non-tensor type calculations, and can cope with various complex computing scenarios.

[0066] The data vision preprocessing module is mainly used to preprocess the input data, including but not limited to data cleaning, normalization, data enhancement, etc. In AI applications, high-quality input data is the prerequisite for obtaining good model performance, so the importance of the data preprocessing module is self-evident. By optimizing the data preprocessing process, the efficiency and accuracy of data processing can be improved, thereby providing more valuable data input for subsequent AI Core and AI CPU calculations.

[0067] The task scheduler is responsible for coordinating the task allocation and scheduling between the AI ​​Core, AI CPU, and memory resources. It rationally arranges the use of computing resources and memory resources according to the nature and priority of the tasks to ensure the efficiency and stability of the overall operation of the AI ​​processor. The task scheduler usually adopts a variety of optimization strategies, such as load balancing and priority scheduling, to cope with different computing scenarios and requirements.

[0068] L2 cache, HBM (high bandwidth memory) and DDR (double data rate memory) together constitute the global memory system (GM) of the Ascend AI processor. These storage resources provide the necessary data storage space for the AI ​​Core and AI CPU. Among them, HBM is particularly important in AI applications with its high bandwidth and low latency. It can significantly improve data access speed and reduce waiting time during the calculation process. DDR, with its large capacity and high cost-effectiveness, provides AI processors with more abundant data storage options.

[0069] refer to Figure 2 , is a schematic diagram of the hardware architecture of the AI ​​Core provided in an embodiment of the present invention.

[0070] AI Core is mainly divided into computing unit, storage system and control unit. The computing unit mainly contains three basic computing resources, namely matrix computing unit (Cube Unit), vector computing unit (Vector Unit) and scalar computing unit (Scalar Unit).

[0071] Among them, the matrix computing unit (Cube Unit) is mainly used to process tensor computing-intensive tasks, such as convolution and matrix multiplication. These operations are extremely common in deep learning algorithms and are the core calculations in the forward and backward propagation processes of neural networks. The vector computing unit (Vector Unit) is mainly used to perform vector computing-intensive tasks. Vector computing is highly efficient in processing data sequences, array operations, etc., and can accelerate key steps in many AI algorithms. The scalar computing unit (Scalar Unit) is mainly responsible for executing the control logic of the program, such as branch judgment, loop control, etc. Although this unit is not the most computing-intensive part of the AI ​​Core, it is crucial to ensure the correct execution of the program and process control.

[0072] The storage system can be divided into LOA / LOB / LOC buffer areas, unified buffer (UB), and scalar buffer areas / registers.

[0073] Among them, the LOA / LOB / LOC caches are optimized for tensors, vectors, and specific types of calculations, respectively. By storing frequently accessed data in a cache closer to the computing unit, the latency and bandwidth consumption of accessing external storage (such as L2 cache, HBM, or DDR) can be reduced. The Unified Buffer (UB) can provide a more flexible cache mechanism to support different types of computing and data access modes, which can be shared by multiple computing units to improve cache utilization and reduce data redundancy. The scalar cache / register is mainly used to store the data and operation results required by the scalar computing unit. Since scalar data is usually small and frequently accessed, storing this data in registers can further increase the access speed. The L1 cache is located inside the AI ​​Core and is used to temporarily store data that needs to be used repeatedly. By reducing the number of times data is read and written from the bus, the L1 cache can significantly reduce latency and improve overall computing performance.

[0074] The control unit is mainly responsible for coordinating operations between the computing unit and the storage system to ensure the correct execution of instructions and the effective flow of data. It includes components such as instruction decoder, instruction queue, branch predictor, etc. to support complex control flow and data flow management.

[0075] In addition, the operator running on the AI ​​Core of the Ascend AI processor is called the TBE operator, which has higher execution efficiency. Ascend provides two development methods for TBE operators, namely TBE-DSL (Tensor Boost Engine-DomainSpecific Language) and TBE-TIK (Tensor Iterator Kernel). The TBE-DSL development method provides a pre-packaged function interface for developers to call, and can only implement a simple combination of the provided interfaces. The TBE-TIK development method has the highest flexibility, and can directly open the underlying instructions of the AI ​​Core to developers to call, and the developed operators run more efficiently.

[0076] Therefore, in the embodiments of the present application, the TBE-TIK development method is adopted.

[0077] refer to Figure 3 , which is a schematic diagram of an optimization system for an attention operator based on an Ascend AI processor provided in an embodiment of the present invention.

[0078] Specifically, the method can be implemented in five modules, including an input module 302 , a data preparation module 304 , an interpolation calculation module 306 , a matrix multiplication module 308 , and an output module 3010 .

[0079] Among them, the input module 302 is responsible for moving the input data of the multi-scale deformable attention operator from the external storage area (GM) to the unified buffer (UB) of the AI ​​Core.

[0080] The data preparation module 304 is responsible for organizing and transforming the input data carried by the input module 302 for use by the subsequent calculation module. In addition, the initialization and dimension transformation of some constant vectors and the dimension transformation of the intermediate results of the calculation module are also performed in this module.

[0081] The interpolation calculation module 306 calculates the interpolation point based on the bilinear interpolation algorithm. It has two inputs: one is the coordinates of the interpolation point, and the other is the width and height of the feature map where the interpolation point is located. Its output is the value of the interpolation point. It mainly completes three calculation tasks: one is to calculate the coordinates of the four reference interpolation points near the interpolation point based on the two inputs. , , , ; The second is to calculate the weights of the four reference interpolation points near the interpolation point , , , ; The third is to calculate the value of the target interpolation point .

[0082] The matrix multiplication module 308 is responsible for calculating matrix multiplication. It has two inputs: one is the sample data obtained from the first input data based on the fourth input data, and the other is the fifth input data. The output is the matrix multiplication of the two inputs in the L*P dimension, which is performed N*Lq*M times in total.

[0083] The output module 3010 is responsible for moving the output data from the unified memory area (UB) of the AI ​​Core to the external storage area (GM) for output.

[0084] Specifically, the data preparation module 304 performs calculations on the storage conversion unit of the AI ​​Core. Common data dimension conversion operations are:

[0085] Dimension transformation: for example, transform [bs,Lq,M,L,P] to [bs,Lq ,L,P,M].

[0086] Dimension expansion: for example, transforming [L] to [M,L,P].

[0087] Data concatenation: For example, concatenate 4 [M, L, P] data into data of size [4, M, L, P].

[0088] For the interpolation calculation module 306, there are two inputs: one is the coordinates of the target interpolation point, expressed as (locH, locW), and its dimension is [bs, Lq, M, L, P]; the other is the width and height of the target feature map where the target interpolation point is located, expressed as (spatialH, spatialW). It has one output: the value of the target interpolation point, expressed as val, and its dimension is [bs, Lq, M, L, P, D]. This module completes the calculation on the vector calculation unit of the AI ​​Core. Its basic calculation process is to execute a loop of bs*Lq*M*L*P times, and for each loop:

[0089] First, the coordinates of the four reference points near the interpolation point are calculated based on the two inputs. , , , .

[0090] refer to Figure 4 , is a schematic diagram of target interpolation points and reference interpolation points provided in an embodiment of the present invention.

[0091]

[0092] in, is the rounding function, is the first coordinate of the target interpolation point, is the second coordinate of the target interpolation point, is the first reference coordinate of the target interpolation point, is the second reference coordinate of the target interpolation point, is the third reference coordinate of the target interpolation point, The fourth reference coordinate of the target interpolation point.

[0093] It should be noted that It can be understood as the coordinate of the target interpolation point in the height dimension of the target feature map. It can be understood as the coordinate of the target interpolation point in the width dimension of the target feature map, Figure 4 The target interpolation point is the center, its first reference coordinate is the coordinate of the reference interpolation point in the upper left corner, its second reference coordinate is the coordinate of the reference interpolation point in the upper right corner, its third reference coordinate is the coordinate of the reference interpolation point in the lower left corner, and its fourth reference coordinate is the coordinate of the reference interpolation point in the lower right corner.

[0094] Further, , , , The input module is sent to the input module, and the input module is indexed from the first input data external storage area according to the dimension index. , , , At the coordinate position, the data of length D (i.e. the reference interpolation corresponding to each reference interpolation point) is moved to the UB memory area of ​​the AI ​​Core to obtain , , , (All dimensions are [D]) 4 data.

[0095] Furthermore, the weights of the four reference points near the interpolation point are calculated based on the two inputs: , , , ,in:

[0096]

[0097] in, is the weight of the first reference interpolation, is the weight of the second reference interpolation, is the weight of the third reference interpolation, is the weight of the fourth reference interpolation.

[0098] Finally, the target interpolation value of the target interpolation point is calculated (Dimension [D]):

[0099]

[0100] in, is the target interpolation point, is the first reference interpolation, is the second reference interpolation, is the third reference interpolation, Interpolate for the fourth reference.

[0101] For the matrix multiplication module, there are two inputs: one is the sampled data obtained from the first input data based on the fourth input data (i.e., the target interpolation) , whose dimension is [bs,Lq,M,L,P,D]; the second is the fifth input data, represented as attnWeight, whose dimension is [bs,Lq,M,L,P]. The output is the final calculation result of the operator. This module performs N*Lq*M matrix multiplications of the [L*P] matrix and the [L*P,D] matrix. This module completes the calculation on the matrix calculation unit of the AI ​​Core.

[0102] The parallel optimization methods used in the embodiments of the present application include multi-core parallelism, vectorized parallelism, memory access optimization, and double buffer mechanism, among which:

[0103] For multi-core parallelism, an Ascend AI processor contains multiple AI Cores, which are independent of each other and can perform the same computing tasks in parallel. In the TBE TIK operator development tool, you can set the parallelism of multi-core parallelism on the outermost loop, and the setting of parallelism usually needs to be greater than or equal to the actual number of AI Cores so that the scheduler can perform full-load scheduling.

[0104] In this embodiment of the present invention, the parallelism of multi-core parallelism is set to bs*Lq, and the index on each AI Core is expressed as .

[0105] For vectorized parallelism, vectorized parallelism is usually implemented using Single-Instruction-Multi-Data (SIMD) instructions. For AI Core, the matrix calculation unit can complete a 16x16 and 16x16 matrix multiplication (4096) of fp16 in one clock cycle. The vector calculation unit can complete two 128-length fp16 type vector operations in one clock cycle. The storage conversion unit can complete a 16x16 and 16x16 matrix transposition of fp16 in one clock cycle.

[0106] In an embodiment of the present invention, the degree of parallelism of vectorized parallelism of each AI Core is set to M*L*P.

[0107] That is, for each AI Core:

[0108] In the data preparation module, it is necessary to move M*L*P*2 size data from the starting position of the fourth input data GM storage area coreIdx*M*L*P*2 into the UB memory area as (dimensions are [M,L,P]) and (Dimensions are [M,L,P]).

[0109] Move M*L*P size data from the starting position of the fifth input data GM storage area coreIdx*M*L*P into the UB memory area as attnWeight (dimension is [M,L,P]).

[0110] A total of M*L*P*4 (4 reference points) times [D] size data is moved from the first input data GM storage area through the UB memory area as v1, v2, v3, v4 (dimension is [M, L, P, D]).

[0111] In the interpolation calculation module, the vector calculation unit is required to calculate data of size M*L*P each time.

[0112] In the matrix multiplication module, since matrix multiplication needs to be performed in the L*P dimension, a loop of M times is set. For each loop, the matrix multiplication of the [L*P] matrix and the [L*P,D] matrix is ​​calculated.

[0113] Regarding memory access optimization, in the multi-scale deformable attention operator, the first input data needs to be moved N*Lq*M*L*P times of data of size [D] from the GM storage area to the UB memory area in actual operation, and the address of each transfer is not continuous. This feature leads to the emergence of a memory access bottleneck and becomes the focus of optimization.

[0114] In this solution, considering the physical relationship between the first input data and the second input data and the third input data, an on-chip cache is used to reduce the interaction with the external storage area. At the same time, considering that the coordinates of each input feature map in the L input feature maps may be more continuous, the data dimensions [M, L, P] in the AI ​​Core are transformed from [L, P, M] with M as the first dimension to [L, P, M] with L as the first dimension. Specifically:

[0115] Set a first input data on-chip cache in the UB memory area, with a size of K*M*D;

[0116] The coordinates of the four reference points obtained by the interpolation calculation module are , , , (The dimension is [M, L, P]), first perform dimension transformation, and the transformed dimension is [L, P, M];

[0117] The coordinates of the four reference points are concatenated together to obtain data of dimension [L, 4, P, M].

[0118] Construct a loop of number L, and for each iteration:

[0119] Get the maximum value max, minimum value min and the difference diff between them in the [4,P,M] length data (diff=max-min);

[0120] When diff<=K, data of size diff*M*D is pulled from the starting position min of the first input data GM storage area to the cache, and subsequent v1, v2, v3, and v4 are directly read from the cache;

[0121] When diff>K, v1, v2, v3, v4 are read from the first input data GM storage area.

[0122] For the Double Buffer mechanism, vector instructions, matrix instructions, and storage conversion instructions on the AI ​​Core are sent to the vector operation unit, matrix operation unit, and storage conversion unit through different queues for scheduling. Instructions in these different queues can be executed in parallel.

[0123] In this solution, the interpolation calculation module is executed on the vector operation unit, the matrix multiplication module is executed on the matrix calculation unit, and the data preparation module is executed on the storage conversion unit. The three modules are executed in parallel through the double buffer mechanism.

[0124] Below, the optimization method process of the attention operator based on the Ascend AI processor provided by the present invention will be described in detail in combination with specific embodiments.

[0125] refer to Figure 5 , which is a flow chart of an optimization method for an attention operator based on an Ascend AI processor provided in an embodiment of the present invention.

[0126] Step S502, obtain the first input data, the second input data, the third input data, the fourth input data and the fifth input data of the attention operator, and transfer the second input data, the third input data and the fourth input data to AICore.

[0127] As an optional embodiment, the obtaining of the first input data, the second input data, the third input data, the fourth input data and the fifth input data of the attention operator includes: storing the first input data, the second input data, the third input data, the fourth input data and the fifth input data in the AI Core's external storage area; wherein, the first input data includes a concatenated feature map concatenated by L input feature maps, and the size is [bs, S, M, D]; the second input data includes the width and height of each input feature map in the L input feature maps, and the size is [L, 2]; the third input data includes the starting position of each input feature map in the concatenated feature map in the L input feature maps, and the size is [L]; the fourth input data includes the coordinates of the interpolation points sampled from the input feature map of the first input data, and the size is [bs, Lq, M, L, P, 2]; the fifth input data includes the attention weights corresponding to the coordinates of the interpolation points sampled on the input feature map of the first input data, and the size is [bs, Lq, M, L, P]; bs represents the number of samples used in a training process, S represents the length of the input feature map, M represents the number of attention heads in the self-attention mechanism, D represents the hidden dimension, L represents the number of input feature maps, Lq represents the ratio of the encoder feature map length to the decoder Query length, and P represents the number of target interpolation points.

[0128] Specifically, the multi-scale deformable attention operator has one output, represented as output data (i.e., output value), with a size of (bs, Lq, M, D). There are five inputs, represented as:

[0129] The first input data In1, of size [bs, S, M, D], represents a total input feature map (i.e., a concatenated feature map) concatenated from L input feature maps.

[0130] The second input data In2 has a size of [L, 2], which represents the width and height of each input feature map in the L input feature maps.

[0131] The third input data In3 has a size of [L], which represents the starting position of each of the L input feature maps in the concatenated feature map.

[0132] The fourth input data In4, whose size is [bs, Lq, M, L, P, 2], represents the coordinates of the sampling points sampled from the first input data (that is, the coordinates of the interpolation points sampled from the input feature map of the first input data), and a total of bs*Lq*M*L*P sampling times are required.

[0133] The fifth input data, of size [bs, Lq, M, L, P], represents the attention weight, which corresponds one-to-one to the sampling points obtained by sampling the first input data, that is, the first input data is sampled bs*Lq*M*L*P times in total, corresponding to bs*Lq*M*L*P attention weights of the fifth input data.

[0134] Among them, bs represents the batch size. bs (Batch Size) represents the number of samples processed simultaneously during a training process. In deep learning, data is divided into multiple batches for iterative training, and each batch contains a fixed number of samples. S represents the length of the input feature map. M represents the number of attention heads in the self-attention mechanism. In the multi-head attention mechanism, M represents the number of attention heads. Each head independently learns different representations of the input data and splices these representations together to obtain richer information. Multi-head attention is one of the core components of the Transformer model and is widely used in natural language processing and computer vision tasks. D represents the hidden dimension. L represents the number of input feature maps. In computer vision tasks, L can represent the number or length of the input feature maps. When processing feature maps of multiple scales, L can refer to the number of feature maps at different scales. Lq represents the ratio of the encoder feature map length to the decoder query length. In multi-scale deformable attention or other similar mechanisms, Lq represents a length measure or factor related to the query, which can be used to adjust the interaction between the query and the key or value, especially when processing sequences or feature maps of different lengths. P represents the number of target interpolation points. In the multi-scale deformable attention mechanism, P usually represents the number of points sampled at each query or position. Deformable attention allows the model to dynamically adjust the sampling position according to the characteristics of the input data, so as to capture relevant information more accurately. P can determine the degree of fineness of sampling at each query or position, which has a certain impact on the performance of the model.

[0135] refer to Figure 6 , which is a schematic diagram of a splicing feature map when L=4 provided in an embodiment of the present invention.

[0136] As an optional embodiment, the relationship between the first input and the second input is: the first input data In1 is composed of L feature maps, the size of each feature map is N*M*D, N[l]=In2[l][0]*In2[l][1], 0<= l <L。

[0137] The relationship between the first input data In1 and the third input data In3 is: the third input data In3 represents the starting position of each feature map in the L input feature maps in the first input data In1.

[0138] The relationship between the second input data In2 and the third input data In3 is: In3[l]=In3[l-1]+In2[l][0]×In2[l][1], where 0<=l <L,In3[0] = 0。

[0139] As an optional embodiment, the moving the second input data, the third input data, and the fourth input data to the AI ​​Core includes: moving the second transfer data of size [L, 2] from the second input data located at the starting position 0 of the external storage area to the AI ​​Core; moving the third transfer data of size [L] from the third input data located at the starting position 0 of the external storage area to the AI ​​Core; and moving the fourth transfer data of size [M, L, P, 2] from the fourth input data located at the starting position coreIdx*M*L*P*2 of the external storage area to the AI ​​Core.

[0140] Step S504, determining the width and height of the target feature map according to the second input data, determining the starting position of the target feature map in the spliced ​​feature map according to the third input data, and determining the coordinates of the target interpolation point according to the fourth input data; wherein the spliced ​​feature map is obtained by splicing the input feature maps of all attention operators, the target feature map is included in the spliced ​​feature map, and the target interpolation point is included in the target feature map.

[0141] As an optional embodiment, determining the width and height of the target feature map based on the second input data includes: transforming the dimension of the second transfer data from [L, 2] to [2, L, P, M], and splitting the dimensionally transformed second transfer data into the width and height of the target feature map according to the highest dimension; wherein the width of the target feature map is [L, P, M], and the height of the target feature map is [L, P, M].

[0142] As an optional embodiment, determining the position of the target feature map in the concatenated feature map based on the third input data also includes: transforming the dimension of the third transfer data from [L] to [L, P, M].

[0143] As an optional embodiment, determining the coordinates of the target interpolation point according to the fourth input data includes: transforming the dimension of the fourth transfer data from [M, L, P, 2] to [2, L, P, M], and splitting the dimensionally transformed fourth transfer data into a first coordinate and a second coordinate of the target interpolation point according to the highest dimension; wherein the size of the first coordinate of the target interpolation point is [L, P, M], and the size of the second coordinate of the target interpolation point is [L, P, M].

[0144] Specifically, the transfer data loaded into the UB memory needs to be sorted and dimensionally transformed. Specifically:

[0145] Transform the dimension of the second transfer data from [L, 2] to [2, L, P, M], and split it into two input data according to the highest dimension, denoted as spatialH (size is [L, P, M]) and spatialW (size is [L, P, M]);

[0146] Transform the dimension of the third transfer data from [L] to [L, P, M];

[0147] Transform the dimension of the fourth transfer data from [M, L, P, 2] to [2, L, P, M], and split it into two input data according to the highest dimension, denoted as locW (size is [L, P, M]) and locH (size is [L, P, M]);

[0148] In addition, you need to create a one-dimensional data with a length of M and storage content (1,2,3,…,M), represented as mIdx, and then transform its dimension from [M] to [L,P,M].

[0149] Step S506, obtaining at least four reference coordinates of the target interpolation point according to the coordinates of the target interpolation point.

[0150] As an optional embodiment, at least four reference coordinates of the target interpolation point are obtained according to the coordinates of the target interpolation point, including:

[0151]

[0152] in, is the rounding function, is the first coordinate of the target interpolation point, is the second coordinate of the target interpolation point, is the first reference coordinate of the target interpolation point, is the second reference coordinate of the target interpolation point, is the third reference coordinate of the target interpolation point, The fourth reference coordinate of the target interpolation point.

[0153] Step S508, determine the indexes of at least four reference interpolations corresponding to at least four reference coordinates according to the width and height of the target feature map, the starting position of the target feature map in the spliced ​​feature map, the coordinates of the target interpolation point, and at least four reference coordinates of the target interpolation point, and move the at least four reference interpolations from the first input data to AICore according to the indexes.

[0154] As an optional embodiment, the step of determining the indexes of at least four reference interpolations corresponding to at least four reference coordinates according to the width and height of the target feature map, the position of the target feature map in the spliced ​​feature map, the coordinates of the target interpolation point, and at least four reference coordinates of the target interpolation point includes:

[0155]

[0156] in, is the height of the target feature map, is the width of the target feature map, is the index of the first reference coordinate and the second reference coordinate at the height of the target feature map, is the index of the third reference coordinate and the fourth reference coordinate at the height of the target feature map, is the index of the first reference coordinate and the third reference coordinate on the width of the target feature map, is the index of the second reference coordinate and the fourth reference coordinate on the width of the target feature map, is the second input data, is the index of AI Core, is the index of the attention head in the self-attention mechanism, is the index of the first reference interpolation corresponding to the first reference coordinate, is the index of the second reference interpolation corresponding to the second reference coordinate, is the index of the third reference interpolation corresponding to the third reference coordinate, is the index of the fourth reference interpolation value corresponding to the fourth reference coordinate.

[0157] It should be noted that the calculation process executes SIMD vectorized parallelism with a parallelism of L*P*M, and the dimensions of the output coordinates of the four reference points are all [L, P, M].

[0158] refer to Figure 7 , which is another flow chart of the optimization method of the attention operator based on the Ascend AI processor provided in an embodiment of the present invention.

[0159] Step S702: Calculate at least four reference interpolation weights corresponding to at least four reference interpolations respectively.

[0160] As an optional embodiment, respectively calculating at least four reference interpolation weights corresponding to at least four reference interpolation values ​​includes:

[0161]

[0162] in, is the weight of the first reference interpolation, is the weight of the second reference interpolation, is the weight of the third reference interpolation, is the weight of the fourth reference interpolation, is the index of the target interpolation point at the height of the target feature map, The index of the target interpolation point on the width of the target feature map.

[0163] It should be noted that in this step, the dimensions of w1, w2, w3, and w4 need to be transformed from [L, M, P] to [L, M, P, D].

[0164] Step S704: determining a target interpolation value of a target interpolation point according to at least four reference interpolations and at least four reference interpolation weights.

[0165] As an optional embodiment, determining a target interpolation value of a target interpolation point according to at least four reference interpolations and at least four reference interpolation weights includes:

[0166]

[0167] in, is the target interpolation point, is the first reference interpolation, is the second reference interpolation, is the third reference interpolation, Interpolate for the fourth reference.

[0168] It should be noted that in this step, val needs to be transformed in dimension from [L, P, M, D] to [M, D, L, P].

[0169] Step S706: The fifth input data is transferred to the AI ​​Core, and a target interpolation weight of the target interpolation point is determined according to the fifth input data.

[0170] As an optional embodiment, the fifth input data is transferred to the AI ​​Core, including:

[0171] From the fifth input data located at the starting position coreIdx*M*L*P in the external storage area, move the fifth transfer data of size [M,L,P] to the AI ​​Core.

[0172] Step S708, determining the output value of the attention operator according to the target interpolation and the target interpolation weight.

[0173] As an optional embodiment, in this step, val (dimension is [M, D, L, P]) is used as input one, and the target interpolation weight obtained from the fifth input data in step S5014 is used as input two, and the following matrix multiplication is performed: [D, LP]x [LP] -> [D], and finally an output value with a dimension of [M, D] is obtained.

[0174] Step S710, moving the output value out of the AI ​​Core.

[0175] As an optional embodiment, the output value is moved to the starting position of the output data external storage area coreIdx*M*D.

[0176] It should be noted that the Ascend AI processor used in the test of the embodiment of the present invention is Ascend 910ProB, and the development tool uses CANN 7.0 RC1 version. In addition, the optimization method of the attention operator based on the Ascend AI processor described in the present invention can also use other versions according to actual needs.

[0177] Refer to Table 1 for a comparison of the operating efficiency of the multi-scale deformable attention operator on the Ascend 910ProB.

[0178] Specifically, the AI ​​CPU implementation means that the multi-scale deformable attention operator is decomposed into a set of operations with grid sampling as the core on the Ascend AI processor, and grid sampling, as a core operation, can only be run on the AI ​​CPU.

[0179] The AI ​​Core implementation represents the multi-scale deformable attention operator using the AI ​​Core end-to-end implementation proposed in this solution, where [bs, S, Lq, M, D, L, P] represents different sizes of the input.

[0180] Table 1

[0181]

[0182] As can be seen from Table 1, this solution can bring about a 10-fold acceleration gain, greatly improving the operating efficiency of operators on the Ascend AI processor.

[0183] Refer to Table 2 for a comparison of the running speed of models using the multi-scale deformable attention operator on the Ascend 910ProB.

[0184] Table 2

[0185]

[0186] Specifically, the implementation of a multi-scale deformable attention operator on the Ascend AI processor is decomposed into a set of operations with grid sampling as the core. Grid sampling, as a core operation, can only be run on the AI ​​CPU.

[0187] The implementation of the two-representation multi-scale deformable attention operator adopts the AI ​​Core end-to-end implementation proposed in this solution.

[0188] GroudingDINO in Table 2 is a target detection model based on transformer architecture. The model contains 6 layers of multi-scale deformable attention operators in the Encoder module and the Decoder module respectively. The input size of the multi-scale deformable attention operator in the Encoder module is [bs, S, Lq, M, D, L, P] = [1, 34000, 34000, 8, 32, 4, 4], and the input size of the multi-scale deformable attention operator in the Decoder module is [bs, S, Lq, M, D, L, P] = [1, 34000, 900, 8, 32, 4, 4].

[0189] As can be seen from Table 2, the overall running speed of the model is increased by 6 times by adopting this solution.

[0190] As can be seen from the above, the present invention provides an optimization method for the attention operator based on the Ascend AI processor. Through the data transfer strategy, the input data of the multi-scale deformable attention operator (the second, third, fourth input data and the part of the first input data required subsequently) is transferred to the AI ​​Core (the core acceleration unit of the Ascend AI processor) to reduce the overhead and delay of data movement. The second, third, and fourth input data are directly used inside the AI ​​Core to determine the position, size, and coordinates of the target feature map and the target interpolation point, and the reference coordinates and corresponding reference interpolation indexes required for the target interpolation point are calculated, avoiding the need to move a large amount of data to the CPU for processing. Through parallel computing and local interpolation strategies, the target interpolation of the target interpolation point and its weight are directly calculated on the AI ​​Core, and the fifth input data related to the target interpolation weight is also transferred to the AI ​​Core, and the calculation of the target interpolation weight is completed inside the AI ​​Core, reducing the intervention of the CPU and improving the overall efficiency. Finally, the output value calculation of the multi-scale deformable attention operator is completed inside the AI ​​Core, and the output value is moved out to the subsequent processing unit, reducing the number of data transmissions between the inside and outside of the processor and improving the overall processing efficiency.

[0191] It should be noted that the method of the embodiment of the present invention can be performed by a single device, such as a computer or a server. The method of this embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present invention, and the multiple devices will interact with each other to complete the described method.

[0192] It should be noted that some embodiments of the present invention are described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0193] Based on the same inventive concept, corresponding to the method provided in any of the above embodiments, the present invention also provides an optimization system based on an attention operator based on an Ascend AI processor.

[0194] like Figure 3 As shown, it is a schematic diagram of the optimization system of the attention operator based on the Ascend AI processor provided in an embodiment of the present invention.

[0195] The Ascend AI processor-based attention operator optimization system 300 includes:

[0196] An input module 302 is configured to obtain first input data, second input data, third input data, fourth input data, and fifth input data of the attention operator, and transfer the second input data, the third input data, and the fourth input data to the AI ​​Core;

[0197] The data preparation module 304 is configured to determine the width and height of the target feature map according to the second input data, determine the starting position of the target feature map in the spliced ​​feature map according to the third input data, and determine the coordinates of the target interpolation point according to the fourth input data; wherein the spliced ​​feature map is obtained by splicing the input feature maps of all attention operators, the target feature map is included in the spliced ​​feature map, and the target interpolation point is included in the target feature map;

[0198] The interpolation calculation module 306 is configured to obtain at least four reference coordinates of the target interpolation point according to the coordinates of the target interpolation point; determine the indexes of at least four reference interpolations corresponding to the at least four reference coordinates according to the width and height of the target feature map, the starting position of the target feature map in the spliced ​​feature map, the coordinates of the target interpolation point, and the at least four reference coordinates of the target interpolation point, and move the at least four reference interpolations from the first input data to the AI ​​Core according to the indexes.

[0199] Optionally, the interpolation calculation module 306 is further configured to:

[0200] respectively calculating at least four reference interpolation weights corresponding to at least four reference interpolation values;

[0201] Determining a target interpolation value of a target interpolation point according to at least four reference interpolation values ​​and at least four reference interpolation weights;

[0202] The fifth input data is transferred to the AI ​​Core, and a target interpolation weight of the target interpolation point is determined according to the fifth input data;

[0203] Determine the output value of the attention operator according to the target interpolation and the target interpolation weight;

[0204] Move the output value out of AI Core.

[0205] Optionally, the input module 302 is further configured to:

[0206] The first input data, the second input data, the third input data, the fourth input data and the fifth input data are stored in AI Core's external storage area; wherein, the first input data includes a concatenated feature map concatenated by L input feature maps, and the size is [bs, S, M, D]; the second input data includes the width and height of each input feature map in the L input feature maps, and the size is [L, 2]; the third input data includes the starting position of each input feature map in the concatenated feature map in the L input feature maps, and the size is [L]; the fourth input data includes the coordinates of the interpolation points sampled from the input feature map of the first input data, and the size is [bs, Lq, M, L, P, 2]; the fifth input data includes the attention weights corresponding to the coordinates of the interpolation points sampled on the input feature map of the first input data, and the size is [bs, Lq, M, L, P]; bs represents the number of samples used in a training process, S represents the length of the input feature map, M represents the number of attention heads in the self-attention mechanism, D represents the hidden dimension, L represents the number of input feature maps, Lq represents the ratio of the encoder feature map length to the decoder Query length, and P represents the number of target interpolation points.

[0207] Optionally, the interpolation calculation module 306 is further configured to:

[0208] From the second input data located at the starting position 0 of the external storage area, move the second transfer data of size [L, 2] to the AI ​​Core;

[0209] From the third input data located at the starting position 0 of the external storage area, move the third transfer data of size [L] to the AI ​​Core;

[0210] From the fourth input data located at the starting position of coreIdx*M*L*P*2 in the external storage area, the fourth transfer data of size [M, L, P, 2] is moved to the AI ​​Core.

[0211] Optionally, the data preparation module 304 is further configured to:

[0212] The dimension of the second transfer data is transformed from [L, 2] to [2, L, P, M], and the dimensionally transformed second transfer data is split into the width and height of the target feature map according to the highest dimension; wherein the width of the target feature map is [L, P, M], and the height of the target feature map is [L, P, M].

[0213] Optionally, the data preparation module 304 is further configured to:

[0214] The dimension of the third transfer data is transformed from [L] to [L, P, M].

[0215] Optionally, the data preparation module 304 is further configured to:

[0216] The dimension of the fourth transfer data is transformed from [M, L, P, 2] to [2, L, P, M], and the dimension-transformed fourth transfer data is split into a first coordinate and a second coordinate of a target interpolation point according to the highest dimension; wherein the size of the first coordinate of the target interpolation point is [L, P, M], and the size of the second coordinate of the target interpolation point is [L, P, M].

[0217] Optionally, the interpolation calculation module 306 is further configured to:

[0218]

[0219] in, is the rounding function, is the first coordinate of the target interpolation point, is the second coordinate of the target interpolation point, is the first reference coordinate of the target interpolation point, is the second reference coordinate of the target interpolation point, is the third reference coordinate of the target interpolation point, The fourth reference coordinate of the target interpolation point.

[0220] Optionally, the interpolation calculation module 306 is further configured to:

[0221]

[0222] in, is the height of the target feature map, is the width of the target feature map, is the index of the first reference coordinate and the second reference coordinate at the height of the target feature map, is the index of the third reference coordinate and the fourth reference coordinate at the height of the target feature map, is the index of the first reference coordinate and the third reference coordinate on the width of the target feature map, is the index of the second reference coordinate and the fourth reference coordinate on the width of the target feature map, is the second input data, is the index of AI Core, is the index of the attention head in the self-attention mechanism, is the index of the first reference interpolation corresponding to the first reference coordinate, is the index of the second reference interpolation corresponding to the second reference coordinate, is the index of the third reference interpolation corresponding to the third reference coordinate, is the index of the fourth reference interpolation value corresponding to the fourth reference coordinate.

[0223] Optionally, the interpolation calculation module 306 is further configured to:

[0224]

[0225] in, is the weight of the first reference interpolation, is the weight of the second reference interpolation, is the weight of the third reference interpolation, is the weight of the fourth reference interpolation, is the index of the target interpolation point at the height of the target feature map, The index of the target interpolation point on the width of the target feature map.

[0226] Optionally, the interpolation calculation module 306 is further configured to:

[0227]

[0228] in, is the target interpolation point, is the first reference interpolation, is the second reference interpolation, is the third reference interpolation, Interpolate for the fourth reference.

[0229] Optionally, the interpolation calculation module 306 is further configured to:

[0230] From the fifth input data located at the starting position coreIdx*M*L*P in the external storage area, move the fifth transfer data of size [M,L,P] to the AI ​​Core.

[0231] According to an optimization system of an attention operator based on an Ascend AI processor of the present invention, the input data of the multi-scale deformable attention operator (the second, third, fourth input data and the part of the first input data required subsequently) is transported to the AI ​​Core (the core acceleration unit of the Ascend AI processor) through a data transport strategy to reduce the overhead and delay of data movement. The second, third, and fourth input data are directly used inside the AI ​​Core to determine the position, size, and coordinates of the target feature map and the target interpolation point, and the reference coordinates and corresponding reference interpolation indexes required for the target interpolation point are calculated, avoiding the need to move a large amount of data to the CPU for processing. Through parallel computing and local interpolation strategies, the target interpolation and its weight of the target interpolation point are directly calculated on the AI ​​Core, and the fifth input data related to the target interpolation weight is also transported to the AI ​​Core, and the calculation of the target interpolation weight is completed inside the AI ​​Core, reducing the intervention of the CPU and improving the overall efficiency. Finally, the output value calculation of the multi-scale deformable attention operator is completed inside the AI ​​Core, and the output value is moved out to the subsequent processing unit, reducing the number of data transmissions between the inside and outside of the processor, and improving the overall processing efficiency.

[0232] For the convenience of description, the above system is described by dividing it into various modules according to its functions. Of course, when implementing the present invention, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0233] The system of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0234] Based on the same inventive concept, corresponding to the method described in any of the above embodiments, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any of the above embodiments when executing the program.

[0235] Figure 8 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 810, a memory 820, an input / output interface 830, a communication interface 840, and a bus 850. The processor 810, the memory 820, the input / output interface 830, and the communication interface 840 are connected to each other in communication within the device through the bus 850.

[0236] The processor 810 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0237] The memory 820 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 820 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 820 and are called and executed by the processor 810.

[0238] The input / output interface 830 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0239] The communication interface 840 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0240] The bus 850 includes a path that transmits information between the various components of the device (eg, the processor 810 , the memory 820 , the input / output interface 830 , and the communication interface 840 ).

[0241] It should be noted that, although the above device only shows the processor 810, the memory 820, the input / output interface 830, the communication interface 840 and the bus 850, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0242] The electronic device of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0243] Based on the same inventive concept, corresponding to the method described in any of the above embodiments, the present invention also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in any of the above embodiments.

[0244] The above-mentioned non-transitory computer-readable storage medium can be any available medium or data storage device that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.

[0245] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the method described in any embodiment of the above exemplary method part, and have the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0246] In addition, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0247] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0248] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the embodiments of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0249] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and this division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims. The scope of the attached claims conforms to the broadest interpretation, thereby including all such modifications and equivalent structures and functions.

Claims

1. An optimization method for an attention operator based on an Ascend AI processor, characterized in that: include: Obtaining first input data, second input data, third input data, fourth input data, and fifth input data of the attention operator includes: The first input data, the second input data, the third input data, the fourth input data, and the fifth input data are stored in the AI Core's external storage area; wherein the first input data includes a concatenated feature map formed by concatenating L input feature maps, and the size is [bs, S, M, D]; the second input data includes the width and height of each input feature map in the L input feature maps, and the size is [L, 2]; the third input data includes the starting position of each input feature map in the concatenated feature map in the L input feature maps, and the size is [L]; the fourth input data includes the coordinates of the interpolation points sampled from the input feature map of the first input data, and the size is [bs, Lq, M, L, P, 2]; the fifth input data includes the attention weights corresponding to the coordinates of the interpolation points sampled on the input feature map of the first input data, and the size is [bs, Lq, M, L, P]; bs represents the number of samples used in a training process, S represents the length of the input feature map, M represents the number of attention heads in the self-attention mechanism, D represents the hidden dimension, L represents the number of input feature maps, Lq represents the ratio of the encoder feature map length to the decoder Query length, and P represents the number of target interpolation points; Transferring the second input data, the third input data, and the fourth input data to the AI ​​Core; Determine the width and height of the target feature map according to the second input data, determine the starting position of the target feature map in the spliced ​​feature map according to the third input data, and determine the coordinates of the target interpolation point according to the fourth input data; wherein the spliced ​​feature map is obtained by splicing the input feature maps of all the attention operators, the target feature map is included in the spliced ​​feature map, and the target interpolation point is included in the target feature map; Obtaining at least four reference coordinates of the target interpolation point according to the coordinates of the target interpolation point; Determine the indexes of at least four reference interpolations corresponding to the at least four reference coordinates according to the width and height of the target feature map, the starting position of the target feature map in the spliced ​​feature map, the coordinates of the target interpolation point, and the at least four reference coordinates of the target interpolation point, and transfer the at least four reference interpolations from the first input data to the AI ​​Core according to the indexes; respectively calculating at least four reference interpolation weights corresponding to the at least four reference interpolation values; Determining a target interpolation value of the target interpolation point according to the at least four reference interpolation values ​​and the at least four reference interpolation weights; The fifth input data is transferred to the AI ​​Core, and a target interpolation weight of the target interpolation point is determined according to the fifth input data; Determining an output value of the attention operator according to the target interpolation and the target interpolation weight; Move the output value out of the AI ​​Core.

2. The method for optimizing the attention operator based on the Ascend AI processor according to claim 1, characterized in that: The transferring the second input data, the third input data, and the fourth input data to the AI ​​Core includes: From the second input data located at the starting position 0 of the external storage area, transfer the second transfer data of size [L, 2] to the AI ​​Core; From the third input data located at the starting position 0 of the external storage area, transfer the third transfer data of size [L] to the AI ​​Core; From the fourth input data located at the starting position of coreIdx*M*L*P*2 in the external storage area, the fourth transfer data of size [M, L, P, 2] is transferred to the AI ​​Core.

3. The method for optimizing the attention operator based on the Ascend AI processor according to claim 2, characterized in that: The step of determining the width and height of the target feature map according to the second input data comprises: The dimension of the second transfer data is transformed from [L, 2] to [2, L, P, M], and the dimensionally transformed second transfer data is split into the width and height of the target feature map according to the highest dimension; wherein the width of the target feature map is [L, P, M], and the height of the target feature map is [L, P, M].

4. The method for optimizing the attention operator based on the Ascend AI processor according to claim 3, characterized in that: The step of determining the position of the target feature map in the spliced ​​feature map according to the third input data further includes: The dimension of the third transfer data is transformed from [L] to [L, P, M].

5. The method for optimizing the attention operator based on the Ascend AI processor according to claim 4, characterized in that: The step of determining the coordinates of the target interpolation point according to the fourth input data comprises: The dimension of the fourth transfer data is transformed from [M, L, P, 2] to [2, L, P, M], and the dimensionally transformed fourth transfer data is split into a first coordinate and a second coordinate of the target interpolation point according to the highest dimension; wherein the size of the first coordinate of the target interpolation point is [L, P, M], and the size of the second coordinate of the target interpolation point is [L, P, M].

6. The method for optimizing the attention operator based on the Ascend AI processor according to claim 5, characterized in that: The obtaining at least four reference coordinates of the target interpolation point according to the coordinates of the target interpolation point includes: in, is the rounding function, is the first coordinate of the target interpolation point, is the second coordinate of the target interpolation point, is the first reference coordinate of the target interpolation point, is the second reference coordinate of the target interpolation point, is the third reference coordinate of the target interpolation point, The fourth reference coordinate of the target interpolation point.

7. The method for optimizing the attention operator based on the Ascend AI processor according to claim 6, characterized in that: The determining, according to the width and height of the target feature map, the position of the target feature map in the spliced ​​feature map, the coordinates of the target interpolation point, and the at least four reference coordinates of the target interpolation point, indexes of at least four reference interpolations corresponding to the at least four reference coordinates includes: in, is the height of the target feature map, is the width of the target feature map, is the index of the first reference coordinate and the second reference coordinate at the height of the target feature map, is the index of the third reference coordinate and the fourth reference coordinate at the height of the target feature map, is the index of the first reference coordinate and the third reference coordinate on the width of the target feature map, is the index of the second reference coordinate and the fourth reference coordinate on the width of the target feature map, is the second input data, is the index of AI Core, is the index of the attention head in the self-attention mechanism, is the index of the first reference interpolation value corresponding to the first reference coordinate, is the index of the second reference interpolation value corresponding to the second reference coordinate, is the index of the third reference interpolation value corresponding to the third reference coordinate, is the index of the fourth reference interpolation value corresponding to the fourth reference coordinate.

8. The method for optimizing the attention operator based on the Ascend AI processor according to claim 7, characterized in that: The respectively calculating at least four reference interpolation weights corresponding to the at least four reference interpolation values ​​comprises: in, is the weight of the first reference interpolation, is the weight of the second reference interpolation, is the weight of the third reference interpolation, is the weight of the fourth reference interpolation, is the index of the target interpolation point at the height of the target feature map, The index of the target interpolation point on the width of the target feature map.

9. The method for optimizing the attention operator based on the Ascend AI processor according to claim 8, characterized in that: The determining the target interpolation value of the target interpolation point according to the at least four reference interpolation values ​​and the at least four reference interpolation weights includes: in, is the target interpolation point, is the first reference interpolation, is the second reference interpolation, is the third reference interpolation, Interpolate for the fourth reference.

10. The method for optimizing the attention operator based on the Ascend AI processor according to claim 9, characterized in that: The step of transferring the fifth input data to the AI ​​Core includes: From the fifth input data located at the starting position coreIdx*M*L*P of the external storage area, the fifth transfer data of size [M, L, P] is transferred to the AI ​​Core.

11. An optimization system for an attention operator based on an Ascend AI processor, characterized in that: include: An input module, configured to obtain first input data, second input data, third input data, fourth input data, and fifth input data of an attention operator, including: The first input data, the second input data, the third input data, the fourth input data, and the fifth input data are stored in the AI Core's external storage area; wherein the first input data includes a concatenated feature map formed by concatenating L input feature maps, and the size is [bs, S, M, D]; the second input data includes the width and height of each input feature map in the L input feature maps, and the size is [L, 2]; the third input data includes the starting position of each input feature map in the concatenated feature map in the L input feature maps, and the size is [L]; the fourth input data includes the coordinates of the interpolation points sampled from the input feature map of the first input data, and the size is [bs, Lq, M, L, P, 2]; the fifth input data includes the attention weights corresponding to the coordinates of the interpolation points sampled on the input feature map of the first input data, and the size is [bs, Lq, M, L, P]; bs represents the number of samples used in a training process, S represents the length of the input feature map, M represents the number of attention heads in the self-attention mechanism, D represents the hidden dimension, L represents the number of input feature maps, Lq represents the ratio of the encoder feature map length to the decoder Query length, and P represents the number of target interpolation points; Transferring the second input data, the third input data, and the fourth input data to the AI ​​Core; A data preparation module is configured to determine the width and height of a target feature map according to the second input data, determine the starting position of the target feature map in the spliced ​​feature map according to the third input data, and determine the coordinates of a target interpolation point according to the fourth input data; wherein the spliced ​​feature map is obtained by splicing the input feature maps of all the attention operators, the target feature map is included in the spliced ​​feature map, and the target interpolation point is included in the target feature map; an interpolation calculation module, configured to obtain at least four reference coordinates of the target interpolation point according to the coordinates of the target interpolation point; determine the indexes of at least four reference interpolations corresponding to the at least four reference coordinates according to the width and height of the target feature map, the starting position of the target feature map in the spliced ​​feature map, the coordinates of the target interpolation point, and the at least four reference coordinates of the target interpolation point, and carry the at least four reference interpolations from the first input data to the AI ​​Core according to the indexes; respectively calculating at least four reference interpolation weights corresponding to the at least four reference interpolation values; Determining a target interpolation value of the target interpolation point according to the at least four reference interpolation values ​​and the at least four reference interpolation weights; The fifth input data is transferred to the AI ​​Core, and a target interpolation weight of the target interpolation point is determined according to the fifth input data; Determining an output value of the attention operator according to the target interpolation and the target interpolation weight; Move the output value out of the AI ​​Core.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the optimization method of the attention operator based on the Ascend AI processor as described in any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the optimization method of the attention operator based on the Ascend AI processor described in any one of claims 1 to 10.

Images