Visual model operator optimization method and device for domestic hardware

Abstract

The invention provides a domestic hardware-oriented visual model operator optimization method and device, which are applied to a visual model, and the method comprises the steps: expanding a plurality of position coding modes of the visual model according to a relative position code and a condition position code of a relative position coding module of the visual model, performing position coding on the visual model based on a plurality of position coding modes to obtain a structure-optimized target visual model corresponding to each position coding mode, inputting a preset target picture into one target visual model, performing operator abstraction on the target visual model to obtain a corresponding abstracted target visual model, and displaying the abstracted target visual model in a display screen. And obtaining a corresponding relationship between the neural network operator and a domestic hardware operator provided by the domestic hardware, and implementing the neural network operator on the domestic hardware based on the corresponding relationship.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and more specifically, to a method and device for optimizing visual model operators for domestic hardware. Background Art

[0002] As the demand for deep learning grows, technological innovation in AI processors based on neural networks is booming. The neural network processor (NPU) is specially optimized for matrix operations, solving the problem of low efficiency of traditional chips in neural network operations. With the development of domestic neural network processors, it provides powerful computing power support for artificial intelligence and is gradually widely used in various fields. However, as a computing-driven technology, deep learning has an endless demand for computing, which has also driven a large number of researchers to engage in the research and development of computing-friendly deep learning algorithms.

[0003] At present, the Transformer model has introduced the self-attention mechanism and has become the mainstream model in the field of natural language, and has achieved cross-domain development. The Vision Transformer model (ViT) is a deep learning model based on the Transformer architecture. It has made continuous breakthroughs in the field of computer vision and has great application potential in actual production. As a representative neural network model in the field of computer vision, Swin Transformer has good performance in image classification, target detection and semantic segmentation tasks.

[0004] However, the Swin Transformer model has a large number of parameters and high computational complexity, and its inference performance is poor on domestic hardware such as domestic neural network processors. Summary of the invention

[0005] In view of this, the purpose of this application is to provide a visual model operator optimization method and device for domestic hardware, which optimizes the structure of the visual model through multiple position encoding methods to obtain a target visual model, and implements the neural network operator abstracted from the target visual model on domestic hardware, thereby optimizing the visual model structure and reducing the computational complexity of the model. At the same time, the reasoning performance on domestic hardware, such as domestic neural network processors, is also improved.

[0006] In a first aspect, an embodiment of the present application provides a visual model operator optimization method for domestic hardware, which is applied to a visual model, wherein the visual model includes a sliding window self-attention module and a relative position encoding module; the optimization method includes:

[0007] For the visual model, multiple position coding modes of the visual model are expanded according to the relative position coding and conditional position coding of the relative position coding module of the visual model;

[0008] Based on the multiple position encoding methods, the visual models are position-encoded respectively to obtain a target visual model with optimized structure corresponding to each position encoding method; wherein different target visual models represent different model structures; the target visual model adopts a combination of window self-attention and linear space reduction self-attention;

[0009] Inputting a preset target image into a target visual model, performing operator abstraction on the target visual model, and obtaining a corresponding abstracted target visual model; wherein the abstracted target visual model corresponds to the abstracted neural network operator;

[0010] The correspondence between the neural network operator and the domestic hardware operator provided by the domestic hardware is obtained, and the neural network operator is implemented on the domestic hardware based on the correspondence.

[0011] In a possible implementation, the linear space reduction self-attention module of the target visual model is obtained by the following steps:

[0012] The computational complexity formulas of various self-attention calculation methods are used to calculate the computational complexity of each self-attention calculation method.

[0013] Based on the computational amount of each self-attention calculation method, it is determined that the target self-attention calculation method with the smallest computational amount is the linear space reduced self-attention, and the sliding window self-attention module of the visual model is replaced by the target self-attention calculation method to obtain a linear space reduced self-attention module and a corresponding target visual model.

[0014] In a possible implementation, the visual model includes small models and medium models of different sizes; the target visual model includes a plurality of target small models and a target medium model; and position encoding the visual models based on the plurality of position encoding methods respectively includes:

[0015] Determine the optimization parameters of the small model and the medium model respectively according to the basic parameters of the small model and the medium model;

[0016] Based on the optimization parameters and the multiple position encoding methods, the small model and the medium model are optimized respectively to obtain a target small model and a target medium model corresponding to each position encoding method.

[0017] In a possible implementation, performing operator abstraction on the target visual model includes:

[0018] Get the mapping relationship table between different self-attention calculation operations and corresponding operation operators;

[0019] In response to executing a target self-attention calculation operation for the target visual model, a corresponding operation operator is determined in the mapping relationship table based on the target self-attention calculation operation and executed to perform operator abstraction on the target visual model.

[0020] In a possible implementation, the implementing the neural network operator on the domestic hardware based on the corresponding relationship includes:

[0021] In response to inputting a tensor and a window size parameter to the neural network operator, obtaining multi-dimensional data of the tensor;

[0022] On the domestic hardware, based on the first target operator development method on the tensor acceleration engine provided by the domestic hardware and the multi-dimensional data, operator development, data transfer for a preset number of times and dimension setting of the tensor are performed in sequence to obtain a corresponding output tensor.

[0023] In a possible implementation, the implementing the neural network operator on the domestic hardware based on the corresponding relationship includes:

[0024] In response to inputting a tensor, a window size parameter and a feature map parameter into the neural network operator, obtaining multi-dimensional data of the tensor; wherein the feature map parameter includes a feature map height and a feature map width of an original feature map before dividing the window;

[0025] On the domestic hardware, based on the first target operator development method on the tensor acceleration engine provided by the domestic hardware and the multi-dimensional data, operator development, data transfer for a preset number of times and dimension setting of the tensor are performed in sequence to obtain a corresponding output tensor.

[0026] In a possible implementation, the implementing the neural network operator on the domestic hardware based on the corresponding relationship includes:

[0027] In response to inputting a tensor, weights and offsets into the neural network operator, the mean and variance are calculated based on the second target operator development method on the tensor acceleration engine, and normalization and affine transformation are performed to obtain the corresponding output tensor.

[0028] In a second aspect, the embodiment of the present application further provides a visual model operator optimization device for domestic hardware, which is applied to a visual model, wherein the visual model includes a sliding window self-attention module and a relative position encoding module; the optimization device includes:

[0029] An extension module, used for extending a plurality of position encoding modes of the visual model according to the relative position encoding and conditional position encoding of the relative position encoding module of the visual model for the visual model;

[0030] An optimization module, configured to position-encode the visual model based on the plurality of position encoding methods to obtain a target visual model after structural optimization corresponding to each position encoding method; wherein different target visual models represent different model structures; wherein the target visual model adopts a combination of window self-attention and linear space reduction self-attention;

[0031] An abstraction module is used to input a preset target image into a target visual model, perform operator abstraction on the target visual model, and obtain a corresponding abstracted target visual model; wherein the abstracted target visual model corresponds to the abstracted neural network operator;

[0032] An implementation module is used to obtain the correspondence between the neural network operator and the domestic hardware operator provided by the domestic hardware, and implement the neural network operator on the domestic hardware based on the correspondence.

[0033] In a possible implementation, the linear space reduction self-attention module of the target visual model is obtained by the following steps:

[0034] The computational complexity formulas of various self-attention calculation methods are used to calculate the computational complexity of each self-attention calculation method.

[0035] Based on the computational amount of each self-attention calculation method, it is determined that the target self-attention calculation method with the smallest computational amount is the linear space reduced self-attention, and the sliding window self-attention module of the visual model is replaced by the target self-attention calculation method to obtain a linear space reduced self-attention module and a corresponding target visual model.

[0036] In a possible implementation, the visual model includes small models and medium models of different sizes; the target visual model includes a plurality of target small models and target medium models; and the optimization module is specifically used to:

[0037] Determine the optimization parameters of the small model and the medium model respectively according to the basic parameters of the small model and the medium model;

[0038] Based on the optimization parameters and the multiple position encoding methods, the small model and the medium model are optimized respectively to obtain a target small model and a target medium model corresponding to each position encoding method.

[0039] In a possible implementation manner, the abstract module is specifically used to:

[0040] Get the mapping relationship table between different self-attention calculation operations and corresponding operation operators;

[0041] In response to executing a target self-attention calculation operation for the target visual model, a corresponding operation operator is determined in the mapping relationship table based on the target self-attention calculation operation and executed to perform operator abstraction on the target visual model.

[0042] In a possible implementation manner, the implementation module is specifically used to:

[0043] In response to inputting a tensor and a window size parameter to the neural network operator, obtaining multi-dimensional data of the tensor;

[0044] On the domestic hardware, based on the first target operator development method on the tensor acceleration engine provided by the domestic hardware and the multi-dimensional data, operator development, data transfer for a preset number of times and dimension setting of the tensor are performed in sequence to obtain a corresponding output tensor.

[0045] In a possible implementation manner, the implementation module is specifically used to:

[0046] In response to inputting a tensor, a window size parameter and a feature map parameter into the neural network operator, obtaining multi-dimensional data of the tensor; wherein the feature map parameter includes a feature map height and a feature map width of an original feature map before dividing the window;

[0047] On the domestic hardware, based on the first target operator development method on the tensor acceleration engine provided by the domestic hardware and the multi-dimensional data, operator development, data transfer for a preset number of times and dimension setting of the tensor are performed in sequence to obtain a corresponding output tensor.

[0048] In a possible implementation manner, the implementation module is specifically used to:

[0049] In response to inputting a tensor, weights and offsets into the neural network operator, the mean and variance are calculated based on the second target operator development method on the tensor acceleration engine, and normalization and affine transformation are performed to obtain the corresponding output tensor.

[0050] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the visual model operator optimization method for domestic hardware as described in any one of the first aspects.

[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the visual model operator optimization method for domestic hardware described in any one of the first aspects are executed.

[0052] The embodiment of the present application provides a method and device for optimizing visual model operators for domestic hardware. According to the relative position coding and conditional position coding of the relative position coding module of the visual model, a plurality of position coding methods of the visual model are expanded. The visual model is position-coded based on the plurality of position coding methods to obtain a target visual model after structural optimization corresponding to each position coding method. A preset target image is input into a target visual model, and the operator is abstracted on the target visual model to obtain the corresponding abstracted target visual model. The correspondence between the neural network operator and the domestic hardware operator provided by the domestic hardware is obtained, and the neural network operator is implemented on the domestic hardware based on the correspondence. In the present application, the target visual model is structurally optimized by a plurality of position coding methods to obtain the target visual model, and the neural network operator abstracted from the target visual model is implemented on the domestic hardware, so as to optimize the structure of the visual model, thereby reducing the computational complexity of the model. At the same time, the reasoning performance on the domestic hardware, such as the domestic neural network processor, is also improved.

[0053] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 is a flow chart of a visual model operator optimization method for domestic hardware according to an embodiment of the present application;

[0056] Figure 2It is a schematic diagram of the visual model and Block structure;

[0057] Figure 3 It is a schematic diagram of the target visual model and Block structure;

[0058] Figure 4 It is a schematic diagram of the Transformer Block structure;

[0059] Figure 5 It is a schematic diagram of the statistics of self-attention calculation;

[0060] Figure 6 is a flow chart of a visual model operator optimization method for domestic hardware according to another embodiment of the present application;

[0061] Figure 7 It is a schematic diagram of the Window Partition process;

[0062] Figure 8 This is a schematic diagram of the Window Reverse process;

[0063] Fig. 9 It is a schematic diagram of the Patch Merging process;

[0064] Fig.10 It is a structural schematic diagram of a visual model operator optimization device for domestic hardware according to an embodiment of the present application;

[0065] Fig.11 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0066] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0067] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0068] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0069] Considering the increasing demand for deep learning, the technological innovation of AI processors based on neural networks is booming. The neural network processor (NPU) is specially optimized for matrix operations, solving the problem of low efficiency of traditional chips in neural network operations. With the development of domestic neural network processors, it provides strong computing power support for artificial intelligence and has gradually been widely used in various fields. However, as a computing-driven technology, deep learning has an endless demand for computing, which has also driven a large number of researchers to devote themselves to the research and development of computing-friendly deep learning algorithms.

[0070] At present, the Transformer model has introduced the self-attention mechanism and has become the mainstream model in the field of natural language, and has achieved cross-domain development. The Vision Transformer model (ViT) is a deep learning model based on the Transformer architecture. It has made continuous breakthroughs in the field of computer vision and has great application potential in actual production. As a representative neural network model in the field of computer vision, Swin Transformer has good performance in image classification, target detection and semantic segmentation tasks.

[0071] However, the Swin Transformer model has a large number of parameters and high computational complexity, and its inference performance is poor on domestic hardware such as domestic neural network processors.

[0072] To address this problem, the present application provides a method and device for optimizing visual model operators for domestic hardware. The visual model is structurally optimized through a variety of position encoding methods to obtain a target visual model, and the neural network operator abstracted from the target visual model is implemented on domestic hardware, thereby optimizing the visual model structure and reducing the computational complexity of the model. At the same time, the inference performance on domestic hardware, such as domestic neural network processors, is also improved.

[0073] Figure 1 is a flowchart of a visual model operator optimization method for domestic hardware according to an embodiment of the present application, such as Figure 1 As shown, the visual model operator optimization method for domestic hardware in the embodiment of the present application is applied to the visual model, and specifically may include:

[0074] S101, for a visual model, a plurality of position coding modes of the visual model are expanded according to relative position coding and conditional position coding of a relative position coding module of the visual model.

[0075] S102, position encoding the visual model based on multiple position encoding methods, to obtain a target visual model with optimized structure corresponding to each position encoding method.

[0076] S103, inputting a preset target image into a target visual model, performing operator abstraction on the target visual model, and obtaining a corresponding abstracted target visual model.

[0077] S104, obtaining a correspondence between a neural network operator and a domestic hardware operator provided by the domestic hardware, and implementing the neural network operator on the domestic hardware based on the correspondence.

[0078] In the above-mentioned visual model operator optimization method for domestic hardware, the visual model is structurally optimized through a variety of position encoding methods to obtain the target visual model, and the neural network operator abstracted from the target visual model is implemented on domestic hardware, thereby optimizing the visual model structure and reducing the computational complexity of the model. At the same time, the inference performance on domestic hardware such as domestic neural network processors is also improved.

[0079] The above exemplary steps of the embodiment of the present application are described below with reference to specific examples:

[0080] S101, for a visual model, a plurality of position coding modes of the visual model are expanded according to relative position coding and conditional position coding of a relative position coding module of the visual model.

[0081] It should be noted that the visual model is the visual model to be subjected to operator optimization, and the visual model is the visual Transformer model. This application describes the swin Transformer model as the visual Transformer model, but does not constitute a limitation thereto.

[0082] In the embodiment of the present application, the visual model includes a sliding window self-attention (SW-MSA) module and a relative position encoding module. The position encoding method is a lightweight and more optimal position encoding method. The position encoding method includes MLP position encoding, QV matrix position encoding, and input X position encoding. According to the relative position encoding and conditional position encoding of the relative position encoding module of the visual model, a variety of position encoding methods of the visual model are expanded for subsequent processing. For example, Figure 2 As shown in the figure, it represents the visual model, namely the Swin Transformer model and Block structure. It can be seen that the overall design of Swin Transformer is a pyramid structure, which is divided into four stages (Stage). Each Stage is composed of a Patch Merging layer / LinearEmbedding layer and several Swin Transformer blocks (Block). The dimension of the feature map of each Stage is different. The height and width of the feature map of the latter Stage are half of that of the previous Stage, and the number of channels is twice that of the previous Stage.

[0083] It should be noted that this application designs three position encoding methods based on depth-by-depth convolution: MLP position encoding, QV matrix position encoding, and input X position encoding. These three position encoding methods are used to replace the relative position encoding used in the original SwinTransformer model to generate three network structures. Among them, the mechanisms of these three position encoding methods are the same, but they act at different positions in the model.

[0084] S102, position encoding the visual model based on multiple position encoding methods, to obtain a target visual model with optimized structure corresponding to each position encoding method.

[0085] In an embodiment of the present application, the target visual model is a visual model obtained after structural optimization of the visual model. The target visual model adopts a combination of window self-attention (W-MSA) and linear space reduction self-attention (LSRA). The target visual model includes a window self-attention (W-MSA) module and a linear space reduction self-attention (LSRA) module. The visual models are position-encoded based on the multiple position encoding methods obtained in step S101 to obtain a structurally optimized target visual model corresponding to each position encoding method.

[0086] Among them, different target vision models represent different model structures or network structures. Compared with the visual model, the target vision model reduces the amount of calculation while ensuring the model effect. It reduces the amount of calculation of the model by reducing the computational complexity of the model, thereby speeding up the model's reasoning speed.

[0087] It should be noted that, for example, the optimized target visual model and Block structure are as follows Figure 3 As shown in the figure, the model still adopts a pyramid structure with a total of 4 stages, and each stage is composed of several Transformer Blocks. Patch Merging connects two stages to implement downsampling. For an input of size H×W×3, it first passes through the Token Embedding layer to slice the image and linearly map it into tokens with a dimension of C. After being processed by N1 Transformer Blocks, it enters the Patch Merging layer for downsampling, the height and width of the feature map are halved, and the number of channels is doubled. Enter Stage2 and then pass through N2 Transformer Blocks. The processing flow of Stage3 and Stage4 is similar.

[0088] Continuing, the Transformer Block structure is as follows Figure 4 As shown in the figure, each Block contains a self-attention module, an MLP module, two LN layers and two residual connections. There are two types of self-attention modules: W-MSA extracts local attention information and LSRA interacts with global attention information. Blocks containing W-MSA and LSRA appear one after another.

[0089] Among them, the linear space reduction self-attention module of the target visual model is obtained by the following steps:

[0090] Based on the preset computational complexity formulas of multiple self-attention calculation methods, the computational amount of each self-attention calculation method is calculated respectively; based on the computational amount of each self-attention calculation method, the target self-attention calculation method with the smallest computational amount is determined to be the linear space reduction self-attention, and the target self-attention calculation method replaces the sliding window self-attention module of the visual model to obtain the linear space reduction self-attention module and the corresponding target visual model. Among them, the multiple self-attention calculation methods include at least W-MSA, Linformer MSA and LSRA.

[0091] It should be noted that based on the computational complexity formulas of different self-attention calculation methods, their computational amounts are statistically analyzed. From the computational complexity formula, it can be seen that the computational amount is mainly related to the resolution and number of channels of the feature map. Therefore, the number of channels is fixed to 128 here, and the computational amount GFLOPs corresponding to different self-attention calculation methods at different feature map resolutions is tested. Figure 5 shown.

[0092] It can be seen that Figure 5 As shown in the figure, although Linformer MSA has reduced the amount of attention calculation to a certain extent, its complexity is still high, so as the feature map increases, its calculation amount grows at the fastest rate. The linearized LSRA even has a smaller amount of calculation than W-MSA, and LSRA uses global modeling to make up for the shortcomings of W-MSA local modeling. Therefore, it is considered to use LSRA to replace SW-MSA and introduce global modeling. The self-attention module uses a combination of W-MSA and LSRA. In two consecutive Transformer Blocks, the first Block uses W-MSA to calculate local self-attention, and then uses LSRA in the next Block to calculate global attention.

[0093] To continue, the visual model includes small models and medium models of different sizes, and the target visual model includes multiple target small models and target medium models. Optionally, when the visual models are position-encoded based on a plurality of position encoding methods, the optimization parameters of the small model and the medium model are determined respectively for the basic parameters of the small model and the medium model; based on the optimization parameters and a plurality of position encoding methods, the small model and the medium model are optimized respectively to obtain the target small model and the target medium model corresponding to each position encoding method. Among them, the basic parameters include the basic channel number and the basic block number, and the optimization parameters include the target channel number and the target block number.

[0094] Specifically, Swin Transformer has two sizes of models, namely the small model Swin-t and the medium model Swin-s. Swin-t and Swin-s have similar structures, except for the number of Transformer Blocks in the third stage. Based on Swin-t and Swin-s, the optimization model uses the same block number setting, namely the target block number, and reduces the number of channels, namely the target channel number. The optimization model uses keyword naming, and different names are used for models using different position encoding methods. The model that uses LSRA to replace SW-MSA and uses MLP position encoding is named win-lsra-mlpdw, the model that uses LSRA to replace SW-MSA and uses QV matrix position encoding is named win-lsra-qvdw, the model that uses LSRA to replace SW-MSA and uses input X position encoding is named win-lsra-xdw, and the detailed settings of the target small model and target medium model of different variants are shown in Table 1 below.

[0095] Table 1

[0096]

[0097] As can be seen from Table 1 above, the basic parameters of the small model Swin-t are the basic channel number 96 and the basic block number "2, 2, 6, 2", so the optimization parameters of the small model are determined to be the target channel number 64 and the target block number "2, 2, 6, 2". Similarly, the basic parameters of the medium model Swin-s are the basic channel number 96 and the basic block number "2, 2, 18, 2", so the optimization parameters of the medium model are determined to be the target channel number 64 and the target block number "2, 2, 18, 2". Obviously, the target block number of the optimization parameter is the same as the basic block number of the basic parameter, while the target channel number is reduced relative to the basic channel number.

[0098] S103, inputting a preset target image into a target visual model, performing operator abstraction on the target visual model, and obtaining a corresponding abstracted target visual model.

[0099] In the embodiment of the present application, the target image is an image in a target format, for example, in RGB format. The present application describes an image in RGB format as the target image, but does not constitute a limitation thereto. The abstracted target visual model corresponds to the abstracted neural network operator. The neural network operator (that is, the abstract operator) includes a Window Partition operator, a WindowReverse operator, a Patch Merging operator, a LayerNorm operator, and a GELU operator. The preset target image is input into the target visual model obtained in step S102, and the operator abstraction is performed on the target visual model to obtain the corresponding abstracted target visual model.

[0100] Optionally, when performing operator abstraction on the target visual model, a mapping relationship table of different self-attention calculation operations and corresponding operation operators is obtained; in response to executing the target self-attention calculation operation on the target visual model, the corresponding operation operator is determined in the mapping relationship table based on the target self-attention calculation operation and executed to perform operator abstraction on the target visual model. The self-attention calculation operation includes at least W-MSA, LSRA, MLP, and Patch Merging.

[0101] Among them, the operators corresponding to W-MSA and related operations are shown in Table 2 below:

[0102] Table 2

[0103]

[0104] The operators corresponding to LSRA are shown in Table 3 below:

[0105] Table 3

[0106]

[0107] The operators corresponding to MLP are shown in Table 4 below:

[0108] Table 4

[0109]

[0110] The operators corresponding to Patch Merging are shown in Table 5 below:

[0111] Table 5

[0112]

[0113] In addition, the operators corresponding to other operations are shown in Table 6 below:

[0114] Table 6

[0115]

[0116] It should be noted that for the optimized target visual model, that is, the target small model and the target medium model mentioned above, the operator types used by each module of the model can be analyzed according to the processing flow of the model. The image in RGB format is first input, and is first processed into a token form that adapts to the Transformer structure by the Token Embedding module, and then processed by several Transformer Blocks, and then by a Patch Merging module, and so on.

[0117] S104, obtaining a correspondence between a neural network operator and a domestic hardware operator provided by the domestic hardware, and implementing the neural network operator on the domestic hardware based on the correspondence.

[0118] In an embodiment of the present application, the domestic hardware may be a domestic neural network processor, and the present application is described using this as an example; the correspondence between the neural network operator obtained in step S103 and the domestic hardware operator provided by the domestic hardware is obtained, and the neural network operator is implemented on the domestic hardware based on the correspondence.

[0119] People in this field can understand that although different hardware will provide neural network operator libraries, some operators in different neural network operator libraries may be similar in function, but their naming methods are slightly different. Therefore, it is necessary to match these abstracted neural network operators with the operators provided by domestic hardware, so as to understand the differences between the operator implementation methods of domestic hardware and neural network operators, and then determine the direction of operator optimization.

[0120] Among them, the correspondence between neural network operators and domestic hardware operators is shown in Table 7.

[0121] Table 7

[0122]

[0123] The visual model operator optimization method for domestic hardware provided in the embodiment of the present application expands a variety of position coding methods of the visual model according to the relative position coding and conditional position coding of the relative position coding module of the visual model, performs position coding on the visual model based on the multiple position coding methods, obtains the target visual model after structural optimization corresponding to each position coding method, inputs a preset target image into a target visual model, performs operator abstraction on the target visual model, obtains the corresponding abstracted target visual model, obtains the correspondence between the neural network operator and the domestic hardware operator provided by the domestic hardware, and implements the neural network operator on the domestic hardware based on the correspondence. The visual model operator optimization method for domestic hardware of the present application optimizes the structure of the visual model through multiple position coding methods to obtain the target visual model, and implements the neural network operator abstracted from the target visual model on the domestic hardware, thereby optimizing the structure of the visual model, thereby reducing the computational complexity of the model, and at the same time, the reasoning performance on domestic hardware such as the domestic neural network processor is also improved.

[0124] Further, such as Figure 6 As shown, when the neural network operator is a Window Partition operator, the "implementing the neural network operator on domestic hardware based on the corresponding relationship" in step S104 of the above embodiment may specifically include the following steps:

[0125] S601, in response to inputting a tensor and a window size parameter to a neural network operator, obtaining multi-dimensional data of the tensor.

[0126] S602, on domestic hardware, based on the first target operator development method and multi-dimensional data on the tensor acceleration engine provided by the domestic hardware, perform operator development, data transfer for a preset number of times, and tensor dimension setting in sequence to obtain the corresponding output tensor.

[0127] In an embodiment of the present application, a tensor and a window size are input to a neural network operator, namely, a Window Partition operator, to obtain multi-dimensional data of the tensor, and operator development, data transport, and dimension setting are performed based on the first target operator development method and multi-dimensional data on a tensor acceleration engine provided by domestic hardware to obtain an output tensor, thereby realizing the neural network operator.

[0128] It should be noted that the window attention calculation mechanism needs to divide the original entire image into windows of the same size and calculate the attention within the window, and the windows are independent of each other. The Window Partition operator implements the window partitioning. Specifically, the function of this operator is: before calculating the window attention, the data is rearranged and the tensor is divided into windows according to the specified window size for the calculation of the window attention. The partitioning process is as follows: Figure 7 shown.

[0129] In the open source deep learning framework Pytorch, Window Partition corresponds to a function that requires an input tensor x and a window size attribute window_size, i.e., the window size parameter. First, obtain the multi-dimensional data of the input tensor x, i.e., four dimensions: Batch size, feature map height, feature map width, and number of channels. Then use the View operator to divide the window size by the feature map height and feature map width dimensions to obtain the number of windows obtained in the two dimensions. Then use the Permute operator to rearrange the dimensions, and combine the number of windows corresponding to the feature map height and the number of windows corresponding to the feature map width to obtain the total number of windows. Then use the Contiguous operator to ensure that the tensor is continuous in memory. Finally, use the View operator to merge the Batch with the number of windows, the length and width of the feature map become the size of the window, and the number of channels remains the same as the original input.

[0130] The Window Partition operator is implemented on domestic hardware. After analysis, it can be found that the WindowPartition operator rearranges the data in the memory. The operator has an input tensor x, a window size parameter win, and an output tensor res. The operator is developed using the first target operator development method TBE TIK. The data_move interface is required to move data between the global memory and the cache in the AI ​​Core. Finally, the reshape interface is used to set the dimension of the data in the memory.

[0131] As a result, the Window Partition operator was implemented on domestic hardware.

[0132] Further, when the neural network operator is a Window Reverse operator, when the neural network operator is implemented on domestic hardware based on the corresponding relationship, in response to inputting a tensor, a window size parameter and a feature map parameter to the neural network operator, the multi-dimensional data of the tensor is obtained; on the domestic hardware, based on the first target operator development method and multi-dimensional data on the tensor acceleration engine provided by the domestic hardware, the operator development, the preset number of data transfers and the tensor dimension setting are performed in sequence to obtain the corresponding output tensor. Among them, the feature map parameters include the feature map height and feature map width of the original feature map before the window is divided.

[0133] It should be noted that during the entire model execution process, window partitioning is only required when calculating window attention. Other operations are performed on the entire image. Therefore, after the window attention calculation is completed, the data arrangement needs to be restored to its original state. The Window Reverse operator is the inverse process of Window partition, which is responsible for rearranging the partitioned windows and converting them into the format before the window partition. The processing flow is as follows: Figure 8 shown.

[0134] This function requires an input tensor x and three attributes, namely: window size parameter window_size, feature map parameters (height H of the original feature map, width W of the original feature map). First, get the 0 dimension of the input x, which is the product of the Batch dimension of the original tensor and the number of windows. Therefore, the number of windows can be obtained through the height and width of the original feature map and the size of the window, and then the original Batch size can be restored. Then use the View operator to expand the window multiples corresponding to the Batch and feature map height and the window multiples corresponding to the feature map width in the input 0 dimension. Then use the Permute operator to rearrange the dimensions. Then use the Contiguous operator to ensure that the tensor is continuous in memory. Finally, use the View operator to restore Zhang Liang's dimension to before the window is divided.

[0135] The Window Reverse operator is implemented on domestic hardware. After analysis, it is known that the Window Reverse operator rearranges the data in the memory. The operator has an input tensor x, two parameters, which are the height H and width W of the feature map before the window is divided, and an output tensor res. When implementing the operator, the data_move interface and reshape interface of TBE TIK are still used. The data_move interface realizes data transfer, and the reshape interface realizes dimension setting.

[0136] As a result, the Window Reverse operator was implemented on domestic hardware.

[0137] Continuing, when the neural network operator is a Patch Merging operator, generally speaking, in the pyramid model structure, different stages correspond to different feature map sizes and numbers of channels. In convolutional neural networks, this downsampling is achieved through pooling. Many Transformer models use the Patch Merging operation, which can achieve downsampling without losing information. The function of this operator is to connect two stages, downsample the output of the previous stage, halve the feature map, expand the number of channels, and then input it to the next stage. The process is as follows Fig. 9 shown.

[0138] The operator requires an input tensor x. First, the dimension of the input tensor is set according to the parameter. Then, elements are selected in the row and column directions according to the position interval of 2, and these elements are spliced ​​into a new patch. Finally, the Concat operator is used to splice these local area patches in the channel dimension as a whole tensor.

[0139] Since domestic hardware does not support the direct conversion of Pytorch-generated models into the corresponding OM models on domestic hardware, it can support the conversion of ONNX models to OM models. Therefore, it is necessary to first convert the Pytorch-generated model into an ONNX model, and then convert the ONNX model into an OM model. In ONNX, the Patch Merging operation corresponds to multiple small operators, such as Slice, Reshape, and Concat, and the Slice operator is used eight times. The startup of multiple operators is bound to bring additional overhead, so consider merging these small operators into a large operator and implement it on domestic hardware, that is, the PatchMerging operator.

[0140] On domestic hardware, the operator development is carried out using the data_move interface and reshape interface of TBE TIK. The input of the operator is the tensor x, and the output is the tensor res. The data is stored continuously in the global memory, and the data is accessed in strides. The output address is mapped with the input address, and continuous data fragments can be taken for transmission at one time.

[0141] As a result, the Patch Merging operator was implemented on domestic hardware.

[0142] Furthermore, when the neural network operator is a LayerNorm operator, when the neural network operator is implemented on domestic hardware based on the corresponding relationship, in response to inputting a tensor, weight and offset to the LayerNorm operator, the mean and variance are calculated based on the second target operator development method on the tensor acceleration engine, and normalization and affine transformation are performed to obtain the corresponding output tensor.

[0143] Specifically, the TBE DSL interface is used for implementation on domestic hardware, and the input of the LayerNorm operator is determined to be three, tensor x, weight weight, offset bias, and the output is a y. The parameters are eps and dim, which correspond to the parameters eps and normalized_shape of the original Pytorch operator respectively. The implementation process of the operator mainly includes the calculation of the mean E(x), the calculation of the variance Var(x), the normalization calculation, and the affine transformation. Among them, the calculation of the mean mainly uses the reduce_sum interface, vmuls interface, and broadcast interface; the calculation of the variance is similar to the interface used for calculating the mean, and the vsub interface, vmul interface, reduce_sum interface, vmuls interface, and broadcast interface are required; after obtaining the mean and variance, the input tensor needs to be normalized, and the normalized result needs to be affine transformed. When taking the square root of the sum of the variance and eps, in order to ensure the efficiency and accuracy of the calculation, the logarithmic function and the exponential function are combined to simplify the calculation.

[0144] As a result, the LayerNorm operator was implemented on domestic hardware.

[0145] Furthermore, when the neural network operator is a GELU operator, the implementation of the GELU operator on domestic hardware needs to be adapted to the GELU operator of Pytorch. It also uses a single input and a single output and is implemented using the TBE DSL development method, which will not be described in detail here.

[0146] As a result, various abstracted neural network operators were implemented on domestic hardware.

[0147] In general, this application is processed in two parts. The first part is the optimization of the Swin Transformer model structure. Specifically, by improving the sliding window self-attention module and relative position encoding module of the Swin Transformer, three optimized network structures are designed, which reduce the amount of calculation while ensuring the model effect. First, based on the analysis of the sliding window self-attention module and the relative position encoding module of the Swin Transformer, an improvement idea is proposed. Three position encoding methods are extended from relative position encoding and conditional position encoding, including: MLP position encoding, QV matrix position encoding, and input X position encoding; after statistics on the computational amount of several efficient self-attention mechanisms, it is proposed to use the linear space reduced self-attention LSRA with the least computational amount to replace the sliding window self-attention used by the Swin Transformer, while reducing the amount of calculation, introducing the modeling of global information; and three network structures are designed in combination with the three position encoding methods; the second part is the optimization and implementation of the neural network operator on the domestic hardware, specifically, a number of neural network operators are abstracted from the optimized Swin Transformer model structure, and implemented and optimized on the domestic hardware. First, several neural network operators are abstracted based on the optimized model structure and core algorithm; then, based on the characteristics of domestic hardware and the underlying operator design, operator fusion technology is used to utilize the memory access locality between operators to fuse multiple operators to reduce the number of global memory accesses and the additional overhead caused by operator startup. The operators implemented include: Window Partition operator, Window Reverse operator, Patch Merging operator, LayerNorm operator, and GELU operator.

[0148] It should be noted that the visual model operator optimization method for domestic hardware in this application is a visual model operator optimization method for domestic neural network processors, that is, a visual Transformer model operator optimization method for domestic hardware.

[0149] Fig.10 is a flow chart of a visual model operator optimization device for domestic hardware according to an embodiment of the present application, which is applied to a visual model, wherein the visual model includes a sliding window self-attention module and a relative position encoding module; Fig.10 As shown, the visual model operator optimization device 1000 for domestic hardware in the embodiment of the present application may specifically include:

[0150] The extension module 1001 is used to extend a plurality of position coding modes of the visual model according to the relative position coding and conditional position coding of the relative position coding module of the visual model.

[0151] The optimization module 1002 is used to position-encode the visual model based on multiple position encoding methods to obtain a target visual model with optimized structure corresponding to each position encoding method; wherein different target visual models represent different model structures; wherein the target visual model adopts a combination of window self-attention and linear space reduction self-attention.

[0152] The abstraction module 1003 is used to input a preset target image into a target visual model, perform operator abstraction on the target visual model, and obtain a corresponding abstracted target visual model; wherein the abstracted target visual model corresponds to the abstracted neural network operator.

[0153] The implementation module 804 is used to obtain the correspondence between the neural network operator and the domestic hardware operator provided by the domestic hardware, and implement the neural network operator on the domestic hardware based on the correspondence.

[0154] In one possible implementation, the linear space reduction self-attention module of the target visual model is obtained by the following steps:

[0155] The computational complexity formulas of various self-attention calculation methods are used to calculate the computational complexity of each self-attention calculation method.

[0156] Based on the computational amount of each self-attention calculation method, it is determined that the target self-attention calculation method with the smallest computational amount is the linear space reduced self-attention, and the target self-attention calculation method replaces the sliding window self-attention module of the visual model to obtain the linear space reduced self-attention module and the corresponding target visual model.

[0157] In a possible implementation, the visual model includes small models and medium models of different sizes; the target visual model includes a plurality of target small models and target medium models; and the optimization module is specifically used to:

[0158] Based on the basic parameters of the small model and the medium model, the optimization parameters of the small model and the medium model are determined respectively;

[0159] Based on the optimization parameters and multiple position encoding methods, the small model and the medium model are optimized respectively, and the target small model and the target medium model corresponding to each position encoding method are obtained.

[0160] In a possible implementation, the abstract module is specifically used to:

[0161] Get the mapping relationship table between different self-attention calculation operations and corresponding operation operators;

[0162] In response to executing a target self-attention calculation operation for a target visual model, a corresponding operation operator is determined in a mapping relationship table based on the target self-attention calculation operation and executed to perform operator abstraction on the target visual model.

[0163] In a possible implementation, the implementation module is specifically used for:

[0164] In response to inputting a tensor and a window size parameter to a neural network operator, obtaining multi-dimensional data of the tensor;

[0165] On domestic hardware, based on the first target operator development method and multi-dimensional data on the tensor acceleration engine provided by the domestic hardware, operator development, preset number of data transfers and tensor dimension setting are performed in sequence to obtain the corresponding output tensor.

[0166] In a possible implementation, the implementation module is specifically used for:

[0167] In response to inputting a tensor, a window size parameter and a feature map parameter into a neural network operator, multi-dimensional data of the tensor is obtained; wherein the feature map parameter includes a feature map height and a feature map width of an original feature map before dividing the window;

[0168] On domestic hardware, based on the first target operator development method and multi-dimensional data on the tensor acceleration engine provided by the domestic hardware, operator development, preset number of data transfers and tensor dimension setting are performed in sequence to obtain the corresponding output tensor.

[0169] In a possible implementation, the implementation module is specifically used for:

[0170] In response to inputting a tensor, weights and offsets into a neural network operator, a second target operator development method based on a tensor acceleration engine is used to calculate the mean and variance, and normalize and affine transform are performed to obtain a corresponding output tensor.

[0171] The visual model operator optimization device for domestic hardware provided in the embodiment of the present application expands a variety of position coding methods of the visual model according to the relative position coding and conditional position coding of the relative position coding module of the visual model, performs position coding on the visual model based on the multiple position coding methods, obtains the target visual model after structural optimization corresponding to each position coding method, inputs a preset target image into a target visual model, performs operator abstraction on the target visual model, obtains the corresponding abstracted target visual model, obtains the correspondence between the neural network operator and the domestic hardware operator provided by the domestic hardware, and implements the neural network operator on the domestic hardware based on the correspondence. The visual model operator optimization device for domestic hardware of the present application performs structural optimization on the visual model through multiple position coding methods to obtain the target visual model, and implements the neural network operator abstracted from the target visual model on the domestic hardware, thereby optimizing the structure of the visual model, thereby reducing the computational complexity of the model, and at the same time, the reasoning performance on domestic hardware such as the domestic neural network processor is also improved.

[0172] like Fig.11 As shown, an electronic device 1100 provided in an embodiment of the present application includes: a processor 1101, a memory 1102 and a bus, the memory 1102 stores machine-readable instructions executable by the processor 1101, and when the electronic device is running, the processor 1101 communicates with the memory 1102 through the bus, and the processor 1101 executes the machine-readable instructions to perform the steps of the above-mentioned visual model operator optimization method for domestic hardware.

[0173] Specifically, the above-mentioned memory 1102 and processor 1101 can be general-purpose memory and processor, which are not specifically limited here. When the processor 1101 runs the computer program stored in the memory 1102, it can execute the above-mentioned visual model operator optimization method for domestic hardware.

[0174] Corresponding to the above-mentioned visual model operator optimization method for domestic hardware, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, the steps of the above-mentioned visual model operator optimization method for domestic hardware are executed.

[0175] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0176] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0178] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the deployment method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0179] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A visual model operator optimization method for domestic hardware, applied to a visual model, wherein the visual model includes a sliding window self-attention module and a relative position encoding module; characterized in that: The optimization method comprises: For the visual model, multiple position coding modes of the visual model are expanded according to the relative position coding and conditional position coding of the relative position coding module of the visual model; Based on the multiple position encoding methods, the visual models are position-encoded respectively to obtain a target visual model with optimized structure corresponding to each position encoding method; wherein different target visual models represent different model structures; the target visual model adopts a combination of window self-attention and linear space reduction self-attention; Inputting a preset target image into a target visual model, performing operator abstraction on the target visual model, and obtaining a corresponding abstracted target visual model; wherein the abstracted target visual model corresponds to the abstracted neural network operator; The correspondence between the neural network operator and the domestic hardware operator provided by the domestic hardware is obtained, and the neural network operator is implemented on the domestic hardware based on the correspondence.

2. The method according to claim 1, characterized in that The linear space reduction self-attention module of the target visual model is obtained by the following steps: The computational complexity formulas of various self-attention calculation methods are used to calculate the computational complexity of each self-attention calculation method. Based on the computational amount of each self-attention calculation method, it is determined that the target self-attention calculation method with the smallest computational amount is the linear space reduced self-attention, and the sliding window self-attention module of the visual model is replaced by the target self-attention calculation method to obtain a linear space reduced self-attention module and a corresponding target visual model.

3. The method according to claim 1, characterized in that The visual model includes small models and medium models of different sizes; the target visual model includes a plurality of target small models and a target medium model; and the position encoding of the visual models based on the plurality of position encoding methods includes: Determine the optimization parameters of the small model and the medium model respectively according to the basic parameters of the small model and the medium model; Based on the optimization parameters and the multiple position encoding methods, the small model and the medium model are optimized respectively to obtain a target small model and a target medium model corresponding to each position encoding method.

4. The method according to claim 1, characterized in that: The performing operator abstraction on the target visual model includes: Get the mapping relationship table between different self-attention calculation operations and corresponding operation operators; In response to executing a target self-attention calculation operation for the target visual model, a corresponding operation operator is determined in the mapping relationship table based on the target self-attention calculation operation and executed to perform operator abstraction on the target visual model.

5. The method according to claim 1, characterized in that The implementing the neural network operator on the domestic hardware based on the corresponding relationship includes: In response to inputting a tensor and a window size parameter to the neural network operator, obtaining multi-dimensional data of the tensor; On the domestic hardware, based on the first target operator development method on the tensor acceleration engine provided by the domestic hardware and the multi-dimensional data, operator development, data transfer for a preset number of times and dimension setting of the tensor are performed in sequence to obtain a corresponding output tensor.

6. The method according to claim 5, characterized in that The implementing the neural network operator on the domestic hardware based on the corresponding relationship includes: In response to inputting a tensor, a window size parameter and a feature map parameter into the neural network operator, obtaining multi-dimensional data of the tensor; wherein the feature map parameter includes a feature map height and a feature map width of an original feature map before dividing the window; On the domestic hardware, based on the first target operator development method on the tensor acceleration engine provided by the domestic hardware and the multi-dimensional data, operator development, data transfer for a preset number of times and dimension setting of the tensor are performed in sequence to obtain a corresponding output tensor.

7. The method according to claim 6, characterized in that The implementing the neural network operator on the domestic hardware based on the corresponding relationship includes: In response to inputting a tensor, weights and offsets into the neural network operator, the mean and variance are calculated based on the second target operator development method on the tensor acceleration engine, and normalization and affine transformation are performed to obtain the corresponding output tensor.

8. A visual model operator optimization device for domestic hardware, applied to a visual model, wherein the visual model includes a sliding window self-attention module and a relative position encoding module; characterized in that: The optimization device comprises: An extension module, used for extending a plurality of position encoding modes of the visual model according to the relative position encoding and conditional position encoding of the relative position encoding module of the visual model for the visual model; An optimization module, configured to position-encode the visual model based on the plurality of position encoding methods to obtain a target visual model after structural optimization corresponding to each position encoding method; wherein different target visual models represent different model structures; wherein the target visual model adopts a combination of window self-attention and linear space reduction self-attention; An abstraction module is used to input a preset target image into a target visual model, perform operator abstraction on the target visual model, and obtain a corresponding abstracted target visual model; wherein the abstracted target visual model corresponds to the abstracted neural network operator; An implementation module is used to obtain the correspondence between the neural network operator and the domestic hardware operator provided by the domestic hardware, and implement the neural network operator on the domestic hardware based on the correspondence.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the visual model operator optimization method for domestic hardware as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the visual model operator optimization method for domestic hardware as described in any one of claims 1 to 7.

Images