An ONNX neural network oriented backend deployment conversion method and device

Abstract

The application provides an ONNX neural network rear-end deployment conversion method and device. All operators in an ONNX model are traversed to find Matmul operators and Upsample operators, the Matmul operators are converted into corresponding dense operators or first combination operators or second combination operators in a relayir model, the Upsample operators are converted into corresponding resize operators in the relayir model, and the remaining operators are all directly converted into relayir operators, so that the ONNX model is converted into the relayir model, all operators can be supported by the rear end, the situation that operation cannot normally run is avoided, and the purpose of rapid deployment is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of neural networks, and in particular to an ONNX neural network conversion method and device for backend deployment. BACKGROUND

[0002] When an ONNX (Open Neural Network Exchange) neural network is deployed for a backend, due to limited backend hardware characteristics and corresponding hardware software package support, there are often unsupported operators in the ONNX neural network, which causes inference operation to be unable to normally run. Among these unsupported operators, Matmul operators and Upsample operators are new popular operators in the ONNX neural network in recent times, and the use frequency gradually increases. The non-support of Matmul operators and Upsample operators relative to the backend causes the ONNX neural network to be difficult to quickly deploy to the backend.

[0003] In view of this, overcoming the defects of the prior art is a problem to be solved in the technical field. SUMMARY

[0004] The technical problem to be solved by the present application is how to convert Matmul operators and Upsample operators in an ONNX model into operators supported by a backend.

[0005] The embodiment of the present application adopts the following technical solution:

[0006] In a first aspect, an ONNX neural network conversion method for backend deployment is provided, comprising:

[0007] All operators in an ONNX model are traversed, and the operators include Matmul operators, Upsample operators and regular operators;

[0008] When an operator is a Matmul operator, the operator is converted from a Matmul operator to a corresponding dense operator, a first combination operator or a second combination operator in a relayir model, wherein the first combination operator is a dense and reshape combination operator, and the second combination operator is a reshape and batch_matmul combination operator;

[0009] When an operator is an Upsample operator, the operator is converted from an Upsample operator to a corresponding resize operator in a relayir model;

[0010] When an operator is a regular operator, the operator is converted into a corresponding relayir operator in a relayir model;

[0011] After all the operators in the ONNX model are converted into corresponding operators in the relayir model, the ONNX model is converted into the relayir model.

[0012] Preferably, when the operator is the Matmul operator, the operator is converted from the Matmul operator into a corresponding dense operator, a first combination operator or a second combination operator in the relayir model, specifically including:

[0013] The first input tensor and the second input tensor of the Matmul operator are obtained, and the matrix dimensions of the first input tensor and the matrix dimensions of the second input tensor are obtained;

[0014] According to the matrix dimensions of the first input tensor and the matrix dimensions of the second input tensor, one or more of a dense operator, a reshape operator and a batch_matmul operator are added to the Matmul operator;

[0015] The one or more operators selected from the dense operator, the reshape operator and the batch_matmul operator are converted.

[0016] Preferably, according to the matrix dimensions of the first input tensor and the matrix dimensions of the second input tensor, one or more of a dense operator, a reshape operator and a batch_matmul operator are added to the Matmul operator, specifically including:

[0017] When the first input tensor and the second input tensor are both two-dimensional matrices, the second input tensor is transposed, the dense operator is added, and the first input tensor and the transposed second input tensor are input into the dense operator to obtain a second output tensor;

[0018] The second output tensor is compared with the first output tensor of the Matmul operator, and when the second output tensor is consistent with the first output tensor, it represents that the Matmul operator is converted into a dense operator, and when the second output tensor is inconsistent with the first output tensor, the process is interrupted.

[0019] Preferably, according to the matrix dimensions of the first input tensor and the matrix dimensions of the second input tensor, one or more of a dense operator, a reshape operator and a batch_matmul operator are added to the Matmul operator, further including:

[0020] When the first input tensor is a high-dimensional matrix and the second input tensor is a two-dimensional matrix, the reshape operator is added after the first input tensor to change the shape of the first input tensor;

[0021] The second input tensor is transposed, and the changed shape first input tensor and the transposed second input tensor are input into the dense operator to obtain a third output tensor;

[0022] The shape of the third output tensor is changed, and the changed shape of the third output tensor is compared with the shape of the first output tensor of the Matmul operator. When the changed shape of the third output tensor is consistent with the shape of the first output tensor, it represents that the Matmul operator is converted into the first combined operator. When the changed shape of the third output tensor is inconsistent with the shape of the first output tensor, the process is interrupted.

[0023] Preferably, according to the matrix dimensions of the first input tensor and the matrix dimensions of the second input tensor, one or more of the dense operator, the reshape operator and the batch_matmul operator are added to the Matmul operator, which further comprises:

[0024] When the first input tensor and the second input tensor are both high-dimensional matrices, the reshape operator is added after the first input tensor and the second input tensor to change the shape of the first input tensor and the second input tensor. The batch_matmul operator is added, and the changed shape first input tensor and the changed shape second input tensor are input into the batch_matmul operator to obtain a fourth output tensor;

[0025] The shape of the fourth output tensor is changed, and the changed shape of the fourth output tensor is compared with the shape of the first output tensor of the Matmul operator. When the changed shape of the fourth output tensor is consistent with the shape of the first output tensor, it represents that the Matmul operator is converted into the second combined operator. When the changed shape of the fourth output tensor is inconsistent with the shape of the first output tensor, the process is interrupted.

[0026] Preferably, when the operator is the Upsample operator, the operator is converted from the Upsample operator to the corresponding resize operator in the relayir model, which specifically comprises:

[0027] The third input tensor and the fourth input tensor of the Upsample operator are obtained, and the matrix dimensions of the third input tensor and the scaling factor of the fourth input tensor are obtained;

[0028] interrupting or adding a resize operator to the Upsample operator according to the matrix dimension of the third input tensor.

[0029] Preferably, the interrupting or adding a resize operator to the Upsample operator according to the matrix dimension of the third input tensor specifically comprises:

[0030] when the third input tensor is a five-dimensional matrix, then interrupting execution;

[0031] when the third input tensor is not a five-dimensional matrix, adding the resize operator, inputting the scaling factor of the third input tensor and the fourth input tensor into the resize operator to obtain a sixth output tensor;

[0032] when the sixth output tensor is consistent with the fifth output tensor of the Upsample operator, it represents that the Upsample operator is converted into a resize operator, and when the sixth output tensor is inconsistent with the first output tensor, then interrupting the process.

[0033] Preferably, when the operator is the conventional operator, the operator is converted into a corresponding relayir operator, specifically comprising:

[0034] obtaining the matrix dimension, input tensor shape and parameter information of the input tensor of the conventional operator, corresponding the conventional operator with the operator in relayir through the matrix dimension, input tensor shape and parameter information of the input tensor, and converting the conventional operator into a corresponding relayir operator.

[0035] Preferably, the obtained operators of the relayir model are compiled layer by layer through a GPU compiler or an APU compiler, and are converted into executable binary language.

[0036] In a second aspect, an ONNX neural network deployment conversion device for a backend is provided, comprising at least one processor, and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the ONNX neural network deployment conversion method for a backend as described in the first aspect.

[0037] The embodiment of the present application provides an ONNX neural network facing a backend deployment conversion method and device, all operators in an ONNX model are traversed, a Matmul operator and an Upsample operator are found, the Matmul operator is converted into a corresponding dense operator or a first combination operator or a second combination operator in a relayir model, the Upsample operator is converted into a corresponding resize operator in the relayir model, and the rest of the operators are all directly converted into relayir operators, so that the ONNX model is converted into the relayir model, all operators can be supported by the backend, the situation that operation cannot normally run is avoided, and the purpose of rapid deployment is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0039] Figure 1 It is a method flow chart of an ONNX neural network facing a backend deployment conversion method provided by the embodiment of the present application;

[0040] Figure 2 It is a conversion flow chart when the operator is a Matmul operator in the ONNX neural network facing a backend deployment conversion method provided by the embodiment of the present application;

[0041] Figure 3 It is a conversion flow chart when the dense operator is added in the ONNX neural network facing a backend deployment conversion method provided by the embodiment of the present application;

[0042] Figure 4 It is a conversion flow chart when the reshape operator and the dense operator are added in the ONNX neural network facing a backend deployment conversion method provided by the embodiment of the present application;

[0043] Figure 5 It is a conversion flow chart when the reshape operator and the batch_matmul operator are added in the ONNX neural network facing a backend deployment conversion method provided by the embodiment of the present application;

[0044] Figure 6 It is a conversion flow chart of an Upsample operator in the ONNX neural network facing a backend deployment conversion method provided by the embodiment of the present application;

[0045] Figure 7is a method flow chart of an ONNX neural network facing backend deployment conversion method provided by an embodiment of the present application.

[0046] Figure 8 is a structural schematic diagram of an ONNX neural network facing backend deployment conversion device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the objects, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0048] In the description of the present application, the terms "inner", "outer", "longitudinal", "transverse", "upper", "lower", "top", "bottom" and the like indicate the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present application and do not require the present application to be constructed and operated in a particular orientation, therefore should not be understood as a limitation on the present application.

[0049] In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0050] An ONNX neural network facing backend deployment conversion method is provided by an embodiment of the present application, as shown in the figure, the method flow is: Figure 1

[0051] In step 101, all operators in the ONNX model are traversed, and the operators include Matmul operators, Upsample operators and regular operators.

[0052] Among them, the regular operator is all operators in the ONNX model except the Matmul operator and the Upsample operator.

[0053] In an optional embodiment, before traversing all operators in the ONNX model, the parameter information and the forward propagation information of each operation layer of the ONNX model need to be obtained first, and each operator of the ONNX model is corresponded to the relayir operator to be converted through the parameter information and the forward propagation information, so as to determine the conversion object of different operators.

[0054] In step 102, when the operator is the Matmul operator, the operator is converted from the Matmul operator to the corresponding dense operator, the first combination operator or the second combination operator in the relayir model.

[0055] ​The first combination operator is a dense and reshape combination operator, and the second combination operator is a reshape and batch_matmul combination operator.

[0056] In step 103, when the operator is the Upsample operator, the operator is converted from the Upsample operator to the corresponding resize operator in the relayir model.

[0057] In step 104, when the operator is the conventional operator, the operator is converted into the corresponding relayir operator in the relayir model.

[0058] The dense operator, the first combination operator, the second combination operator, the resize operator, and the relayir operator are all operators in the relayir model, and all operators in the relayir model can be supported by the backend.

[0059] In the embodiment, the backend refers to various AI chips, and the backend deployment is a process of deploying an object to various AI chip operations and facing the bottom chip side.

[0060] In step 105, after all operators in the ONNX model are converted into corresponding operators in the relayir model, the ONNX model is converted into the relayir model.

[0061] When the existing ONNX neural network faces the backend deployment, due to the limited hardware characteristics of the backend and the limited support degree of the corresponding hardware software package, there are often operators that are not supported by the backend in the ONNX neural network, which leads to the failure of the inference operation. The Matmul operator and the Upsample operator in the ONNX model are operators that are not supported by the backend. Since the operators in the relayir model can be supported by the backend, the Matmul operator and the Upsample operator are converted into operators in the relayir model, and the ONNX model is converted into the relayir model, so that all operators can be supported by the backend.

[0062] The embodiment finds the Matmul operator and the Upsample operator in the ONNX model by traversing all operators in the ONNX model. In the past, when a neural network is deployed for a backend, operations and acceleration of traditional convolution (conv), full connection (Dense), Resize, and other operation operators are mainly involved. When MatMul and UpSampling operators are encountered, the chip operation cannot be deployed. However, the Matmul operation process is essentially repeated multiplication operation and addition operation, and the operation process is similar to the Dense operator operation. Therefore, the MatMul operator can be converted into the Dense operator under certain conditions. However, the Matmul operator has multiple cases, that is, different input dimension problems, and there are still some differences in form between the original Dense operator. Therefore, the Matmul operator is converted into the corresponding dense operator or the first combination operator or the second combination operator in the relayir model, the Upsample operator is converted into the corresponding resize operator in the relayir model, and the remaining operators are directly converted into relayir operators, so that the ONNX model is converted into the relayir model, all operators can be supported by the backend, the operation cannot be normally run, and the purpose of rapid deployment is achieved.

[0063] Since the matrix dimensions of the two input tensors in the Matmul operator can be different, the Matmul operator should be converted into an operation operator in a different relayir model during conversion. Therefore, the following design is also involved in the optional embodiment.

[0064] As shown in FIG. 8, it is a conversion flowchart when the operator is a Matmul operator. The flow is as follows: Figure 2

[0065] In step 201, when the operator is the Matmul operator, the first input tensor, the second input tensor, and the first output tensor of the Matmul operator are obtained, and the matrix dimension of the first input tensor and the matrix dimension of the second input tensor are obtained.

[0066] According to the matrix dimension of the first input tensor and the matrix dimension of the second input tensor, one or more of the dense operator, the reshape operator, and the batch_matmul operator are added to the Matmul operator.

[0067] In step 202, when the first input tensor and the second input tensor are both two-dimensional matrices, the dense operator is added to the Matmul operator, and the Matmul operator is converted into the dense operator.

[0068] ​In step 203, when the first input tensor is a high-dimensional matrix and the second input tensor is a two-dimensional matrix, a reshape operator and a dense operator are added to the Matmul operator, and the Matmul operator is converted into a first combined operator.

[0069] In step 204, when the first input tensor and the second input tensor are both high-dimensional matrices, a reshape operator and a batch_matmul operator are added to the Matmul operator, and the Matmul operator is converted into a second combined operator.

[0070] Since the operation mode of the Matmul operator is different when the output tensors of the Matmul operator are different, the shapes of the output are also different, and therefore a corresponding operator needs to be selected for conversion.

[0071] When the number of input tensors of the Matmul operator is not 2, it means that there is a problem with the ONNX model, and in this case the process needs to be interrupted and the ONNX model needs to be checked.

[0072] It should be noted that the first input tensor, the second input tensor, and the first output tensor of the Matmul operator are the input tensors and the output tensor of the Matmul operator in the original ONNX model, and the first input tensor, the second input tensor, and the first output tensor are determined before the conversion process of the present embodiment starts.

[0073] When the first input tensor and the second input tensor of the Matmul operator are obtained, the shapes corresponding to the first input tensor and the second input tensor are also obtained, and in the present embodiment, the first input tensor is set as input_a, the second input tensor is set as input_b, the shape of the first input tensor is set as a_shape, the shape of the second input tensor is set as b_shape, the matrix dimension of the first input tensor is set as a_rank, and the matrix dimension of the second input tensor is set as b_rank, which facilitates subsequent operation.

[0074] When the first input tensor and the second input tensor are both two-dimensional matrices (i.e., a_rank=2 and b_rank=2), as shown in FIG. 2A, the process is as follows: Figure 3

[0075] In step 301, the second input tensor is transposed.

[0076] In an optional embodiment, transposing the second input tensor means transposing the 0th dimension and the 1st dimension of input_b to obtain the transposed second input tensor input_b_transpose.

[0077] ​In step 302, the dense operator is added, and the first input tensor and the transposed second input tensor are input into the dense operator to obtain a second output tensor.

[0078] The first input tensor input_a and the transposed second input tensor input_b_transpose are input into the dense operator to obtain a second output tensor.

[0079] In step 303, it is determined whether the second output tensor is consistent with the first output tensor (i.e., the output tensor of the original Matmul operator). When the second output tensor is consistent with the first output tensor, the process jumps to step 304. When the second output tensor is inconsistent with the first output tensor, the process jumps to step 305.

[0080] In step 304, when the second output tensor is consistent with the first output tensor, it means that the Matmul operator is converted into the dense operator.

[0081] In step 305, when the second output tensor is inconsistent with the first output tensor, the process is interrupted.

[0082] When the second output tensor is consistent with the first output tensor, it means that the Matmul operator is converted into the dense operator. When the second output tensor is inconsistent with the first output tensor, it means that there is a problem with the ONNX model, and the process needs to be interrupted for checking.

[0083] When the first input tensor is a high-dimensional matrix and the second input tensor is a two-dimensional matrix (i.e., a_rank>2 and b_rank=2), as shown in FIG. 4A, the process is as follows: Figure 4

[0084] In step 401, the reshape operator is added after the first input tensor to change the shape of the first input tensor and transpose the second input tensor.

[0085] Since the first input tensor is a high-dimensional matrix, and the dense operator has certain requirements for the shape of the input, which is different from the original Matmul operator, the shape of the high-dimensional matrix needs to be changed accordingly when converting the operator.

[0086] In an optional embodiment, the reshape operator is added after the first input tensor to change the shape of the first input tensor, which is as follows:

[0087] First, the new shape of the first input tensor input_a is calculated:

[0088] ​new_shape_a = [-1] + a_shape[a_rank - 1 : a_rank].

[0089] A reshape operator is added after input_a, and shape changing is performed according to the new shape new_shape_a, and input_a_reshape (the first input tensor after shape changing) is obtained.

[0090] In an optional embodiment, two dimensions of the second input tensor input_b are exchanged, and input_b_transpose is obtained by transposition.

[0091] In step 402, the dense operator is added, and the first input tensor after shape changing and the second input tensor after transposition are input into the dense operator to obtain a third output tensor.

[0092] The first input tensor after shape changing input_a_reshape and the second input tensor after transposition input_b_transpose are input into the dense operator to obtain a third output tensor.

[0093] In step 403, the shape of the third output tensor is changed.

[0094] The method for changing the shape of the third output tensor is as follows:

[0095] First, the shape of the third output tensor is calculated through the shapes of the two input quantities input_a_reshape and input_b_transpose input into the dense operator, as follows:

[0096] a_reshape = a_shape;

[0097] out_batch_a = a_shape[0, a_rank - 2], out_batch_b = b_shape[0, b_rank - 2];

[0098] Since a_rank > b_rank, out_batch = out_batch_a.

[0099] The shape final_shape of the third output tensor is calculated, and final_shape = out_batch + a_shape[a_rank - 2, a_rank - 1] + b_shape[b_rank - 1, b_rank].

[0100] A reshape operator is added after the final_shape, so as to change the shape of the third output tensor and make it consistent with the shape of the first output tensor (i.e. the output tensor of the original Matmul operator).

[0101] In step 404, it is determined whether the shape of the third output tensor after being changed is consistent with the shape of the first output tensor (i.e. the output tensor of the original Matmul operator). If yes, the process jumps to step 405; if no, the process jumps to step 406.

[0102] In step 405, when the shape of the third output tensor after being changed is consistent with the shape of the first output tensor, it means that the Matmul operator is converted into the first combination operator.

[0103] In step 406, when the shape of the third output tensor after being changed is not consistent with the shape of the first output tensor, it means that there is a problem in the ONNX model, and the process is interrupted.

[0104] When the first input tensor and the second input tensor are both high-dimensional matrices (i.e. a_rank>2 and b_rank>2), as shown in FIG. 5, the process is as follows: Figure 5

[0105] In step 501, the reshape operator is added after the first input tensor and the second input tensor, so as to change the shape of the first input tensor and the second input tensor.

[0106] In an optional embodiment, the reshape operator is added after the first input tensor and the second input tensor, so as to change the shape of the first input tensor and the second input tensor, which is as follows:

[0107] First, the new shape of the first input tensor input_a and the second input tensor input_b is calculated:

[0108] new_shape_a = [-1] + a_shape[a_rank-2, a_rank];

[0109] new_shape_b = [-1] + b_shape[b_rank-2, b_rank];

[0110] The reshape operator is added after input_a, and the shape is changed according to the new shape new_shape_a, so as to obtain input_a_reshape.

[0111] ​A reshape operator is added after input_b, and shape changing is performed according to the new shape new_shape_b, so that input_b_reshape is obtained.

[0112] In step 502, the batch_matmul operator is added, and the first input tensor after shape changing and the second input tensor after shape changing are input into the batch_matmul operator to obtain the fourth output tensor.

[0113] The first input tensor after shape changing input_a_reshape and the second input tensor after shape changing input_b_reshape are input into the batch_matmul operator to obtain the fourth output tensor.

[0114] In step 503, the fourth output tensor is changed in shape.

[0115] The method for changing the shape of the fourth output tensor is as follows:

[0116] First, the shape of the fourth output tensor is calculated through the shapes of the two input quantities input_a_reshape and input_b_reshape input into the batch_matmul operator, as follows:

[0117] a_reshape=a_shape;

[0118] b_reshape=b_shape;

[0119] out_batch_a=a_shape[0,a_rank-2],out_batch_b=b_shape[0,b_rank-2];

[0120] a_rank=b_rank, then traverse the elements in out_batch_a and out_batch_b, and the i-th element out_batch[i] in out_batch takes the larger value of out_batch_a[i] and out_batch_b[i].

[0121] The shape final_shape of the fourth output tensor is calculated, final_shape=out_batch+a_shape[a_rank–2,a_rank-1]+b_shape[b_rank–1,b_rank].

[0122] A reshape operator is added after the final_shape, so that the fourth output tensor is changed in shape and consistent with the shape of the first output tensor.

[0123] In step 504, it is determined whether the shape of the fourth output tensor after being changed is consistent with the shape of the first output tensor (i.e., the original output tensor of the Matmul operator). If yes, the process jumps to step 505. If no, the process jumps to step 506.

[0124] In step 505, when the shape of the fourth output tensor after being changed is consistent with the shape of the first output tensor, it means that the Matmul operator is converted into the second combined operator.

[0125] In step 506, when the shape of the fourth output tensor after being changed is inconsistent with the shape of the first output tensor, it means that there is a problem in the ONNX model, and the process is interrupted.

[0126] In this embodiment, when the operator is the Upsample operator, the operator is converted from the Upsample operator to the resize operator corresponding to relayir, as shown in the following flow: Figure 6

[0127] In step 601, the third input tensor, the fourth input tensor and the fifth output tensor of the Upsample operator are obtained, and the matrix dimension of the third input tensor and the scaling factor of the fourth input tensor are obtained.

[0128] The process is interrupted according to the matrix dimension of the third input tensor, or a resize operator is added to the Upsample operator.

[0129] In this embodiment, the third input tensor is set as input_1, the shape of the third input tensor is set as input_1_shape, the matrix dimension of the third input tensor is set as 1_rank, and the fourth input tensor is set as input_2.

[0130] In step 602, according to the matrix dimension 1_rank of the third input tensor, it is determined whether the third input tensor is a five-dimensional matrix. If yes, the process jumps to step 603. If no, the process jumps to step 604.

[0131] In step 603, when the third input tensor is a five-dimensional matrix, it means that there is a problem in the ONNX model, and the process needs to be interrupted for checking the ONNX model.

[0132] In step 604, when the third input tensor is not a five-dimensional matrix, the resize operator is added, and the scaling factor of the third input tensor and the fourth input tensor is input into the resize operator to obtain a sixth output tensor.

[0133] ​It should be noted that the scaling factor of the third input tensor and the fourth input tensor is input to the resize operator, that is, the third input tensor is scaled according to the scaling factor of the fourth input tensor.

[0134] The scaling factor of the fourth input tensor is obtained by analyzing the fourth input tensor input_2, and the scaling factor is scale_h=input_2[2] and scale_w=input_2[3], where scale_h is the scaling factor of the four-dimensional data H (height), and scale_w is the scaling factor of the four-dimensional data W (width).

[0135] The resize operator is added, and the third input tensor input_1 and the scaling factors scale_w and scale_h of the fourth input tensor are input to the resize operator, so that the third input tensor is scaled according to the scaling factor of the fourth input tensor, where the layout is NCHW, and the method attribute is determined according to the operation mode mode of the Unsample operator. If the mode is "nearest", the resize operation method method of relayir corresponding to the resize operation method method of relayir is "nearest_neighbor"; if the mode is "linear", the resize operation method method of relayir corresponding to the resize operation method method of relayir is "bilinear"; align_corners is set to False; the sixth output tensor is obtained, and the sixth output tensor is the scaled third input tensor, and the sixth output tensor is compared with the fifth output tensor.

[0136] In step 605, it is judged whether the sixth output tensor is consistent with the fifth output tensor, if yes, jump to step 606, if not, jump to step 607.

[0137] In step 606, when the sixth output tensor is consistent with the fifth output tensor, it means that the Upsample operator is converted to the resize operator.

[0138] In step 607, when the sixth output tensor is consistent with the fifth output tensor of the Upsample operator, the process is interrupted.

[0139] In addition to the Matmul operator and the Upsample operator, other conventional operators of the ONNX model also need to be converted into corresponding relayir operators, so the present embodiment also relates to the following design:

[0140] The matrix dimension, input tensor shape and parameter information of the input tensor of the conventional operator are acquired, the conventional operator is corresponded to the operator in relayir through the matrix dimension, input tensor shape and parameter information of the input tensor, and the conventional operator is converted into the corresponding relayir operator.

[0141] Since the corresponding conversion relationship between the operators in the ONNX model and the operators in the relayir model has been determined before all the operators in the ONNX model are traversed, when the parameter information of the conventional operator is acquired, the relayir model operator that needs to be converted by the conventional operator can be found according to the parameter information. Since the conventional operator itself is supported by the backend, the conventional operator can be directly converted into the relayir model operator.

[0142] After the ONNX model is completely converted into the relayir model, the obtained relayir model operators are compiled layer by layer through a GPU compiler or an APU compiler, and are converted into executable binary language, that is, can be supported by the backend, so that the purpose of fast deployment of the ONNX neural network to the backend is achieved.

[0143] In combination with FIGS Figure 7 The embodiment provides a flow of an ONNX neural network deployment conversion method for a backend, specifically:

[0144] 1. Defining the relayir operation operator corresponding to the different operation operators of the ONNX model.

[0145] 2. Converting the ONNX model into relayir, and the specific conversion steps are as follows:

[0146] @1. Acquiring the parameter information and forward propagation information of each operation layer of the ONNX neural network model.

[0147] @2. Traversing all the nodes (nodes) in the ONNX neural network model and converting one by one:

[0148] @a. Judging whether the operator is a Matmul operator, if it is a Matmul operator, step @b is executed, if it is not a Matmul operator, step @j is executed.

[0149] @b. Acquiring the input tensor of the Matmul operator, judging whether the number of input tensors is 2, if not, interrupting; if equal to 2, setting the two input tensors of the Matmul operator as input_a and input_b, and executing step @c.

[0150] @C, get the shape a_shape and b_shape of input_a and input_b respectively, and the dimension number a_rank, b_rank of input_a and input_b.

[0151] @D, judge whether the matrix dimension of the two input tensors is two-dimensional matrix, if yes, execute step @d1, if not, execute step @e:

[0152] @D1, transpose the 0th and 1st dimensions of input_b to obtain input_b_transpose.

[0153] @D2, add a dense operator, where the two inputs of the dense operator are input_a and input_b_transpose respectively, and output output, compare output with the output tensor of the original Matmul operator, if they are consistent, it means that the Matmul operator is converted to the dense operator, and return to step 2, if not, interrupt the process.

[0154] @E, judge whether the matrix dimension of the two input tensors is one high-dimensional matrix and one two-dimensional matrix, if yes, execute step @e1, if not, execute @f:

[0155] @E1, calculate the new shape new_shape of input_a = [-1] + a_shape[a_rank–1:a_rank], add a reshape operator after input_a, reshape input_a to the shape of new_shape, and set the output of the operator as input_a_reshape.

[0156] @E2, transpose input_b to obtain input_b_transpose.

[0157] @E3, add a dense operator, where the input of the dense operator is input_a_reshape and input_b_transpose, and the output is output, execute step @g.

[0158] @F, both input tensors are high-dimensional matrices, the conversion steps are as follows:

[0159] @f1, compute new shape of input_a as new_shape_a = [-1] + a_shape[a_rank-2, a_rank] and new shape of input_b as new_shape_b = [-1] + b_shape[b_rank-2, b_rank].

[0160] @f2, add reshape operator after input_a to convert input_a to input_a_reshape with shape new_shape_a; add reshape operator after input_b to convert input_b to input_b_reshape with shape new_shape_b.

[0161] @f3, add batch_matmul operator with input of input_a_reshape and input_b_reshape and output of output; go to step g.

[0162] @g, compute out_batch_a = a_shape[0, a_rank-2] and out_batch_b = b_shape[0, b_rank-2].

[0163] @g1, if a_rank > b_rank, then out_batch = out_batch_a.

[0164] @g2, if a_rank < b_rank, then out_batch = out_batch_b.

[0165] @g3, if a_rank = b_rank, then iterate through elements in out_batch_a and out_batch_b, and the i-th element in out_batch, out_batch[i], takes the larger value between out_batch_a[i] and out_batch_b[i].

[0166] @h, compute the final output of the added combined operator as final_shape = out_batch + a_shape[a_rank-2, a_rank-1] + b_shape[b_rank-1, b_rank].

[0167] @i. Add a reshape operator to change the shape of final_shape to be the same as the shape of the output tensor of the Matmul operator. This means that the Matmul operator is converted to the corresponding relayir operator, and the process returns to step 2.

[0168] @j. Determine whether the operator is an Upsample operator. If it is, convert the Upsample operator to the corresponding resize operator in relayir, and if it is not, proceed to step k.

[0169] @j1. Determine whether the Upsample operator has two inputs. If it does, proceed to step j2; if not, interrupt the process. input_1 is one of the input tensors, and input_2 is the scaling factor for the other input tensor.

[0170] @j2. Analyze the operation mode of the Upsample operator. If the mode is "nearest", the corresponding resize operation method in relayir is "nearest_neighbor". If the mode is "linear", the corresponding resize operation method in relayir is "bilinear".

[0171] @j3. Obtain the shape of the tensor input_1, which is input_1_shape. Based on input_1_shape, determine whether input_1 is a 5-dimensional tensor. If it is, proceed to step j4; if not, proceed to steps j5 and j6.

[0172] @j4. Interrupt the process.

[0173] @j6. Analyze the scaling values in input_2, where scale_h = input_2[2] and scale_w = input_2[3]. scale_h and scale_w correspond to the scaling factors for the height (H) and width (W) of the four-dimensional data, respectively.

[0174] @j7. Add the corresponding Resize operator in relayir, with input_1, scale_w, and scale_h as inputs, layout as NCHW, method attribute as the value of method in j2, and align_corners set to False. Return to step 2.

[0175] @k, the node is a regular operator, obtaining the input tensor shape, the output tensor shape of the node, and the parameter information corresponding to the node. @l, based on the operation operator defined in steps 1 and 2, the node is converted into the corresponding relayir operator.

[0176] @4, based on relayir, the relayir model operator is compiled layer by layer with a GPU compiler or other APU compiler, and converted into executable binary language.

[0177] As Figure 8 shown is an ONNX neural network for backend deployment conversion device schematic diagram of an embodiment of the application. The ONNX neural network for backend deployment conversion device of the embodiment includes one or more processors 81 and a memory 82. Among them, Figure 8 In an embodiment, the processor 81 is taken as an example.

[0178] The processor 81 and the memory 82 can be connected through a bus or other means, Figure 8 In an embodiment, the connection through the bus is taken as an example.

[0179] The memory 82, as a kind of non-volatile computer readable storage medium, can be used to store non-volatile software programs and non-volatile computer executable programs, such as the ONNX neural network for backend deployment conversion method in the above embodiment. The processor 81 executes the non-volatile software programs and instructions stored in the memory 82, thereby executing the ONNX neural network for backend deployment conversion method.

[0180] The memory 82 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 82 can optionally include a memory remotely arranged with respect to the processor 81, which can be connected to the processor 81 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0181] The program instructions / modules are stored in the memory 82, and when executed by the one or more processors 81, the ONNX neural network for backend deployment conversion method in the above embodiment is executed, for example, the above-described Figures 1 to 7 each step shown.

[0182] The above only describes the preferred embodiments of the application and does not limit the application. Any modification, equivalent replacement and improvement made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. An ONNX neural network backend deployment conversion method, characterized in that, The method comprises the following steps: traversing all operators in the ONNX model, wherein the operators comprise a Matmul operator, an Upsample operator and a regular operator; when the operator is the Matmul operator, converting the operator from the Matmul operator to a corresponding dense operator, a first combination operator or a second combination operator in the relayir model, wherein the first combination operator is a dense and reshape combination operator, and the second combination operator is a reshape and batch_matmul combination operator; obtaining a first input tensor and a second input tensor of the Matmul operator, and obtaining matrix dimensions of the first input tensor and matrix dimensions of the second input tensor; adding one or more of a dense operator, a reshape operator and a batch_matmul operator to the Matmul operator according to the matrix dimensions of the first input tensor and the matrix dimensions of the second input tensor; and performing operator conversion on the one or more operators selected from the dense operator, the reshape operator and the batch_matmul operator; when the first input tensor and the second input tensor are both two-dimensional matrices, transposing the second input tensor; adding the dense operator, and inputting the first input tensor and the transposed second input tensor into the dense operator to obtain a second output tensor; comparing the second output tensor with a first output tensor of the Matmul operator, and when the second output tensor is consistent with the first output tensor, it is determined that the Matmul operator is converted into the dense operator, and when the second output tensor is inconsistent with the first output tensor, the process is interrupted; when the operator is the Upsample operator, converting the operator from the Upsample operator to a corresponding resize operator in the relayir model; when the operator is the regular operator, converting the operator into a corresponding relayir operator in the relayir model; after all operators in the ONNX model are converted into corresponding operators in the relayir model, converting the ONNX model into the relayir model.

2. The ONNX neural network backend deployment conversion method of claim 1, wherein, The step of adding one or more of the dense operator, the reshape operator and the batch_matmul operator to the Matmul operator according to the matrix dimensions of the first input tensor and the matrix dimensions of the second input tensor further comprises the following steps: when the first input tensor is a high-dimensional matrix and the second input tensor is a two-dimensional matrix, adding the reshape operator after the first input tensor to change the shape of the first input tensor; transposing the second input tensor; adding the dense operator, and inputting the first input tensor with the changed shape and the transposed second input tensor into the dense operator to obtain a third output tensor; and The shape of the third output tensor is changed, and compared with the shape of the first output tensor of the Matmul operator. When the shape of the third output tensor after the change is consistent with the shape of the first output tensor, it means that the Matmul operator is converted into a first combined operator. When the shape of the third output tensor after the change is inconsistent with the shape of the first output tensor, the process is interrupted.

3. The ONNX neural network backend deployment conversion method of claim 1, wherein, The adding of one or more of the dense operator, the reshape operator and the batch_matmul operator to the Matmul operator according to the matrix dimension of the first input tensor and the matrix dimension of the second input tensor further comprises: When the first input tensor and the second input tensor are both high-dimensional matrices, the reshape operator is added after the first input tensor and after the second input tensor to change the shape of the first input tensor and the second input tensor; the batch_matmul operator is added to input the first input tensor after the shape change and the second input tensor after the shape change into the batch_matmul operator to obtain a fourth output tensor; The shape of the fourth output tensor is changed, and compared with the shape of the first output tensor of the Matmul operator. When the shape of the fourth output tensor after the change is consistent with the shape of the first output tensor, it means that the Matmul operator is converted into a second combined operator. When the shape of the fourth output tensor after the change is inconsistent with the shape of the first output tensor, the process is interrupted.

4. The ONNX neural network backend deployment conversion method of claim 1, wherein, The converting of the operator from the Upsample operator to a corresponding resize operator in the relayir model when the operator is the Upsample operator comprises: The third input tensor and the fourth input tensor of the Upsample operator are obtained, and the matrix dimension of the third input tensor and the scaling factor of the fourth input tensor are obtained; The matrix dimension of the third input tensor is used to interrupt or add a resize operator to the Upsample operator.

5. The ONNX neural network backend deployment conversion method of claim 4, wherein, The matrix dimension of the third input tensor is used to interrupt or add a resize operator to the Upsample operator, which comprises: When the third input tensor is a five-dimensional matrix, the execution is interrupted; When the third input tensor is not a five-dimensional matrix, the resize operator is added to input the third input tensor and the scaling factor of the fourth input tensor into the resize operator to obtain a sixth output tensor; When the sixth output tensor is consistent with the fifth output tensor of the Upsample operator, it means that the Upsample operator is converted into a resize operator. When the sixth output tensor is inconsistent with the first output tensor, the process is interrupted.

6. The ONNX neural network backend deployment conversion method of claim 1, wherein, The converting of the operator into a corresponding relayir operator in the relayir model when the operator is the conventional operator comprises: The matrix dimension, input tensor shape and parameter information of the input tensor of the conventional operator are acquired, the conventional operator is corresponded to the operator in the relayir through the matrix dimension, input tensor shape and parameter information of the input tensor, and the conventional operator is converted into the corresponding relayir operator.

7. The ONNX neural network backend deployment conversion method of claim 1, wherein, The obtained operators of the relayir model are compiled layer by layer through a GPU compiler or an APU compiler, and are converted into executable binary language.

8. An ONNX neural network backend deployment conversion device, comprising: The method comprises at least one processor, and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the ONNX neural network backend deployment conversion method in any one of claims 1-7.

Images