Model processing method, storage medium, and electronic device

Abstract

The present application relates to the technical field of artificial intelligence, in particular to a model processing method, a storage medium and an electronic device. In the method, the electronic device can remove the transpose operator introduced by the application program deploying the neural network in the process of converting the neural network model in the first data format into the neural network model in the second data format, and adjust the arrangement order of the data in the reference identifier of the parameters of each format-related operator in the neural network model to ensure that the data referenced by the parameters in the converted neural network model in the second data format is unchanged, thereby ensuring the correctness of the processing result of each operator on the input data. In this way, since no additional transpose operator is introduced, the running efficiency of the converted neural network model in the second data format is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a model processing method, storage medium, and electronic device. Background Technology

[0002] With the development of artificial intelligence technology, the application of neural network models is becoming increasingly widespread, and the number of hardware and software environments supporting the operation of neural network models is also increasing. Because different operating environments have different requirements for the data format (layout) (e.g., the order of the dimensions of tensors) in neural network models, it is necessary to convert the data format of the neural network model when deploying a neural network model with one data format (hereinafter referred to as the source format) to an operating environment that supports another data format (hereinafter referred to as the destination format).

[0003] Typically, when electronic devices use applications that deploy neural network models to convert a source-format neural network model to a target-format neural network model, they add an operator to convert the target format to the source format (e.g., a transpose operator) before each operator in the source-format neural network model, and another operator to convert the source format to the target format (e.g., a transpose operator) after each operator. This results in a large number of transpose operators in the converted target-format neural network model, reducing its running speed. Summary of the Invention

[0004] In view of the above, embodiments of this application provide a model processing method, storage medium, and electronic device, which are beneficial to improving the running speed of neural network models after data format conversion.

[0005] In a first aspect, embodiments of this application provide a model processing method applied to an electronic device. The method includes: obtaining a first model in a second data format to be deployed to a runtime environment of a first data format, wherein the first model includes multiple operators; adjusting the reference identifiers of parameters in at least some of the multiple operators from a first identifier to a second identifier to obtain the first model in the first data format, wherein the data of a tensor with a second data format referenced by the parameter using the first identifier is the same as the data content of the tensor with a first data format referenced by the parameter using the second identifier.

[0006] Thus, in the process of converting the first model in the second data format to the first model in the first data format, no additional transpose operator is introduced, which helps to improve the running speed of the first model in the first data format in the first data format runtime environment.

[0007] In one possible implementation of the first aspect above, adjusting the reference identifier of the parameter in at least some of the multiple operators from a first identifier to a second identifier to obtain a first model with a first data format includes: determining a first operator among the multiple operators, wherein the processing result of the first operator on the input data is associated with the data format supported by the operating environment of the first model; and adjusting the reference identifier of the parameter in the first operator from the first identifier to the second identifier based on the first data format and the second data format.

[0008] In this way, the electronic device does not need to adjust the reference identifiers of parameters in operators other than the first operator, which helps to improve the conversion efficiency.

[0009] In one possible implementation of the first aspect above, determining the first operator among a plurality of operators includes: obtaining an intermediate model obtained by a first model of a first application converting a second data format, wherein the intermediate model is obtained by adding an adjacent first transpose operator before each first operator in the first model of the second data format and adding an adjacent second transpose operator before each first operator, the first transpose operator being used to convert the first data format to the second data format, and the second transpose operator being used to convert the second data format to the first data format; and determining the operator in the intermediate model that has an adjacent first transpose operator before the operator and an adjacent second transpose operator after the operator as the first operator.

[0010] In other words, the electronic device can obtain an intermediate model (e.g., neural network model 20 below) obtained by a first application (e.g., an application for deploying a neural network model) converting a first model into a second data format, and then perform a conversion based on the intermediate model.

[0011] In one possible implementation of the first aspect above, adjusting the reference identifier of the parameter in the first operator from the first identifier to the second identifier based on the first data format and the second data format includes: removing the first transpose operator and the second transpose operator in the intermediate model, and adjusting the reference identifier of the parameter in the first operator from the first identifier to the second identifier based on the first data format and the second data format.

[0012] In one possible implementation of the first aspect above, the tensor corresponding to the first data format and the second data format includes N dimensions, the first identifier includes N data, where N is an integer greater than 1; and, based on the first data format and the second data format, adjusting the reference identifier of the parameter in the first operator from the first identifier to the second identifier includes: taking the Kth data among the N data included in the first identifier as the Mth data in the second identifier, where K is an integer less than or equal to N, the Kth dimension in the first data format is the first dimension and the first dimension is the Mth dimension in the second data format.

[0013] In one possible implementation of the first aspect above, N is 4.

[0014] In other words, the tensor dimension in the first neural network model is 4.

[0015] In one possible implementation of the first aspect above, the first data format and / or the second data format is any one of the following data formats: NCHW format, NHWC format, NHCW format, NCWH format, NWCH format, NWHC format, CNHW format, CHWN format, CHNW format, CNWH format, CWNH format, CWHN format, HCNW format, HNWC format, HNCW format, HCWN format, HWCN format, HWNC format, WCHN format, WHNC format, WHCN format, WCNH format, WNCH format, and WNHC format.

[0016] In one possible implementation of the first aspect described above, the first model is a neural network model.

[0017] Secondly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to implement the method provided in the first aspect and any possible implementation of the first aspect.

[0018] Thirdly, embodiments of this application provide an electronic device, the electronic device comprising: a memory for storing instructions executed by one or more processors of the electronic device; and a processor, one of the processors of the electronic device, for executing the instructions stored in the memory to implement the methods provided in the first aspect and any possible implementation of the first aspect.

[0019] Fourthly, embodiments of this application provide a program product that, when run on an electronic device, enables the electronic device to implement the methods provided in the first aspect and any possible implementation of the first aspect. Attached Figure Description

[0020] Figure 1 According to some embodiments of the present invention, a schematic diagram of a 4-dimensional tensor in NHWC format with N=1 is shown;

[0021] Figure 2 According to some embodiments of the present invention, a schematic diagram of converting neural network model 10 into neural network model 20 is shown;

[0022] Figure 3 According to some embodiments of the present invention, a schematic diagram of converting neural network model 20 into neural network model 30 is shown;

[0023] Figure 4 According to some embodiments of the present invention, a schematic diagram of converting neural network model 20 into neural network model 40 is shown;

[0024] Figure 5 According to some embodiments of the present invention, a schematic flowchart of a model processing method is shown;

[0025] Figure 6A According to some embodiments of the present invention, a schematic diagram is shown of the data arrangement order in the reference identifier of the parameter Q1(u,x,y,z) in the adjustment operator 12;

[0026] Figure 6B According to some embodiments of the present invention, a schematic diagram is shown of the data arrangement order in the reference identifier of the parameter Q2(u,x,y,z) in the operator 13.

[0027] Figure 6C According to some embodiments of the present invention, a schematic diagram is shown of the data arrangement order in the reference identifier of parameter Q3(u,x,y,z) in the adjustment operator 15;

[0028] Figure 7 According to some embodiments of this application, a schematic diagram of a process for deleting the transpose operator introduced during the transformation of the first model is shown;

[0029] Figure 8 According to some embodiments of the present invention, a flowchart of another model processing method is shown;

[0030] Figure 9 According to some embodiments of the present invention, a structural diagram of an electronic device 100 is shown. Detailed Implementation

[0031] The illustrative embodiments of the present invention include, but are not limited to, a model processing method, a storage medium, and an electronic device.

[0032] For the purposes of this description, we will first introduce the terminology used in this application.

[0033] (1) Data format (layout)

[0034] Data format refers to the order in which the dimensions of a tensor are arranged in a neural network model.

[0035] For example, a tensor P of dimension n can be denoted as [p1, p2, p3, ..., p... n ], where p1, p2, ..., p n Let p1, p2, ..., p represent the data of one dimension of tensor P. nThe order in which the data is arranged determines the data format.

[0036] For example, in neural networks, data can be represented using a 4-dimensional tensor. The four dimensions of the tensor can be called batch (N), channel (C), height (H), and width (W), respectively. The order of N, C, H, and W determines the data format of the neural network model. Examples include NCHW, NHWC, NHCW, NCWH, NWCH, NWHC, CNHW, CHWN, CHNW, CNWH, CWNH, CWHN, HCNW, HNWC, HNCW, HCWN, HWCN, HWNC, WCHN, WHNC, WHCN, WCNH, WNCH, and WNHC formats.

[0037] It should be understood that a 4D tensor can be used to represent data of any form, including but not limited to images, text, audio, and feature data. When a 4D tensor is used to represent an image, N dimension is used to represent different images, C dimension is used to represent different color channels of the image, H dimension is used to represent the rows of pixels in the image, and W dimension is used to represent the columns of pixels in the image. For example, corresponding to N=1, C=2, H=3, W=4, it is used to represent the value of the pixel in the third row and fourth column of the first image in the second color channel (e.g., the G channel in an image using the RGB color standard, or the M channel in an image using the CMYK color standard).

[0038] (2) Reference mark

[0039] A reference identifier is used to indicate one or more data points within a tensor. The reference identifier for a tensor typically includes the position of the referenced data within each dimension of the tensor. For an n-dimensional tensor P, this can be represented by parameters P(m1, m2, m3, ..., m...). n The tensor is used to reference data. Where m1, m2, m3, ..., m n These can be called reference identifiers, used to indicate the position of the referenced data in the i-th dimension (i = 1, 2, ..., n), and the actual dimension corresponding to the i-th reference identifier is determined by the data format of tensor P. For example, suppose the data format of tensor P is [p1, p2, p3, ..., p...]. n If m2 indicates the data at position m2 in dimension p2; assuming the data format of tensor P is [p1, p2, ..., p2], then m2 indicates the data at position m2 in dimension p2. n p2, p3, ..., p n-1 If m2 indicates p, then p is the value indicated by m2. n The position in the dimension is m2 data.

[0040] In other words, the same reference identifier refers to different data depending on the data format.

[0041] For example, for a neural network model using NCHW data format, the parameter P(u, x, y, z) is used to indicate the data in tensor P with batch number u, channel number x, height number y, and width number z; while for a neural network model using NHWC data, the parameter P(u, x, y, z) is used to indicate the data in tensor P with batch number u, height number x, width number y, and channel number z.

[0042] For example, assuming that the values ​​of N, C, H, and W in tensor P are all in the range of [1, 5], then each batch of data in tensor P corresponds to 5 × 5 × 5 = 125 data points.

[0043] refer to Figure 1 The 25 data points in the first channel of the first batch are a matrix of size 5 rows × 5 columns, with a1 to a25 from left to right (W takes 1 to 5 respectively) and from top to bottom (H takes 1 to 5 respectively).

[0044] The 25 data points in the second channel of the first batch are b1 to b25 from left to right and from top to bottom;

[0045] The 25 data points in the third channel of the first batch are c1 to c25 from left to right and from top to bottom, respectively.

[0046] The 25 data points in the fourth channel of the first batch are d1 to d25 from left to right and from top to bottom;

[0047] The 25 data points in the 5th channel of the first batch are e1 to e25 from left to right and from top to bottom.

[0048] Corresponding to Figure 1 As shown, if the NCHW data format is used, the value corresponding to parameter P(1, 2, 3, 4) is b14; if the NHWC data format is used, the value corresponding to parameter P(1, 2, 3, 4) is d8.

[0049] (3) Transpose operator

[0050] A transpose operator is an operator used to convert data from one data format to another. Examples include the transpose operator.

[0051] The technical solution of the present invention will now be described in conjunction with the accompanying drawings.

[0052] It should be understood that the technical solutions provided in the embodiments of this application can be applied to neural network models with data formats of arbitrary dimensions and formats. For example, the following describes the technical solutions of this application by converting a neural network model in NCHW format to a neural network model in NHWC format.

[0053] It should be understood that the neural network model applicable to the embodiments of this application can be any form or function of neural network model, such as neural network models used for image classification, object detection, speech recognition, natural language processing, recommendation systems, autonomous driving, etc.

[0054] Typically, electronic devices utilize applications that deploy neural network models (such as TensorFlow). TM PyTorch TM MXNet TM In the process of converting a source-format neural network model to a target-format neural network model, an operator is added before each operator in the source-format neural network model to convert the target format to the source format (e.g., a transpose operator), and another operator is added after each operator to convert the source format to the target format (e.g., a transpose operator). As a result, the converted target-format neural network model contains a large number of transpose operators, which reduces the running speed of the neural network model.

[0055] For example, Figure 2 According to some embodiments of this application, a schematic diagram is shown of converting a neural network model 10 with data format NCHW into a neural network model 20 with data format NHWC.

[0056] like Figure 2 As shown, the data format of the neural network model 10 is NCHW, and the neural network model 10 includes operators 11, 12, 13, 14, and 15. The processing results of operators 12, 13, and 15 on the input data are related to the data format of the neural network model 10 (that is, these operators produce different output data for input data with different data formats but the same content), while the processing results of operators 11 and 14 on the input data are independent of the data format of the neural network model (that is, these operators produce the same output data for input data with different data formats but the same content).

[0057] For ease of description, operators that process input data with different formats but the same content and produce different output data are called format-dependent operators; operators that process input data with different formats but the same content and produce the same output data are called format-independent operators.

[0058] Since the format-related operators in neural network model 10 are designed to process NCHW format data, when an electronic device uses an application that deploys the neural network model to convert neural network model 10 into neural network model 20 with NHWC format data, it can add a transpose operator (hereinafter referred to as the NHWC-NCHW operator) before each format-related operator to convert the data format from NHWC (as the destination format) to NCHW (as the source format), and add a transpose operator (hereinafter referred to as the NCHW-NHWC operator) after each format-related operator. In this way, since the actual data input to the format-related operators is still in NCHW format, it will not affect the results of each format-related operator. However, for each format-related operator included in neural network model 10, two transpose operators need to be added during the conversion process, which reduces the running speed of the converted neural network model 20.

[0059] For example, refer to Figure 2 Electronic devices can utilize an application that deploys a neural network model to add an NHWC-NCHW operator 12A before operator 12, an NCHW-NHWC operator 12B after operator 12, an NHWC-NCHW operator 13A before operator 13, an NCHW-NHWC operator 13B after operator 13, an NHWC-NCHW operator 15A before operator 15, and an NCHW-NHWC operator 15B after operator 15, resulting in a neural network model 20. Thus, neural network model 20, compared to neural network model 10, has six more transpose operators. Electronic devices deploying neural network model 20 will run six more transpose operators, affecting the running speed of the converted neural network model 20.

[0060] In some embodiments, after an electronic device converts a neural network model 10 into a neural network model 20 using an application that deploys the neural network model, it can remove adjacent transpose operators with opposite transformation relationships from the neural network model 20 to improve the running speed of the neural network model 20. For example, refer to Figure 3For the aforementioned neural network model 20, the NCHW-NHWC operator 12B and the NHWC-NCHW operator 13A are adjacent (there are no other operators between these two operators), and their transformation relationships are opposite (NCHW-NHWC operator 12B is used to convert the NCHW format to the NHWC format, and NHWC-NCHW operator 13A is used to convert the NHWC format to the NCHW format). Removing the NCHW-NHWC operator 12B and the NHWC-NCHW operator 13A will not affect the output results between operators 12 and 13. Electronic devices can remove the NCHW-NHWC operator 12B and the NHWC-NCHW operator 13A from the neural network model 20 to obtain the neural network model 30. Compared to the neural network model 20, the neural network model 30 reduces two transpose operators, which can improve the running efficiency of the converted neural network model 30 to some extent, but multiple transpose operators still exist.

[0061] To further improve the running speed of the converted neural network model, this application provides a model processing method. In this method, when it is necessary to convert the data format of a first model (e.g., a neural network model) from a second data format to a first data format (e.g., when it is necessary to deploy the first model in the second data format to a runtime environment that supports the first data format), the electronic device can first determine the format-related operators in the first model. Then, the electronic device can adjust the order of the data in the reference identifiers of the parameters of the format-related operators in the first model in the second data format to correspond to the first data format, thus obtaining the first model in the first data format. Specifically, after adjusting the order of the data in the reference identifiers of the parameters to correspond to the first data format, the data in a tensor using the first data format referenced by the adjusted reference identifier is the same as the data in the tensor using the second data format referenced by the original reference identifier.

[0062] Thus, since no additional transpose operator is introduced in the first model of the first data format, the first model of the first data format is adapted to the operating environment of the second data format by adjusting the reference identifier of the parameters in the format-related operators, thereby improving the operating efficiency of the first model of the first data format.

[0063] Specifically, when it is necessary to convert the data format of the first model from the second data format to the first data format, the electronic device can first use an application that deploys a neural network model to add an adjacent first transpose operator before each format-related operator in the first model of the second data format, and an adjacent second transpose operator after each format-related operator in the first model of the second data format, to obtain an intermediate model of the first data format. The first transpose operator is used to convert the first data format to the second data format, and the second transpose operator is used to convert the second data format to the first data format.

[0064] Then, the electronic device can remove the first transpose operator adjacent to each format-related operator and the second transpose operator adjacent to each format-related operator in the intermediate model, and adjust the arrangement order of the data in the reference identifiers of each parameter in each format-related operator according to the second data format and the first data format to correspond to the first data format, thus obtaining the first model in the first data format. Specifically, after the arrangement order of the data in the reference identifiers of each parameter is adjusted to correspond to the first data format, the data in a tensor using the first data format referenced by the adjusted reference identifier is the same as the data in the tensor using the second data format referenced by the original reference identifier.

[0065] In this way, without adding extra transpose operators before and after the format-related operators in the first model, the data format of the first model can be converted from the second data format to the first data format, which is beneficial to improving the running speed of the first model with the first data format.

[0066] For example, refer to Figure 4 For the aforementioned neural network model 10 (as the first model in the second data format) with NCHW (as the second data format), the electronic device can first use the application that deploys the neural network model to convert the neural network model into the aforementioned neural network model 20 (as the intermediate model) with NHWC (as the first data format).

[0067] Then, the electronic device can remove the NHWC-NCHW operator (as the first transpose operator) adjacent to each format-related operator in the neural network model 20 and the NCHW-NHWC operator (as the second transpose operator) adjacent to each format-related operator, and move the second data in the reference identifier of the parameter of each format-related operator to the fourth position, so that the data format of the data referenced by each parameter corresponds to the NHWC format, thus obtaining the neural network model 40 (as the first model of the first data format).

[0068] For example, refer to Figure 4In operator 12, parameter Q1(u,x,y,z) references data from tensor Q1 in NCHW format, where the batch size is u, the channel size is x, the height is y, and the width is z. The electronic device can move the second reference marker (reference marker x) to the fourth position, updating parameter Q1(u,x,y,z) to parameter Q1(u,y,z,x), which references data from tensor Q in NHWC format with the batch size u, height y, width z, and channel size x. In other words, parameters Q1(u,x,y,z) and Q1(u,y,z,x) reference the same data in tensor Q1 and will not affect the processing result of operator 12.

[0069] For example, parameter Q2(u,x,y,z) in operator 13 references data from tensor Q2 in NCHW format, where the batch size is u, the channel size is x, the height is y, and the width is z. The electronic device can move the second reference marker (reference marker x) to the fourth position, updating parameter Q2(u,x,y,z) to parameter Q2(u,y,z,x), which references data from tensor Q in NHWC format with the batch size u, height y, width z, and channel size x. In other words, parameters Q2(u,x,y,z) and Q2(u,y,z,x) reference the same data in tensor Q2 and will not affect the processing result of operator 13.

[0070] For example, parameter Q3(u,x,y,z) in operator 15 references data from tensor Q3 in NCHW format, where the batch size is u, the channel size is x, the height is y, and the width is z. The electronic device can move the second reference marker (reference marker x) to the fourth position, updating parameter Q3(u,x,y,z) to parameter Q3(u,y,z,x), which references data from tensor Q in NHWC format with the batch size u, height y, width z, and channel size x. In other words, parameters Q3(u,x,y,z) and Q3(u,y,z,x) reference the same data in tensor Q3 and will not affect the processing result of operator 15.

[0071] Thus, the number of operators in neural network model 40 is the same as that in neural network model 10. Running neural network model 40 does not require running the transpose operator, which helps to improve the running speed of neural network model after data format conversion.

[0072] further, Figure 5 According to some embodiments of this application, a schematic flowchart of a model processing method is shown. The execution subject of this process is an electronic device, such as... Figure 5As shown, the process includes the following steps:

[0073] S501: Obtain the first model in the second data format.

[0074] An electronic device acquires a first model of data to be converted. The first model has a second data format and includes at least one format-related operator.

[0075] S502: In the first model of the second data format, add an adjacent first transpose operator before each format-related operator and add an adjacent second transpose operator after each format-related operator in the first model of the second data format to obtain an intermediate model of the first data format.

[0076] Electronic devices can utilize applications that deploy neural network models to add adjacent first transpose operators before each format-related operator in the first model of the second data format, and adjacent second transpose operators after each format-related operator in the first model of the second data format, to obtain an intermediate model of the first data format. Specifically, the first transpose operator is used to convert data in the first data format to data in the second data format, and the second transpose operator is used to convert data in the second data format to data in the first data format.

[0077] For example, corresponding to the aforementioned neural network model 10, the electronic device can use an application for deploying the neural network model to determine the format-related operators in the neural network model, including operators 12, 13, and 14. Then, the electronic device can use the application for deploying the neural network model to add an NHWC-NCHW operator 12A (as a first transpose operator) before operator 12 and an NCHW-NHWC operator 12B (as a second transpose operator) after operator 12 in the neural network model 10; add an NHWC-NCHW operator 13A before operator 13 and an NCHW-NHWC operator 13B after operator 13; add an NHWC-NCHW operator 15A before operator 15 and an NCHW-NHWC operator 15B after operator 15, thus obtaining a neural network model 20 (as an intermediate model).

[0078] It should be understood that in some embodiments, the electronic device may also obtain an intermediate model of the first data format without utilizing the application that deploys the neural network model, by executing relevant instructions or other means, by adding adjacent first transpose operators before each format-related operator in the first model of the second data format and adding adjacent second transpose operators after each format-related operator in the first model of the second data format, which is not limited here.

[0079] It should be understood that in some embodiments, the electronic device may also directly obtain the intermediate model from other devices without the electronic device itself performing the conversion of the first model in the second data format into the intermediate model in the first data format, which is not limited here.

[0080] It should be understood that format-dependent operators can be operators that process input data with different formats but the same content to obtain different output data. That is, operators that associate the output data with the data format of the neural network model, including but not limited to: convolution operators, local response normalization (LRN) operators, deconvolution operators, pooling operators, fully connected operators, etc.

[0081] It should be understood that, in the first model, in addition to format-dependent operators, format-independent operators may also be included.

[0082] S503: Delete the first transpose operator adjacent to each format-related operator and the second transpose operator adjacent to each format-related operator in the intermediate model, and adjust the arrangement order of the data in the reference identifiers of each parameter in each format-related operator to correspond to the first data format, so as to obtain the first model of the first data format.

[0083] After obtaining the intermediate model of the first data format, the electronic device can delete the first transpose operator and the second transpose operator adjacent to each format-related operator in the intermediate model, and adjust the arrangement order of the data in the reference identifier of each parameter in each format-related operator to correspond to the first data format (for example, adjust the first reference identifier of each parameter from the first identifier to the second identifier) ​​to obtain the first model of the first data format.

[0084] It should be understood that the data referenced by a parameter when referencing a tensor in a first data format based on the adjusted second identifier is the same as the data referenced by the parameter when referencing the tensor in a second data format based on the unadjusted first identifier.

[0085] For example, suppose that both the second and first data formats correspond to an n-dimensional tensor. The first dimension (any one of the n dimensions) of the n-dimensional tensor is positioned at the Kth position in the second data format, while the first dimension is positioned at the Mth position in the first data format. The electronic device can move the data at the Kth position from the reference identifiers of the parameters of the operators related to each format to the Mth position, so that the reference identifiers of each parameter correspond to the first data format.

[0086] It should be understood that, in the reference identifiers of the parameters of the format-related operators, after the data at position K is moved to position M, the data at position M can be adjusted to position P. Specifically, the second dimension (any dimension among the n dimensions different from the first dimension) corresponding to position M in the first data format is ordered as position P in the second data format. This process continues until the data at each position in the reference identifiers of the parameters of the format-related operators corresponds to the first data format.

[0087] For example, corresponding to the aforementioned neural network model 20 (as an intermediate model), both the second data format and the first data format correspond to a 4-dimensional tensor, and the second data format is NCHW, while the first data format is NHWC.

[0088] refer to Figure 6A For the N-dimensional coordinate system, K = M = 1, for any parameter Q1(u,x,y,z) in operator 12, the position of the first data (u) in the reference identifier remains unchanged. For the C-dimensional coordinate system, K = 2 and M = 4, so the second data (x) in the reference identifier of Q1(u,x,y,z) is adjusted to the fourth position. For the H-dimensional coordinate system, K = 3 and M = 2, so the third data (y) in the reference identifier of Q1(u,x,y,z) is adjusted to the second position. For the W-dimensional coordinate system, K = 4 and M = 3, so the fourth data (z) in the reference identifier of Q1(u,x,y,z) is adjusted to the third position. In other words, parameter Q1(u,x,y,z) is adjusted to Q1(u,y,z,x). In other words, the data referenced by parameter Q1(u,x,y,z) when referencing tensor Q1 in NCHW format is the same as the data referenced by parameter Q1(u,y,z,x) when referencing tensor Q1 in NHWC format.

[0089] refer to Figure 6BFor the N-dimensional coordinate system, K = M = 1, for any parameter Q2(u,x,y,z) in operator 12, the position of the first data (u) in the reference identifier remains unchanged. For the C-dimensional coordinate system, K = 2 and M = 4, so the second data (x) in the reference identifier of Q2(u,x,y,z) is adjusted to the fourth position. For the H-dimensional coordinate system, K = 3 and M = 2, so the third data (y) in the reference identifier of Q2(u,x,y,z) is adjusted to the second position. For the W-dimensional coordinate system, K = 4 and M = 3, so the fourth data (z) in the reference identifier of Q2(u,x,y,z) is adjusted to the third position. In other words, parameter Q2(u,x,y,z) is adjusted to Q2(u,y,z,x). In other words, the data referenced by parameter Q2(u,x,y,z) when referencing tensor Q2 in NCHW format is the same as the data referenced by parameter Q2(u,y,z,x) when referencing tensor Q2 in NHWC format.

[0090] refer to Figure 6C For the N dimension, K = M = 1, for any parameter Q3(u,x,y,z) in operator 12, the position of the first data (u) in the reference identifier remains unchanged; for the C dimension, K = 2 and M = 4, therefore the second data (x) in the reference identifier of Q3(u,x,y,z) is adjusted to the fourth position; for the H dimension, K = 3 and M = 2, therefore the third data (y) in the reference identifier of Q3(u,x,y,z) is adjusted to the second position; for the W dimension, K = 4 and M = 3, therefore the fourth data (z) in the reference identifier of Q3(u,x,y,z) is adjusted to the third position. In other words, parameter Q3(u,x,y,z) is adjusted to Q3(u,y,z,x). In other words, the data referenced by parameter Q3(u,x,y,z) when referencing tensor Q3 in NCHW format is the same as the data referenced by parameter Q3(u,y,z,x) when referencing tensor Q3 in NHWC format.

[0091] In some embodiments, the electronic device may also delete the first transpose operator preceding each format-related operator and the second transpose operator following each format-related operator in the intermediate model in the following manner, and adjust the arrangement order of the data in the reference identifiers of each parameter in each format-related operator to correspond to the first data format, thereby obtaining the first model of the first data format:

[0092] First, for any operator in the intermediate model, if there is an adjacent first transpose operator before the operator and an adjacent second transpose operator after the operator (e.g., a format-dependent operator), the electronic device can swap the order of the first transpose operator and the operator, making the first transpose operator and the second transpose operator adjacent (hereinafter, the first transpose operator and the second transpose operator adjacent to the first transpose operator are referred to as a transpose operator pair). Furthermore, the electronic device can adjust the arrangement order of the data in the reference identifiers of each parameter in the operator to correspond to the first data format (for details, please refer to the above description, which will not be repeated here).

[0093] For example, refer to Figure 7 For any format-dependent operator, the electronic device can swap the order of the first transpose operator and the format-dependent operator, making the first transpose operator and the second transpose operator adjacent. Furthermore, the electronic device can adjust the order of data in the reference identifier of the parameter in the format-dependent operator (see the foregoing description for details, which will not be repeated here).

[0094] Then, the electronic device can delete and adjust the order, and then delete all transpose operator pairs to obtain the first model of the first data format.

[0095] For example, refer to Figure 7 The electronic device can delete the adjacent transpose operator pairs after each format-related operator to obtain the first model of the first data format.

[0096] Based on the above adjustments, neural network model 10 can be converted into the aforementioned neural network model 40 (as the first model in the first data format).

[0097] By using the above method, the electronic device can eliminate the transpose operator in the conversion of the first model in the second data format to the first model in the first data format, which is beneficial to improving the running efficiency of the first model in the first data format.

[0098] In some embodiments, the electronic device may also directly adjust the parameters in the format-related operators in the first model of the second data format instead of first converting the first model of the second data format into an intermediate model.

[0099] For example, Figure 8 According to some embodiments of this application, a flowchart of another model processing method is shown. The execution entity of this process is an electronic device, such as... Figure 8 As shown, the process includes the following steps:

[0100] S801: Obtain the first model in the second data format.

[0101] Electronic devices can acquire a first model in a second data format.

[0102] In some embodiments, the first model of the second data format may include operator identifiers of each operator in the first model and parameters in each operator.

[0103] In some embodiments, the operator identifier can be any identifier that can distinguish different operators, including but not limited to the operator's name, the operator's unique identifier, etc.

[0104] S802: Format-dependent operators in the first model that determine the second data format.

[0105] The electronic device determines the format-related operators in the first model.

[0106] In some embodiments, the electronic device may store operator identifiers of format-dependent operators and compare the operator identifiers of each operator in the first model of the second data format with the stored operator identifiers. If the operator identifier of a certain operator exists in the operator identifiers stored by the electronic device, the operator is determined to be a format-dependent operator.

[0107] It should be understood that in other embodiments, the electronic device may also determine the format-related operators in the first model of the second data format in other ways, which is not limited here.

[0108] S803: Adjust the order of data in the reference identifiers of parameters in each format-related operator to correspond to the first data format, and obtain the first model of the first data format.

[0109] After determining the format-related operators in the first model, the electronic device can adjust the order of the data in the reference identifiers of each parameter in each format-related operator to correspond to the first data format (for example, changing the first identifier of each parameter from the first identifier to the second identifier).

[0110] For example, suppose that both the second and first data formats correspond to an n-dimensional tensor. The first dimension (any one of the n dimensions) of the n-dimensional tensor is positioned at the Kth position in the second data format, while the first dimension is positioned at the Mth position in the first data format. The electronic device can move the data at the Kth position from the reference identifiers of the parameters of the operators related to each format to the referenced Mth position, so that the reference identifiers of each parameter correspond to the first data format. For details, please refer to step S503 above, which will not be elaborated here.

[0111] Using the above method, the electronic device does not need to introduce an additional transpose operator in the conversion of the first model in the second data format to the first model in the first data format, which is beneficial to improving the running efficiency of the first model in the first data format.

[0112] further, Figure 9According to some embodiments of this application, a schematic diagram of an electronic device 100 is shown. This electronic device can be used to implement the model processing methods provided in the foregoing embodiments.

[0113] like Figure 9 As shown, the electronic device 100 includes one or more processors 101, system memory 102, non-volatile memory (NVM) 103, communication interface 104, input / output device 105, and system control logic 106 for coupling the processor 101, system memory 102, non-volatile memory 103, communication interface 104, and input / output (I / O) device 105. Wherein:

[0114] Processor 101 may include one or more processing units, such as a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), microprocessor (MCU), artificial intelligence (AI) processor, field programmable gate array (FPGA), neural network processing unit (NPU), etc. The processing module or circuit may include one or more single-core or multi-core processors. In some embodiments, the CPU may be used to optimize a neural network model to be run, and the NPU may be used to run the neural network model.

[0115] System memory 102 is volatile memory, such as random-access memory (RAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc. System memory is used for temporary storage of data and / or instructions. For example, in some embodiments, system memory 102 can be used to store the first model of the aforementioned second data format, the intermediate model of the first data format, the parameters of each operator, etc., and can also be used to store instructions for the model processing methods provided in the aforementioned embodiments.

[0116] The non-volatile memory 103 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 103 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), a compact disc (CD), a digital versatile disc (DVD), a solid-state drive (SSD), etc. In some embodiments, the non-volatile memory 103 may also be a removable storage medium, such as a secure digital (SD) memory card, etc. In other embodiments, the non-volatile memory 103 may be used for the first model of the aforementioned second data format, an intermediate model of the first data format, the parameters of each operator, etc., and may also be used to store instructions for the model processing methods provided in the foregoing embodiments.

[0117] Specifically, system memory 102 and non-volatile memory 103 may each include a temporary copy and a permanent copy of instruction 107. Instruction 107 may include, when executed by at least one of processors 101, causing electronic device 100 to implement the model processing methods provided in the embodiments of this application.

[0118] The communication interface 104 may include a transceiver for providing a wired or wireless communication interface for the electronic device 100, thereby enabling communication with any other suitable device via one or more networks. In some embodiments, the communication interface 104 may be integrated into other components of the electronic device 100, for example, the communication interface 104 may be integrated into the processor 101. In some embodiments, the electronic device 100 may communicate with other devices through the communication interface 104; for example, the electronic device 100 may obtain a first model in a second data format, an intermediate model in the first data format, etc., from other electronic devices through the communication interface 104.

[0119] The input / output (I / O) device 105 can be an input device such as a keyboard or mouse, and an output device such as a monitor. Users can interact with the electronic device 100 through the input / output (I / O) device 105, such as inputting a first model in a second data format.

[0120] System control logic 106 may include any suitable interface controller to provide any suitable interface to other modules of electronic device 100. For example, in some embodiments, system control logic 106 may include one or more memory controllers to provide an interface to system memory 102 and non-volatile memory 103.

[0121] In some embodiments, at least one of the processors 101 may be packaged together with the logic of one or more controllers for system control logic 106 to form a system-in-package (SiP). In other embodiments, at least one of the processors 101 may also be integrated on the same chip with the logic of one or more controllers for system control logic 106 to form a system-on-chip (SoC).

[0122] Understandable. Figure 9 The structure of the electronic device 100 shown is merely an example. In other embodiments, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0123] It is understood that electronic device 100 can be any electronic device, including but not limited to mobile phones, wearable devices (such as smartwatches), tablets, desktops, laptops, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, etc., and this application embodiment does not limit it.

[0124] This application also provides a computer program product that, when run on an electronic device, enables the electronic device to implement any of the model processing methods provided in this application.

[0125] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0126] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application-specific integrated circuit, or a microprocessor.

[0127] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0128] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, optical discs, magneto-optical disks, read-only memory, random access memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other propagation signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0129] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0130] It should be noted that the units / modules mentioned in the various device embodiments of the present invention are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problem proposed by the present invention. Furthermore, to highlight the innovative aspects of the present invention, the above-described device embodiments of the present invention have not introduced units / modules that are not closely related to solving the technical problem proposed by the present invention. This does not mean that the above-described device embodiments do not contain other units / modules.

[0131] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0132] Although the invention has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the scope of the invention.

Claims

1. A model processing method applied to electronic devices, characterized in that, include: Obtain a first model of a second data format to be deployed to a runtime environment of a first data format, wherein the first model includes multiple operators; The reference identifiers of parameters in at least some of the plurality of operators are adjusted from a first identifier to a second identifier to obtain a first model in a first data format, wherein, The data of the tensor with the second data format referenced by the parameter using the first identifier is the same as the data of the tensor with the first data format referenced by the parameter using the second identifier. The step of adjusting the reference identifiers of parameters in at least some of the plurality of operators from a first identifier to a second identifier to obtain a first model in a first data format includes: A first operator is determined among the plurality of operators, wherein the processing result of the first operator on the input data is associated with the data format supported by the operating environment of the first model; Based on the first data format and the second data format, the reference identifier of the parameter in the first operator is adjusted from the first identifier to the second identifier; Determining the first operator among the plurality of operators includes: An intermediate model is obtained by converting a first model of a first application into a second data format. The intermediate model is obtained by adding an adjacent first transpose operator before each first operator in the first model of the second data format and adding an adjacent second transpose operator after each first operator. The first transpose operator is used to convert the first data format into the second data format, and the second transpose operator is used to convert the second data format into the first data format. In the intermediate model, the operator that has an adjacent first transpose operator before it and an adjacent second transpose operator after it is determined as the first operator.

2. The method according to claim 1, characterized in that, The step of adjusting the reference identifier of the parameter in the first operator from the first identifier to the second identifier based on the first data format and the second data format includes: Remove the first transpose operator and the second transpose operator from the intermediate model, and adjust the reference identifier of the parameter in the first operator from the first identifier to the second identifier based on the first data format and the second data format.

3. The method according to claim 1 or 2, characterized in that, The tensors corresponding to the first and second data formats each have N dimensions, and the first identifier includes N data points, where N is an integer greater than 1; furthermore, The step of adjusting the reference identifier of the parameter in the first operator from the first identifier to the second identifier based on the first data format and the second data format includes: The Kth data among the N data included in the first identifier is used as the Mth data in the second identifier, where K is an integer less than or equal to N, the Kth dimension in the first data format is the first dimension, and the first dimension is the Mth dimension in the second data format.

4. The method according to claim 3, characterized in that, N is 4.

5. The method according to claim 4, characterized in that, The first data format and / or the second data format is any one of the following data formats: NCHW format, NHWC format, NHCW format, NCWH format, NWCH format, NWHC format, CNHW format, CHWN format, CHNW format, CNWH format, CWNH format, CWHN format, HCNW format, HNWC format, HNCW format, HCWN format, HWCN format, HWNC format, WCHN format, WHNC format, WHCN format, WCNH format, WNCH format, WNHC format.

6. The method according to claim 1, characterized in that, The first model is a neural network model.

7. A computer-readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the method of any one of claims 1 to 6.

8. An electronic device, characterized in that, include: Memory is used to store instructions executed by one or more processors of an electronic device; And a processor, one of the processors of the electronic device, for executing instructions stored in the memory to implement the method of any one of claims 1 to 6.

Images