Neural network model compiling method, data loading method, equipment and medium

Abstract

The invention provides a compiling method of a neural network model, a data loading method, equipment and a medium, the neural network model comprises a target calculation module, a data length requirement corresponding to the target calculation module comprises a first length, and the method comprises the following steps: obtaining to-be-calculated data to be loaded to the target calculation module; the data length of the to-be-calculated data is a second length, and the second length is greater than the first length; based on pre-configured first data loading mode information, loading the to-be-calculated data to the target calculation module; wherein the first data loading mode information is obtained by converting the second data loading mode information matched with the second length according to the conversion relation between the first length and the second length. According to the method, data rearrangement is realized, calculation resources do not need to be additionally occupied during operation of the neural network model, and the performance of the neural network model during operation is improved.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a neural network model compilation method, data loading method, device and medium. Background Art

[0002] At present, AI (Artificial Intelligence) technology has been widely used in various fields. In different AI scenarios, the results required by users can be obtained based on neural network calculations. With the development of deep learning technology, the neural networks formed are getting deeper and larger in scale. The model parameters and input data of the neural network occupy a considerable storage space and transmission bandwidth. In addition, a large amount of input data and model parameters require more computing resources to complete a prediction. In order to speed up the operation of the neural network, the model parameters and input data of the neural network need to be rearranged. However, the input data of the neural network is usually unknown and is a tensor (dynamic shape) whose shape can change during runtime. The existing solution adds operators to the neural network model to rearrange the data in the dynamic shape model. This solution will generate additional resource consumption when executing the operator. Summary of the invention

[0003] The embodiments of the present application provide a neural network model compilation method, data loading method, device and medium to solve one or more of the above-mentioned technical problems.

[0004] In a first aspect, an embodiment of the present application provides a data loading method for a neural network model, wherein the neural network model includes a target computing module, and the data length requirement corresponding to the target computing module includes a first length, and the method includes: obtaining data to be calculated to be loaded into the target computing module; wherein the data length of the data to be calculated is a second length, and the second length is higher than the first length; based on pre-configured first data loading method information, loading the data to be calculated into the target computing module; wherein the first data loading method information is obtained by converting the second data loading method information matching the second length according to the conversion relationship between the first length and the second length.

[0005] In a second aspect, an embodiment of the present application provides a method for compiling a neural network model, comprising: obtaining a definition file of a neural network model to be compiled; calling a compiler to compile the definition file into an executable file; wherein, during the compilation process, first data loading mode information is configured to a target computing module included in the neural network model, and the data length requirement corresponding to the target computing module includes a first length; the first data loading mode information is obtained by converting second data loading mode information matching the second length according to a conversion relationship between the first length and the second length; the second length is higher than the first length.

[0006] In the third aspect, an embodiment of the present application provides a data loading device for a neural network model, wherein the neural network model includes a target calculation module, and the data length requirement corresponding to the target calculation module includes a first length, and the device includes: a data module, used to obtain the data to be calculated to be loaded into the target calculation module; wherein the data length of the data to be calculated is a second length, and the second length is higher than the first length; a loading module, used to load the data to be calculated into the target calculation module based on pre-configured first data loading method information; wherein the first data loading method information is obtained by converting the second data loading method information matching the second length according to the conversion relationship between the first length and the second length.

[0007] In a fourth aspect, an embodiment of the present application provides a compilation device for a neural network model, comprising: a file module, used to obtain a definition file of a neural network model to be compiled; a compilation module, used to call a compiler to compile the definition file into an executable file; wherein, during the compilation process, first data loading mode information is configured to a target computing module included in the neural network model, and the data length requirement corresponding to the target computing module includes a first length; the first data loading mode information is obtained by converting second data loading mode information matching the second length according to a conversion relationship between the first length and the second length; the second length is higher than the first length.

[0008] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods described above when executing the computer program.

[0009] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the methods described above is implemented.

[0010] In a seventh aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, it implements any of the methods described above.

[0011] Compared with the related art, this application has the following advantages:

[0012] The present application provides a compilation method, a data loading method, a device and a medium for a neural network model, wherein the neural network model includes a target computing module, and the data length requirement corresponding to the target computing module includes a first length, and the method includes: obtaining the data to be calculated to be loaded into the target computing module; wherein the data length of the data to be calculated is a second length, and the second length is higher than the first length; based on the pre-configured first data loading mode information, the data to be calculated is loaded into the target computing module; wherein the first data loading mode information is obtained by converting the second data loading mode information matching the second length according to the conversion relationship between the first length and the second length. According to an embodiment of the present application, while loading the data to be calculated, the data to be calculated in the dynamic shape model can be rearranged to extract the high and low bits of each value in the data to be calculated respectively, and then the extraction result can be used in the module after the target computing module in the neural network model, and data rearrangement is realized based on the present application, and the neural network model does not need to occupy additional computing resources when running, thereby improving the performance of the neural network model when running.

[0013] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0015] Figure 1 An original model structure diagram provided in an embodiment of the present application is shown;

[0016] Figure 2 A model structure diagram after adding an operator provided in an embodiment of the present application is shown;

[0017] Figure 3 A flow chart showing a method for loading data of a neural network model provided in an embodiment of the present application is shown;

[0018] Figure 4 A schematic diagram showing the arrangement of each byte of the original memory data of the input source provided in an embodiment of the present application is shown;

[0019] Figure 5 A schematic diagram showing the arrangement of each byte of memory data loaded for the first time in an embodiment of the present application is shown;

[0020] Figure 6 A schematic diagram showing the arrangement of each byte of memory data loaded for the second time in an embodiment of the present application is shown;

[0021] Figure 7 It shows a schematic diagram of data arrangement of each byte of the data at the target memory address after all inputs are loaded in the embodiment of the present application;

[0022] Figure 8 A corresponding relationship diagram between the original data and the rearranged data in an embodiment of the present application is shown;

[0023] Fig. 9 A flowchart showing a method for compiling a neural network model provided in an embodiment of the present application is shown.

[0024] Fig.10 A structural block diagram of a data loading device for a neural network model provided in an embodiment of the present application is shown;

[0025] Fig.11 A structural block diagram of a compiling device for a neural network model provided in an embodiment of the present application is shown;

[0026] Fig.12 A block diagram of an electronic device used to implement an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0028] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0029] The scale of neural networks currently in use is getting larger and larger, and more computing resources are required during operation. In some computing scenarios, the operation speed of the model can be accelerated by rearranging the data in the neural network model. For data known during compilation, such as the parameters of a trained neural network model, it can be rearranged during compilation, but for input that can only be obtained at runtime, such as image data input to the neural network model, it needs to be achieved through other methods.

[0030] The existing solution uses 8-bit fixed-point to represent the input data of the neural network model, so the dedicated AI accelerator hardware will accelerate the 8-bit fixed-point operation. However, when it is necessary to use the 8-bit arithmetic unit to support 16-bit fixed-point, the 16-bit data needs to be separated into high and low parts, and the input data of the neural network model needs to be rearranged in order to speed up the operation.

[0031] Dynamic shape means that the dimension value of the tensor depends on the specific operation and cannot be calculated in advance. In deep learning and neural networks, the shape of the tensor can change at runtime instead of being fixed at compile time. In other words, dynamic shape allows the network to have different dimension sizes when processing different batches of data. This is very useful in processing variable-length sequences, image processing, and other tasks that require flexible processing of input data. Data operations in dynamic shape models are usually implemented by inserting various operators into the model. Figure 1 An original model structure diagram provided in an embodiment of the present application is shown, wherein InputA and InputB represent the input of the model, and matmul represents a fragment in the model, for example, it can be a calculation module that is used to implement matrix multiplication operations. Figure 2 A model structure diagram after adding operators is shown, where bitcast and transpose are operators added to the model. Bitcast is used to modify the latitude value and type of a tensor without changing the value of the tensor. Transpose is used to implement the transposition operation of a matrix.

[0032] like Figure 2 As shown, assuming that InputB with a data type of int16 needs to be rearranged, then a bitcast is added to the output data of InputB to convert its data type to int8, and then a transpose operation is performed to rearrange it into the desired arrangement before it can be passed to matmul for operation.

[0033] The above solution can intuitively meet the needs of rearrangement, but there are the following two performance problems, which sometimes make it difficult to meet the performance requirements. When executing bitcast and transpose, the input data needs to be loaded once each time. Specifically, before executing bitcast, the input data involved in the module corresponding to bitcast will be loaded once, and before executing transpose, the input data involved in the module corresponding to transpose will be loaded once, which will cause additional resource consumption. It should be noted that the resources consumed by loading input data are not only the transmission of input data itself, but also many other performance consumptions may be generated according to different runtime implementation methods. For example, if the runtime is a stack machine, the number of instructions will also increase. Another performance point is that a relatively high-performance transpose implementation is required. In addition to performance issues, using the above method to rearrange data each time requires adding two operators to the graph. If multiple rearrangements are required, the complexity of the entire model will increase.

[0034] Based on this, the present application provides a compilation method, data loading method, device and medium for a neural network model. This method can rearrange the data in a dynamic shape model without inserting operators into the model. Compared with existing solutions, the number of operators can be reduced, and the complexity of the model can be reduced. In addition, this method can load the data to be calculated while rearranging the data to be calculated, thereby improving the performance of the neural network model during runtime.

[0035] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0036] The embodiment of the present application provides a method for loading data of a neural network model. The neural network model includes a target calculation module, and the data length requirement corresponding to the target calculation module includes a first length.

[0037] In the embodiments of the present application, the neural network model is a computational model that simulates the connection structure of neurons in the human brain and can be widely used in the fields of machine learning and artificial intelligence. The neural network model consists of multiple layers (also called operators or modules), each layer contains multiple neurons (nodes or units), which are connected by weights and transmit information to each other.

[0038] It should be noted that, at the software level, the execution subject of the data loading method of the neural network model is the executable file (executable program) obtained by compiling the definition file of the neural network model using a compiler. At the hardware level, the execution subject of the data loading method of the neural network model is the computing chip or computing device that runs this executable file or execution program.

[0039] In the embodiment of the present application, the data length is used to describe the storage space occupied by the unit data in the data to be calculated of the neural network model. For example, when the data type of the data to be calculated is int16, the storage space required for the unit data is 16 bits (bits), or 2 bytes, and when the data type of the data to be calculated is int8, the storage space required for the unit data is 8 bits, or 1 byte. Bits or bytes are the units of data length.

[0040] In a possible implementation, the target computing module is any module in the neural network model. The data length requirement corresponding to the target computing module includes at least a first length, that is, it may also include other lengths, which are different from the values ​​of the first length. The value of the first length can be set according to the data length supported by the execution subject of the method at the hardware level, for example, it can be set to 8 bits, 16 bits or 32 bits, etc.

[0041] In a possible implementation, the target calculation module can calculate the first group of data and the second group of data, and the data length requirement corresponding to the target calculation module includes a first length. It can be that the first group of data does not require data length, and the second group of data requires the length to be the first length. It can also be that both the first group of data and the second group of data require the length to be the first length. The specific setting can be based on actual needs.

[0042] It should be noted that in the embodiment of the present application, in order to facilitate the distinction from the data length of subsequent data to be calculated, the length at least included in the data length requirement corresponding to the target calculation module is called the first length, and the data length of the subsequent data to be calculated is called the second length.

[0043] like Figure 3 The flowchart of the data loading method of the neural network model according to one embodiment of the present application is shown. The method may include:

[0044] Step 301, obtaining data to be calculated to be loaded into the target calculation module; wherein the data length of the data to be calculated is a second length, and the second length is greater than the first length;

[0045] In an embodiment of the present application, the data to be calculated needs to be loaded into the target computing module, and the data length of the data to be calculated is a second length, which is higher than the first length included in the data length requirement corresponding to the target computing module.

[0046] In this step, the data to be calculated to be loaded into the target calculation module is obtained, thereby obtaining the data to be calculated to be loaded, and the data to be calculated can be used to input into the target calculation module, without adding additional operators to the neural network model. For example, in the specific implementation, refer to Figure 1 , the target computing module is matmul shown in the figure, and the data length requirement corresponding to this module includes InputB being the first length. Then the obtained data to be calculated InputB can be used to load into the target computing module without following Figure 2 The process shown in the figure adds additional bitcast operators and transpose operators to the model, thereby avoiding changes in the model structure and increasing the complexity of the entire model.

[0047] Step 302, based on the pre-configured first data loading method information, load the data to be calculated into the target computing module; wherein the first data loading method information is obtained by converting the second data loading method information matching the second length according to the conversion relationship between the first length and the second length.

[0048] In the embodiment of the present application, the second length is pre-matched with second data loading mode information, and the second data loading mode information is configured with step information and shape information of the data to be calculated. The second data loading mode information is used to determine the step of loading the data to be calculated into the target calculation module according to the step information and the shape information, for example, according to the step size of 1, the data to be calculated with a shape of [n, c, h, w] is loaded into the target calculation module.

[0049] It should be noted that the shape of the data is [n, c, h, w], which can be understood as a four-dimensional array, where: n is the size of the first dimension (which can be understood as how many such four-dimensional arrays there are). c is the size of the second dimension. h is the size of the third dimension. w is the size of the fourth dimension. For example, there are n images, the number of layers (channels) of the image is c, the height is h, and the width is w.

[0050] There is a conversion relationship between the first length and the second length. In a possible implementation, the second length is a multiple of the first length, for example, it can be one, two, three, four or more times. In a possible implementation, the first length is 8 bits and the second length is 16 bits. According to the conversion relationship between the first length and the second length, the method for converting the step size and / or shape in the second data loading method information can be determined. For example, based on the current shape, the step size can be adjusted from 1 to 2, and then according to the conversion relationship between the first length and the second length, the second data loading method information matching the second length is converted to obtain the first data loading method information.

[0051] The principles involved in the above conversion are explained below.

[0052] Assume that the original data type is int16, that is, each data occupies 16 bits (2 bytes). The shape of the data is [n, c, h, w], and the original stride is [c, h, w, 1]. First, treat the int16 type data as int8 type data. Each int8 data occupies 8 bits (1 byte), so each int16 type data now becomes two int8 data. The two last dimensions h and w are regarded as a new dimension h×w and 1, and the new shape becomes [n, c, h×w, 1], and the new stride becomes [c, h×w, 1, 1] accordingly. Since each int16 data now becomes two int8 data, the size of the lowest dimension needs to be doubled: the new shape becomes [n, c, h×w, 2], and the new stride becomes [c, h×w, 2, 1]. In the new shape and stride, set the stride on the w dimension to 2, so that a value is skipped each time the data is read. This method will read the upper 8 bits of each int16 data. In order to read the lower 8 bits of each int16 data, the data address needs to be offset backward by 1 byte, so that the lower 8 bits just skipped can be read. Finally, the read upper 8 bits and lower 8 bits are combined together to complete the entire data rearrangement process. This process rearranges the data originally stored in int16 into int8, and reads and combines the data according to specific rules. This method can process and store data more efficiently, especially when a large number of operations need to be performed on the data.

[0053] It should be noted that the original shape and stride are: original shape, n, c, h, w, original stride, c, h, w, 1, and the converted shape and stride are: new shape: n, c, h × w, 2, new stride, c, h × w, 2, 1, where the stride indicates the number of bytes required to move to the next element in memory. In order to understand the stride in the w dimension, it is necessary to extract information from the new stride c, h × w, 2, 1: Innermost dimension (last dimension): stride is 1, indicating that the byte interval between adjacent elements (i.e., 2 int8 data) is 1. The second innermost dimension (the second to last dimension): stride is 2, indicating the number of bytes required to cross an int16 data (i.e., 2 int8 data). The third dimension (the third to last dimension): stride is h\times w, indicating the number of bytes required to cross a complete h\times w matrix. The outermost dimension (the first dimension): stride is c, indicating the number of bytes required to cross a complete c channel. In the new shape, the w dimension is actually included in the h\times w dimension. Therefore, it is necessary to understand what the step length is on the w dimension under this new layout. Since each int16 data is split into two int8 data in the new shape [n, c, h\times w, 2], the actual step length should be: each time it moves one position on the w dimension, the step length is 2 (because each int16 data is split into two int8 data).

[0054] In summary, under the new shape and step size definition, the step size in the w dimension is 2. This is because each time you move one position in the w dimension, you actually cross two int8 data (that is, one int16 data is split into two parts).

[0055] In an embodiment of the present application, the data to be calculated with a data length of the second length is split, that is, rearranged into two or more data with a data length of the first length, so as to use the rearranged data of the first length to perform calculations in the target calculation module, and meet the length requirements of the data to be calculated at the hardware level. Among them, rearrangement refers to the processing of data downsizing, reducing data size, etc., for example, rearranging 16-bit data to obtain 8-bit data. It should be noted that, usually due to hardware design limitations, when the target calculation module of the neural network model is running, the supported data length is the first length. For example, the target calculation module is used to perform matrix multiplication calculations, and hardware limitations result in multiplications that can only be performed between matrix data of the first matrix, or multiplications that can only be performed between matrix data of one side of the first length, etc. Therefore, it is necessary to rearrange the data to be calculated that exceeds the first length to meet the data length requirements of the hardware.

[0056] It should also be noted that, in the embodiment of the present application, based on the pre-configured first data loading mode information, the data to be calculated is loaded into the target calculation module to participate in the calculation in the target calculation module. In essence, the data format is changed (or called downgrading, that is, reducing the data length) in the process of loading the data to be calculated. For example, the data to be calculated is matrix B, which will be split into two matrices, namely B1 and B2. When the target calculation module performs calculations, for example, matrix multiplication is performed with matrix A, the product of A and B is the same as the product of A with B1 and B2. That is to say, in the embodiment of the present application, based on the pre-configured first data loading mode information, loading the data to be calculated into the target calculation module will not affect the value of the calculation result output by the target calculation module.

[0057] The embodiment of the present application provides a method for compiling a neural network model, wherein the neural network model includes a target calculation module, and the data length requirement corresponding to the target calculation module includes a first length, and the method includes: obtaining the data to be calculated to be loaded into the target calculation module; wherein the data length of the data to be calculated is a second length, and the second length is higher than the first length; based on the pre-configured first data loading mode information, the data to be calculated is loaded into the target calculation module; wherein the first data loading mode information is obtained by converting the second data loading mode information matching the second length according to the conversion relationship between the first length and the second length. According to the embodiment of the present application, while loading the data to be calculated, the data to be calculated in the dynamic shape model can be rearranged to extract the high and low bits of each value in the data to be calculated respectively, and then the extraction result can be used in the module after the target calculation module in the neural network model. Based on the data rearrangement implemented in the present application, the neural network model does not need to occupy additional computing resources when running, thereby improving the performance of the neural network model when running.

[0058] In a possible implementation, when the target computing module is the input layer of the neural network model, the data to be calculated is the original input data of the neural network model; when the target computing module is the intermediate layer of the neural network model, the data to be calculated is the calculation result output by the previous layer of the intermediate layer.

[0059] In this possible implementation, the target computing module is the input layer or any intermediate layer of the neural network model. In the case where the target computing module is the input layer of the neural network model, the data to be calculated is the original input data of the neural network model. In the embodiment of the present application, the neural network can be used to process data in multiple fields, including but not limited to: medical health: disease diagnosis, personalized treatment, drug discovery, etc.; financial technology: risk management, algorithmic trading, fraud detection, etc.; autonomous driving: environmental perception, path planning, vehicle control, etc.; e-commerce: recommendation system, customer service, demand forecasting, etc.; smart home: voice assistant, smart security, home appliance automation, etc.; education: intelligent tutoring, automatic scoring, learning analysis, etc.; industrial manufacturing: predictive maintenance, quality control, production optimization, etc.; entertainment: content generation, virtual reality, game AI, etc. In the case where the target computing module is the intermediate layer of the neural network model, the data to be calculated is the calculation result output by the upper layer of the intermediate layer.

[0060] In this possible implementation, data can be loaded into the input layer or any intermediate layer of the neural network, thereby improving the scope of application of the data loading method of the neural network model, and the method can be applied to a variety of different fields.

[0061] In a possible implementation manner, the original input data is one or more of picture data, video data, audio data, and text data.

[0062] In this possible implementation, the original input data is one or more of picture data, video data, audio data, and text data. Different types of data have different requirements and characteristics when input into a neural network. Picture and video data usually have high-dimensional characteristics and local or temporal correlations, and are suitable for processing using convolutional neural networks (CNNs) and recurrent neural networks (RNNs). Audio data, as a one-dimensional time series signal, is often processed through frequency domain feature extraction and timing models. Text data, as a discrete symbol sequence, relies on word embedding and context correlation modeling. When processing different types of original input data, selection can be made based on the above requirements and characteristics. Since different types of original input data can be applied, the method can be applied to a variety of different usage scenarios.

[0063] In a possible implementation, based on the pre-configured first data loading mode information, the data to be calculated is loaded into the target calculation module, which can be performed according to the following steps: the high 8 bits of data in every 16 bits of data in the data to be calculated are loaded into the target calculation module; the low 8 bits of data in every 16 bits of data in the data to be calculated are loaded into the target calculation module.

[0064] In this possible implementation, the high-order data refers to the front part of each unit data, and the low-order data refers to the back part of each unit data. For example, for a 16-bit data "aa02", the high-order data refers to the first 8 bits of data, also called the high 8-bit data, and the low-order data refers to the last 8 bits of data, also called the low 8-bit data. The high-order 8 bits of data in each 16-bit data to be calculated are loaded into the target calculation module, so that the high-order 8 bits of data corresponding to each value in the data to be calculated are first loaded into the target calculation module. For example, see Figure 4 The following is a schematic diagram showing the arrangement of each byte of the original memory data of the input source. The data to be calculated includes Figure 4 The string of characters shown in the figure is "aa02 aa02 5405 5405ff07 ff07", where aa02, 5405, and ff07 represent a 16-bit data respectively. The high 8-bit data of each 16-bit data in the data to be calculated is loaded into the target calculation module. Figure 5 The data arrangement diagram of each byte of the first loaded memory data is shown in the figure. The result of executing the loading is that "aa aa 5454ff ff" is loaded into the target computing module. After that, the lower 8 bits of each 16 bits of data in the data to be calculated are loaded into the target computing module, that is, the lower 8 bits of data corresponding to each value in the data to be calculated are loaded into the target computing module. Figure 6 The data arrangement diagram of each byte of the memory data loaded for the second time is shown in the figure. The result of executing the load is to load "02 02 05 05 07 07" into the target computing module. The final result is as follows Figure 7 The data arrangement diagram of each byte of the target memory address after all the inputs are loaded is "aa aa 5454ff ff 02 0205 05 07 07".

[0065] In the above example, the original arrangement of the data is as follows Figure 4 As shown, the first value obtained when taking the stride (the step length required to jump from one element to the next in a specified dimension) with w being 2 is Figure 5 Then add 1 to the starting address, that is, start from position 02 and follow the general method to get the value. Figure 6 By using the load operation with stride to load data into adjacent positions, the value of the second part is placed after the data obtained once. Figure 7 . Figure 8The corresponding relationship diagram between the original data and the rearranged data is shown, and it can be seen how the rearranged data is adjusted compared to the data before the rearrangement.

[0066] Combination Figure 1 and 2 It can be seen that the embodiment of the present application regards the 16-bit data to be rearranged as 8 bits when loading the input through matmul, thereby dividing it into 8-bit high and low bits, and then uses load with stride to retrieve data, and loads the high and low bits into a continuous memory respectively, so as to achieve data rearrangement, so that the 8-bit operator can read the high and low bits in the continuous 16-bit input, without incurring the overhead of executing additional operators to load parameters multiple times, and without having to implement a high-performance transpose.

[0067] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0068] The present application embodiment provides a method for compiling a neural network model, such as Fig. 9 The flowchart of the method for compiling a neural network model according to an embodiment of the present application is shown. The execution subject of the method may be a person or some software that can call a compiler. The method may include:

[0069] Step 901, obtain the definition file of the neural network model to be compiled.

[0070] In an embodiment of the present application, the definition file of the neural network model can be used to parse the structure of the model, including the type, parameters, connection relationships, etc. of each layer.

[0071] In this step, by obtaining the definition file of the neural network model to be compiled, various types of data required for the neural network model compilation process can be obtained.

[0072] Step 902, calling a compiler to compile the definition file into an executable file; wherein, during the compilation process, the first data loading mode information is configured to the target computing module included in the neural network model, and the data length requirement corresponding to the target computing module includes a first length; the first data loading mode information is obtained by converting the second data loading mode information matching the second length according to a conversion relationship between the first length and the second length; the second length is higher than the first length.

[0073] In an embodiment of the present application, the definition file of the neural network model is compiled into an executable file so as to run the executable file on the hardware. Wherein, during the compilation process, the first data loading mode information is configured to the target computing module included in the neural network model, and the data length requirement corresponding to the target computing module includes a first length; the first data loading mode information is obtained by converting the second data loading mode information matching the second length according to the conversion relationship between the first length and the second length; the second length is higher than the first length, so that the executable file obtained after compilation can be used to execute the data loading method of the aforementioned neural network model, and the corresponding description in the data loading method of the aforementioned neural network model can be referred to, and it has corresponding beneficial effects, which will not be repeated here.

[0074] According to the embodiment of the present application, the data to be calculated in the dynamic shape model can be rearranged while loading the data to be calculated, so as to extract the high and low bits of each value in the data to be calculated respectively, and then the extraction results can be used in the module after the target calculation module in the neural network model. Based on the data rearrangement implemented in the present application, the neural network model does not need to occupy additional computing resources when running, thereby improving the performance of the neural network model when running.

[0075] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a data loading device for a neural network model. Fig.10 The figure is a structural block diagram of a data loading device of a neural network model according to an embodiment of the present application, wherein the neural network model includes a target calculation module, the data length requirement corresponding to the target calculation module includes a first length, and the device includes:

[0076] A data module 1001 is used to obtain data to be calculated to be loaded into the target calculation module; wherein the data length of the data to be calculated is a second length, and the second length is higher than the first length; a loading module 1002 is used to load the data to be calculated into the target calculation module based on pre-configured first data loading method information; wherein the first data loading method information is obtained by converting the second data loading method information matching the second length according to a conversion relationship between the first length and the second length.

[0077] According to the embodiments of the present application, the data to be calculated in the dynamic shape model can be rearranged while loading the data to be calculated, so as to extract the high and low bits of each value in the data to be calculated respectively, and then the extraction result can be used in the module after the target calculation module in the neural network model. Based on the data rearrangement implemented in the present application, the neural network model does not need to occupy additional computing resources when running, thereby improving the performance of the neural network model when running.

[0078] In a possible implementation, the target computing module is any module in a neural network model.

[0079] In one possible implementation, when the target computing module is the input layer of the neural network model, the data to be calculated is the original input data of the neural network model; when the target computing module is the intermediate layer of the neural network model, the data to be calculated is the calculation result output by the previous layer of the intermediate layer.

[0080] In a possible implementation manner, the original input data is one or more of picture data, video data, audio data, and text data.

[0081] In a possible implementation manner, the second length is a multiple of the first length.

[0082] In a possible implementation, the first length is 8 bits, and the second length is 16 bits.

[0083] In a possible implementation, based on the pre-configured first data loading method information, the data to be calculated is loaded into the target calculation module, including: loading the high 8 bits of data in every 16 bits of data in the data to be calculated into the target calculation module; loading the low 8 bits of data in every 16 bits of data in the data to be calculated into the target calculation module.

[0084] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a compilation device for a neural network model. Fig.11 The figure is a structural block diagram of a compiling device for a neural network model according to an embodiment of the present application, and the device may include:

[0085] File module 1101, used to obtain a definition file of a neural network model to be compiled;

[0086] A compiling module 1102, used for calling a compiler to compile the definition file into an executable file;

[0087] Among them, during the compilation process, the first data loading mode information is configured to the target computing module included in the neural network model, and the data length requirement corresponding to the target computing module includes a first length; the first data loading mode information is obtained by converting the second data loading mode information matching the second length according to the conversion relationship between the first length and the second length; the second length is higher than the first length.

[0088] According to the embodiments of the present application, the data to be calculated in the dynamic shape model can be rearranged while loading the data to be calculated, so as to extract the high and low bits of each value in the data to be calculated respectively, and then the extraction result can be used in the module after the target calculation module in the neural network model. Based on the data rearrangement implemented in the present application, the neural network model does not need to occupy additional computing resources when running, thereby improving the performance of the neural network model when running.

[0089] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0090] Fig.12 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Fig.12 As shown, the electronic device includes: a memory 1201 and a processor 1202. The memory 1201 stores a computer program that can be run on the processor 1202. When the processor 1202 executes the computer program, the method in the above embodiment is implemented. The number of the memory 1201 and the processor 1202 can be one or more.

[0091] The electronic device also includes:

[0092] The communication interface 1203 is used to communicate with external devices and perform data exchange transmission.

[0093] If the memory 1201, the processor 1202 and the communication interface 1203 are implemented independently, the memory 1201, the processor 1202 and the communication interface 1203 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0094] Optionally, in a specific implementation, if the memory 1201, the processor 1202 and the communication interface 1203 are integrated on a chip, the memory 1201, the processor 1202 and the communication interface 1203 can communicate with each other through an internal interface.

[0095] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0096] An embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0097] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0098] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0099] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0100] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0101] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0102] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0103] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0104] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0105] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.

[0106] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0107] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0108] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for loading data of a neural network model, wherein: The neural network model includes a target calculation module, the data length requirement corresponding to the target calculation module includes a first length, and the method includes: Acquire data to be calculated to be loaded into the target calculation module; wherein the data length of the data to be calculated is a second length, and the second length is greater than the first length; Based on the pre-configured first data loading method information, the data to be calculated is loaded into the target computing module; wherein the first data loading method information is obtained by converting the second data loading method information matching the second length according to the conversion relationship between the first length and the second length.

2. The method according to claim 1, wherein: The target computing module is any module in the neural network model.

3. The method according to claim 2, wherein: In the case where the target computing module is the input layer of the neural network model, the data to be calculated is the original input data of the neural network model; When the target computing module is an intermediate layer of the neural network model, the data to be calculated is a calculation result output by an upper layer of the intermediate layer.

4. The method according to claim 3, wherein: The original input data is one or more of picture data, video data, audio data, and text data.

5. The method according to claim 1, wherein: The second length is a multiple of the first length.

6. The method according to claim 5, wherein: The first length is 8 bits, and the second length is 16 bits.

7. The method according to claim 6, wherein: Based on the pre-configured first data loading mode information, the data to be calculated is loaded into the target calculation module, including: Loading high 8 bits of data in every 16 bits of data to be calculated into the target calculation module; The lower 8 bits of data in every 16 bits of data in the data to be calculated are loaded into the target calculation module.

8. A method for compiling a neural network model, comprising: Get the definition file of the neural network model to be compiled; Invoke a compiler to compile the definition file into an executable file; Among them, during the compilation process, the first data loading mode information is configured to the target computing module included in the neural network model, and the data length requirement corresponding to the target computing module includes a first length; the first data loading mode information is obtained by converting the second data loading mode information matching the second length according to the conversion relationship between the first length and the second length; the second length is higher than the first length.

9. A data loading device for a neural network model, wherein: The neural network model includes a target calculation module, the data length requirement corresponding to the target calculation module includes a first length, and the device includes: A data module, used for acquiring data to be calculated to be loaded into the target calculation module; wherein the data length of the data to be calculated is a second length, and the second length is greater than the first length; A loading module is used to load the data to be calculated into the target calculation module based on pre-configured first data loading method information; wherein the first data loading method information is obtained by converting the second data loading method information matching the second length according to the conversion relationship between the first length and the second length.

10. A compiling device for a neural network model, comprising: The file module is used to obtain the definition file of the neural network model to be compiled; A compiling module, used for calling a compiler to compile the definition file into an executable file; Among them, during the compilation process, the first data loading mode information is configured to the target computing module included in the neural network model, and the data length requirement corresponding to the target computing module includes a first length; the first data loading mode information is obtained by converting the second data loading mode information matching the second length according to the conversion relationship between the first length and the second length; the second length is higher than the first length.

11. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

12. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

13. A computer program product, wherein: The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images