Method, device, equipment, program and product for compiling neural network model

By performing data rearrangement and convolution kernel transformation on convolution layers that do not meet the optimal number of channels, the problem of waste of computing power caused by insufficient number of channels in the neural network model is solved, and the operation efficiency of neural network accelerator is improved.

CN119026641APending Publication Date: 2024-11-26NANJING HOUMO TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202310586761.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-23
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

There are convolutional layers with input and output channels less than 64 in some neural network models, resulting in wasting computing power of neural network accelerators and unable to make full use of the performance of hardware accelerators.

Method used

By determining the convolution layer that does not meet the optimal number of channels, data rearrangement and convolution kernel transformation are performed based on the original convolution parameters and preset step size values, the target feature map is generated until the optimal channel supported by the neural network accelerator is met. and then generate the target neural network model.

Benefits of technology

Without increasing hardware costs, improve the operating efficiency of the neural network model on the neural network accelerator and make full use of the accelerator's computing power.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119026641A_ABST
    Figure CN119026641A_ABST
Patent Text Reader

Abstract

The invention discloses a method, a device, equipment, a program and a product for compiling a neural network model. The method comprises the following steps: determining a first convolutional layer in a neural network model to be compiled; determining a first target convolution parameter based on a first original convolution parameter of the first convolution layer and a preset step value; converting the convolution operation of the first input feature map of the first convolution layer into the convolution operation of the target feature map to obtain a second convolution layer; wherein the convolution operation of the target feature map uses a first target convolution parameter, and the second convolution layer accords with the optimal channel number supported by the neural network accelerator; and generating a target neural network model through compiling processing based on network layers except the first convolutional layer in the to-be-compiled neural network model and the second convolutional layer. In the embodiment of the invention, the computing power of the neural network accelerator can be fully utilized, so that the operation efficiency of the neural network model on the neural network accelerator can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, program and product for compiling a neural network model. Background Art

[0002] A neural network accelerator is an artificial intelligence (AI) hardware acceleration unit, and a neural network accelerator is usually designed according to certain specifications.

[0003] In order to fully utilize the computing power of neural network accelerators, the input and output of the network layers in the neural network model often need to meet specific specifications. For example, some neural network accelerators require that the number of input and output channels of the network layer be a multiple of 64 to achieve the highest execution efficiency. However, some neural network models have convolutional layers with input and output channels less than 64, which will result in a huge waste of computing power. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a method, apparatus, device, program and product for compiling a neural network model.

[0005] According to one aspect of an embodiment of the present disclosure, a method for compiling a neural network model is provided, comprising:

[0006] Determine a first convolutional layer in the neural network model to be compiled; wherein the first convolutional layer does not meet the optimal number of channels supported by the neural network accelerator;

[0007] Determine a first target convolution parameter based on a first original convolution parameter of the first convolution layer and a preset step value;

[0008] The convolution operation of the first input feature map of the first convolution layer is transformed into a convolution operation of a target feature map to obtain a second convolution layer; wherein the target feature map is obtained by rearranging data of the first input feature map, the number of channels of the target feature map is greater than that of the first input feature map, the convolution operation of the target feature map uses the first target convolution parameter, and the second convolution layer meets the optimal number of channels supported by the neural network accelerator;

[0009] Based on the network layers other than the first convolutional layer in the neural network model to be compiled, and the second convolutional layer, a target neural network model is generated through compilation processing.

[0010] According to another aspect of an embodiment of the present disclosure, there is provided an apparatus for compiling a neural network model, comprising:

[0011] A first determination module is used to determine a first convolutional layer in the neural network model to be compiled; wherein the first convolutional layer does not meet the optimal number of channels supported by the neural network accelerator;

[0012] A second determination module, configured to determine a first target convolution parameter based on a first original convolution parameter of the first convolution layer and a preset step value;

[0013] A transformation module, configured to transform a convolution operation of a first input feature map of the first convolutional layer into a convolution operation of a target feature map, so as to obtain a second convolutional layer; wherein the target feature map is obtained by rearranging data of the first input feature map, the number of channels of the target feature map is greater than that of the first input feature map, the convolution operation of the target feature map uses the first target convolution parameter, and the second convolutional layer meets the optimal number of channels supported by the neural network accelerator;

[0014] A generation module is used to generate a target neural network model through compilation processing based on the network layers other than the first convolutional layer in the neural network model to be compiled, and the second convolutional layer.

[0015] According to another aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0016] A memory for storing a computer program product;

[0017] The processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the method for compiling a neural network model is implemented.

[0018] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, characterized in that when the computer program instructions are executed by a processor, the above-mentioned method for compiling a neural network model is implemented.

[0019] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including computer program instructions, which implement the above-mentioned method for compiling a neural network model when executed by a processor.

[0020] In an embodiment of the present disclosure, during the compilation of the neural network model, the convolution operation of the first input feature map of the first convolution layer can be transformed into the convolution operation of the target feature map based on the first original convolution parameter of the first convolution layer and the preset step value. Since the second convolution layer obtained by the operation transformation meets the optimal number of channels supported by the neural network accelerator, the computing power of the neural network accelerator can be more fully utilized during the actual operation of the compiled neural network model, thereby improving the operating efficiency of the neural network model on the neural network accelerator without increasing the hardware cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0022] Figure 1 It is one of the flowcharts of the method for compiling a neural network model provided by an embodiment of the present disclosure.

[0023] Figure 2-1 It is one of the schematic diagrams of the first input feature map in the embodiment of the present disclosure.

[0024] Figure 2-2 It is one of the schematic diagrams of the target feature map in the embodiment of the present disclosure.

[0025] Figure 3 This is the second flowchart of the method for compiling a neural network model provided by an embodiment of the present disclosure.

[0026] Figure 4-1 This is the second schematic diagram of the first input feature map in the embodiment of the present disclosure.

[0027] Figure 4-2 It is one of the schematic diagrams of the first original convolution kernel in the embodiment of the present disclosure.

[0028] Figure 4-3 Schematic diagram of a first original convolution kernel with first edge filling in an embodiment of the present disclosure.

[0029] Figure 4-4 It is one of the schematic diagrams of the first target convolution kernel in the embodiment of the present disclosure.

[0030] Figure 4-5 It is a schematic diagram of a target feature map in an embodiment of the present disclosure.

[0031] Figure 4-6This is the second schematic diagram of the first target convolution kernel in the embodiment of the present disclosure.

[0032] Figure 4-7 This is the third schematic diagram of the first target convolution kernel in the embodiment of the present disclosure.

[0033] Figure 4-8 This is the fourth schematic diagram of the first target convolution kernel in the embodiment of the present disclosure.

[0034] Figure 4-9 This is the second schematic diagram of the first original convolution kernel in the embodiment of the present disclosure.

[0035] Figure 4-10 This is the fifth schematic diagram of the first target convolution kernel in the embodiment of the present disclosure.

[0036] Figure 4-11 This is the sixth schematic diagram of the first target convolution kernel in the embodiment of the present disclosure.

[0037] Figure 4-12 This is the seventh schematic diagram of the first target convolution kernel in the embodiment of the present disclosure.

[0038] Figure 4-13 This is the eighth schematic diagram of the first target convolution kernel in the embodiment of the present disclosure.

[0039] Figure 5 It is one of the structural schematic diagrams of the device for compiling a neural network model provided by an embodiment of the present disclosure.

[0040] Figure 6 This is the second structural diagram of the device for compiling a neural network model provided by an embodiment of the present disclosure.

[0041] Figure 7 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0043] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0044] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and do not represent specific technical meanings or necessarily indicate a logical order.

[0045] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0046] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0047] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0048] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0049] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0050] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0051] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0052] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0053] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0054] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0055] Exemplary Methods

[0056] Figure 1 It is a flowchart of a method for compiling a neural network model provided by an exemplary embodiment of the present disclosure. Figure 1 The method shown may include step 110, step 120, step 130 and step 140, each of which is described below.

[0057] Step 110, determining the first convolutional layer in the neural network model to be compiled; wherein the first convolutional layer does not meet the optimal number of channels supported by the neural network accelerator.

[0058] Optionally, the optimal number of channels supported by the neural network accelerator may refer to an integer multiple of 64, so that the optimal number of channels supported by the neural network accelerator may include 64, 128, 256, etc.

[0059] It should be noted that the neural network model to be compiled refers to a neural network model that needs to be compiled. The neural network model to be compiled may include multiple network layers, including but not limited to convolution (Convolution, Conv) layers, pooling (Pool, Pool) layers, rectified linear unit (Rectified Linear Unit, ReLU) layers, etc.

[0060] If at least one of the number of input channels and the number of output channels of a convolutional layer among the multiple network layers included in the neural network model to be compiled is less than 64, it can be determined that the convolutional layer does not meet the optimal number of channels supported by the neural network acceleration. In this case, the convolutional layer can be used as the first convolutional layer.

[0061] If the number of input channels and the number of output channels of a convolutional layer in the multiple network layers included in the neural network model to be compiled are both integer multiples of 64, it can be determined that the convolutional layer meets the optimal number of channels supported by the neural network accelerator, and then the convolutional layer can be not used as the first convolutional layer.

[0062] In an example, the neural network model to be compiled can be yolov3 (which is a commonly used network for target detection). Yolov3 belongs to a fully convolutional network. The number of input channels of the first network layer of yolov3 is 3, and the number of output channels is 32. Obviously, 3 and 32 are both less than 64, so the first network layer of yolov3 can be used as the first convolutional layer.

[0063] Step 120: Determine a first target convolution parameter based on the first original convolution parameter of the first convolution layer and a preset step value.

[0064] Optionally, the first original convolution parameter of the first convolution layer refers to a parameter required for the first convolution layer to implement a convolution operation, and the first original convolution parameter includes but is not limited to a convolution kernel, a stride value, a padding value, and the like.

[0065] Optionally, the preset step value may be 1.

[0066] In step 120, the first target convolution parameter can be determined with reference to the first original convolution parameter and the preset step value, and the first target convolution parameter can refer to the parameter required for the second convolution layer to implement the convolution operation. It should be noted that there are various ways to determine the first target convolution parameter. For the sake of clarity, examples are given below.

[0067] Step 130, transforming the convolution operation of the first input feature map of the first convolutional layer into the convolution operation of the target feature map to obtain a second convolutional layer; wherein the target feature map is obtained by rearranging data of the first input feature map, the number of channels of the target feature map is greater than that of the first input feature map, the convolution operation of the target feature map uses the first target convolution parameter, and the second convolutional layer meets the optimal number of channels supported by the neural network accelerator.

[0068] It should be noted that the target feature map can be used as the input feature map of the second convolutional layer, and the convolution result of the target feature map can be used as the output feature map of the second convolutional layer. In this way, the number of input channels of the second convolutional layer is the same as the number of channels of the target feature map, and the number of output channels of the second convolutional layer is the same as the number of channels of the convolution result of the target feature map.

[0069] When rearranging the data of the first input feature map, the width and height can be reduced and the number of channels can be increased to obtain a target feature map with a larger number of channels than the first input feature map. This is conducive to making the number of input channels and the number of output channels of the second convolutional layer both integer multiples of 64, so that the second convolutional layer meets the optimal number of channels supported by the neural network accelerator.

[0070] It should be noted that in step 130, the convolution operation of the first input feature map is transformed into the convolution operation of the target feature map. In essence, the convolution operation of the first input feature map is transformed into an equivalent operation. By executing the equivalent operation, the operation result obtained can be equivalent to the convolution result obtained by executing the convolution operation of the first input feature map. For example, the operation result and the convolution result can be the same. For another example, although the operation result and the convolution result are different, the elements included in the operation result are the same as the elements included in the convolution result. The convolution result can be obtained by performing specific processing on the operation result.

[0071] Step 140, based on the network layers other than the first convolutional layer and the second convolutional layer in the neural network model to be compiled, generate a target neural network model through compilation processing.

[0072] In step 140, the compiler backend may perform compilation processing based on the network layers other than the first convolutional layer in the neural network model to be compiled, and the second convolutional layer obtained by computational transformation, so as to generate a binary target neural network model. The specific compilation processing method may be any feasible method according to actual needs, which is not elaborated in the present disclosure.

[0073] In an embodiment of the present disclosure, during the compilation of the neural network model, the convolution operation of the first input feature map of the first convolution layer can be transformed into the convolution operation of the target feature map based on the first original convolution parameter of the first convolution layer and the preset step value. Since the second convolution layer obtained by the operation transformation meets the optimal number of channels supported by the neural network accelerator, the computing power of the neural network accelerator can be more fully utilized during the actual operation of the compiled neural network model, thereby improving the operating efficiency of the neural network model on the neural network accelerator without increasing the hardware cost.

[0074] In an optional example, the target feature map includes: a first feature map, a second feature map, a third feature map, and a fourth feature map arranged in sequence along the channel direction; wherein,

[0075] The number of channels of the first feature map, the second feature map, the third feature map, and the fourth feature map is the same as the number of channels of the first input feature map;

[0076] The first feature map includes: each element in the first input feature map located in odd rows and odd columns;

[0077] The second feature map includes: each element in the first input feature map located in odd rows and even columns;

[0078] The third feature map includes: each element in the first input feature map located in an even row and an odd column;

[0079] The fourth feature map includes: each element in the first input feature map located in an even row and an even column;

[0080] The relative positions of any two elements in any one of the first feature map, the second feature map, the third feature map, and the fourth feature map are consistent with the relative positions of the two elements in the first input feature map.

[0081] Assume that the first input feature map is Figure 2-1 As shown, you can Figure 2-1 Starting from the element in the first row and first column, extract elements at intervals of 1 pixel in the width and height directions, and then arrange all the extracted elements. When arranging, it is necessary to ensure that the relative positions of any two elements after arrangement among all the extracted elements are the same as those of the two elements in Figure 2-1 The relative positions of J1 and J3 are consistent (for details, see Figure 2-1 The relative position in Figure 2-2 ), thereby forming a first feature map.

[0082] In addition, you can Figure 2-1 Starting from the element in the first row and second column, extract elements at intervals of 1 pixel in the width and height directions, and then arrange all the extracted elements. When arranging, it is necessary to ensure that the relative positions of any two elements after arrangement among all the extracted elements are the same as those of the two elements in Figure 2-1 The relative positions in are consistent, thereby forming a second feature map.

[0083] In a similar manner, the third and fourth characteristic maps can also be obtained. The first characteristic map, the second characteristic map, the third characteristic map, and the fourth characteristic map are stacked along the channel direction to obtain Figure 2-2 The target feature map shown.

[0084] It should be noted that the shape of each feature map in the embodiments of the present disclosure can be expressed in the form of height*width*channel, which is represented by Figure 2-1 , Figure 2-2 It can be seen that the shape of the first input feature map can be expressed as H*W*3c, and the shape of the target feature map can be expressed as H / 2*W / 2*12c. That is, compared with the first input feature map, the width and height of the target feature map are both reduced by half, while the number of channels is increased by 4 times.

[0085] It should be noted that the target feature map can be considered as the result of folding the data of the first input feature map along the non-channel dimension (i.e., the dimension composed of the width dimension and the height dimension) and increasing the amount of data in the channel dimension. The target feature map can be regarded as the folded format of the first input feature map.

[0086] It should be noted that Figure 2-2 In the target feature map illustrated in , the first feature map, the second feature map, the third feature map, and the fourth feature map are arranged sequentially along the channel direction. During specific implementation, the relative positions of the first feature map, the second feature map, the third feature map, and the fourth feature map can be adjusted according to actual needs. For example, in the target feature map, the first feature map, the third feature map, the second feature map, and the fourth feature map can be arranged sequentially along the channel direction.

[0087] In the embodiments of the present disclosure, during the actual operation of the compiled neural network model, by extracting elements of the first input feature map according to specific extraction rules and then arranging the element extraction results according to specific arrangement rules, the data of the first input feature map can be rearranged efficiently and reliably, thereby obtaining a target feature map in a folded format that can be regarded as the first input feature map. Compared with the case of performing a convolution operation on the first input feature map, the amount of convolution operation on the target feature map is smaller and the computing power of the neural network accelerator is more fully utilized, thereby effectively improving the computing efficiency.

[0088] In an alternative example,

[0089] In the case where the width and height of the first input feature map are both odd numbers, the target feature map is obtained by performing a first edge padding on the first input feature map and then rearranging data of the first input feature map after performing the first edge padding on the first input feature map;

[0090] The first edge filling is used to fill the right edge and the bottom edge of the first input feature map with 1 pixel respectively, and the filled pixel values ​​are all 0.

[0091] Assumptions Figure 2-1 If the width and height of the first input feature map are not even numbers but odd numbers, then according to the data rearrangement method introduced above, the target feature map with a shape of H / 2*W / 2*12c cannot be obtained, and thus the computational transformation of the convolution operation of the first input feature map cannot be achieved.

[0092] In view of this, in an embodiment of the present disclosure, when the width and height of the first input feature map are both odd numbers, the right edge and the bottom edge of the first input feature map can be filled with 1 pixel respectively, and the value of each filled pixel is set to 0 to obtain the first input feature map filled with the first edge. The width and height of the first input feature map filled with the first edge are both even numbers. Subsequently, the data of the first input feature map filled with the first edge can be rearranged according to the data rearrangement method introduced above to normally obtain the target feature map in a folded format that can be regarded as the first input feature map.

[0093] exist Figure 1 Based on the embodiment shown, Figure 3 As shown, step 120 includes step 1202, step 1204 and step 1206.

[0094] Step 1202: Based on the first original convolution parameter of the first convolution layer, obtain a first original convolution kernel and a first original step value.

[0095] Optionally, the first original convolution parameters may include a convolution kernel, a step value, a padding value, etc. In step 1202, the convolution kernel may be extracted from the first original convolution parameters as the first original convolution kernel, and the step value may be extracted from the first original convolution parameters as the first original step value.

[0096] Step 1204: Generate a first target convolution kernel based on the first original convolution kernel and the first original step value.

[0097] Step 1206, determine first target convolution parameters; wherein the first target convolution parameters include: a first target convolution kernel and a preset step value.

[0098] In a specific implementation, step 1204 includes:

[0099] When the first original step value is 2 and the convolution kernel size of the first original convolution kernel is an odd number, performing a first edge padding on the first original convolution kernel;

[0100] Rearranging data of the first input feature map after the first edge padding to generate a first target convolution kernel;

[0101] Among them, the first edge filling is used to fill the right edge and the bottom edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0. The preset step value is 1. According to the preset step value, the convolution result obtained by convolving the target feature map with the first target convolution kernel is used as the output feature map of the second convolution layer.

[0102] Here, the convolution kernel size of the first original convolution kernel being an odd number may mean that: the width and height of the first original convolution kernel are both odd numbers.

[0103] When the first original step value is 2 and the convolution kernel size of the first original convolution kernel is an odd number, the lower edge and the right edge of the first original convolution kernel can be padded with 1 pixel respectively, and each padded pixel value is set to 0 to obtain the first original convolution kernel padded with the first edge, and the width and height of the first original convolution kernel padded with the first edge are both even numbers. Next, the data rearrangement method introduced above can be used to rearrange the data of the first original convolution kernel padded with the first edge to generate a first target convolution kernel that can be regarded as a folded format of the first original convolution kernel padded with the first edge.

[0104] Since the target feature map can be regarded as the folded format of the first input feature map, the target feature map and the first target convolution kernel of the same folded format can perform convolution operations normally. In addition, since the first original step value is 2, the target feature map and the first target convolution kernel are both obtained based on element interval extraction, and the convolution operation of the target feature map and the first target convolution kernel uses a preset step value of 1. The convolution result of the target feature map and the first target convolution kernel, and the convolution result of the first input feature map and the first original convolution kernel are the same, which can ensure that the output feature map of the first convolution layer before the operational transformation and the output feature map of the second convolution layer after the operational transformation are the same. Therefore, the operational transformation at the first convolution layer will not affect other network layers in the neural network model to be compiled.

[0105] In one example, the first input feature map is Figure 4-1 The single-channel feature map shown in Figure 1 has the first original convolution kernel as Figure 4-2 The single-channel convolution kernel shown in FIG. 1 has a first original step value of 2. The first edge padding is first performed on the first original convolution kernel. The first original convolution kernel after the first edge padding can be referred to as Figure 4-3 , and then rearrange the data of the first input feature map filled with the first edge to generate a first target convolution kernel. The first target convolution kernel can be found in Figure 4-4 , the first target convolution kernel has 4 channels. In addition, the first input feature map can be rearranged to generate a target feature map. The target feature map can be found in Figure 4-5 , the target feature map can have 4 channels.

[0106] If the first original convolution kernel is convolved with the 3*3 area in the upper left corner of the first input feature map, the result y1=1×1+1×2+0×3+1×9+0×10+(-1)×11+0×17+(-1)×18+0×19=-17.

[0107] If the first target convolution kernel is convolved with the 2*2 area in the upper left corner of the target feature map, the result z1=[1×1+0×3+0×17+0×10]+[1×2+0×4+(-1)×18+0×20]+[1×9+(-1)×11+0×25+0×27]+[0×10+0×12+0×26+0×28]=-17.

[0108] Obviously, y1 and z1 are the same, which proves that the convolution result of the target feature map and the first target convolution kernel, and the convolution result of the first input feature map and the first original convolution kernel are the same. In addition, it can be seen that the 3×3 convolution in the 3×3 area before the operation transformation at the first convolution layer is converted into a 2×2 convolution on the 2×2 area.

[0109] In another specific implementation, step 1204 includes:

[0110] When the first original step value is 2 and the convolution kernel size of the first original convolution kernel is an even number, data of the first original convolution kernel is rearranged to generate a first target convolution kernel;

[0111] Among them, the preset step value is 1. According to the preset step value, the convolution result obtained by convolving the target feature map with the first target convolution kernel is used as the output feature map of the second convolution layer.

[0112] Here, the convolution kernel size of the first original convolution kernel being an even number may mean that: the width and height of the first original convolution kernel are both even numbers.

[0113] When the first original step value is 2 and the convolution kernel size of the first original convolution kernel is an even number, since the width and height of the first original convolution kernel are both even numbers, the data rearrangement method described above can be directly used to rearrange the data of the first original convolution kernel to generate a first target convolution kernel that can be regarded as a folded format of the first original convolution kernel. Based on the generated first target convolution kernel, referring to the relevant introduction in the second implementation of step 1204, the generation of the output feature map of the second convolution layer can be realized.

[0114] In another specific implementation, step 1204 includes:

[0115] When the first original step value is 1 and the convolution kernel size of the first original convolution kernel is an odd number, the first original convolution kernel is respectively subjected to first edge padding, second edge padding, third edge padding, and fourth edge padding;

[0116] Rearranging data of the first original convolution kernel filled with the first edge, the first original convolution kernel filled with the second edge, the first original convolution kernel filled with the third edge, and the first original convolution kernel filled with the fourth edge, respectively, to generate four first target convolution kernels;

[0117] Among them, the first edge filling is used to fill the right edge and the bottom edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, the second edge filling is used to fill the left edge and the bottom edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, the third edge filling is used to fill the right edge and the upper edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, the fourth edge filling is used to fill the left edge and the upper edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, and the preset step value is 1. According to the preset step value, the four convolution results obtained by convolving the target feature map with the four first target convolution kernels are stacked along the channel direction to obtain the stacked result as the output feature map of the second convolution layer.

[0118] When the first original step value is 1 and the convolution kernel size of the first original convolution kernel is an odd number, the right edge and the bottom edge of the first original convolution kernel can be filled with 1 pixel respectively, and each filled pixel value is set to 0 to obtain the first original convolution kernel filled with the first edge, and the width and height of the first original convolution kernel filled with the first edge are both even numbers. In a similar manner, the first original convolution kernel filled with the second edge, the first original convolution kernel filled with the third edge, and the first original convolution kernel filled with the fourth edge can be obtained. Next, the data rearrangement method introduced above can be used to rearrange the data of the first original convolution kernel with the first edge filling, the first original convolution kernel with the second edge filling, the first original convolution kernel with the third edge filling, and the first original convolution kernel with the fourth edge filling, so as to generate a first target convolution kernel in a folded format that can be regarded as the first original convolution kernel with the first edge filling, a first target convolution kernel in a folded format that can be regarded as the first original convolution kernel with the second edge filling, a first target convolution kernel in a folded format that can be regarded as the first original convolution kernel with the third edge filling, and a first target convolution kernel in a folded format that can be regarded as the first original convolution kernel with the fourth edge filling, thereby obtaining a total of four first target convolution kernels.

[0119] Since the target feature map can be regarded as a folded format of the first input feature map, the target feature map and each first target convolution kernel of the same folded format can perform convolution operations normally. In addition, since the first original step value is 1, the target feature map and the first target convolution kernel are both obtained based on the interval extraction of elements, and the convolution operation of the target feature map and the first target convolution kernel uses a preset step value of 1. By stacking the target feature map and the four convolution results of the four first target convolution kernels along the channel direction, the stacked result can be used as the output feature map of the second convolution layer. The output feature map of the second convolution layer can be regarded as the result obtained by rearranging the convolution result of the first input feature map and the first original convolution parameter using the data rearrangement method introduced above, that is, the output feature map of the second convolution layer can be regarded as the folded format of the convolution result of the first input feature map and the first original convolution parameter.

[0120] In one example, the first input feature map is Figure 4-1 As shown, the first original convolution kernel is Figure 4-2 As shown, the first original step value is 1, then by performing the first edge filling, the second edge filling, the third edge filling, and the fourth edge filling on the first original convolution kernel, and then combining the data rearrangement of the four first original convolution kernels after edge filling, four first target convolution kernels can be generated. These four first target convolution kernels can be referred to in turn. Figure 4-4 , Figure 4-6 , Figure 4-7 and Figure 4-8 In addition, by rearranging the data of the first input feature map, the generated target feature map can be seen in Figure 4-5 .

[0121] If the first original convolution kernel is convolved with the 3*3 area in the upper left corner of the first input feature map, the result is y1 = -17. If the first original convolution kernel is convolved with the area after sliding one pixel to the right along the 3*3 area in the upper left corner of the first input feature map, the result is y2 = 1×2 + 1×3 + 0×4 + 1×10 + 0×11 + (-1) ×12 + 0×18 + (-1) ×19 + 0×20 = -16. If the first original convolution kernel is convolved with the area after sliding one pixel downward along the 3*3 area in the upper left corner of the first input feature map, the result is y3 = 1×9 + 1×10 + 0×11 + 1×17 + 0×18 + (-1) ×19 + 0×25 + (-1) ×26 + 0×27 = -9. If the first original convolution kernel is convolved with the area after sliding one pixel to the right and downward along the 3*3 area along the upper left corner in the first input feature map, the result y4=1×10+1×11+0×12+1×18+0×19+(-1)×20+0×26+(-1)×27+0×28=-8.

[0122] If Figure 4-4 The first target convolution kernel in is convolved with the 2*2 area in the upper left corner of the target feature map, and the result z1=-17. Figure 4-6 The first target convolution kernel in is convolved with the 2*2 area in the upper left corner of the target feature map, and the result is z2=[0×1+1×3+0×17+(-1)×19]+[1×2+0×4+0×18+0×20]+[0×9+0×11+0×25+0×27]+[1×10+(-1)×12+0×26+0×28]=-16. Figure 4-7 The first target convolution kernel in is convolved with the 2*2 area in the upper left corner of the target feature map, and the result is z3=[0×1+0×3+1×17+(-1)×19]+[0×2+0×4+0×18+0×20]+[1×9+0×11+0×25+0×27]+[1×10+0×12+(-1)×26+0×28]=-9. Figure 4-8 The first target convolution kernel in is convolved with the 2*2 area in the upper left corner of the target feature map, and the result z4=[0×1+0×3+0×17+0×19]+[0×2+0×4+1×18+(-1)×20]+[0×9+1×11+0×25+(-1)×27]+[1×10+0×12+0×26+0×28]=-8.

[0123] Obviously, y1 is the same as z1, y2 is the same as z2, y3 is the same as z3, and y4 is the same as z4. The only difference is that y1, y2, y3, and y4 are located in the same channel of the convolution result of the first input feature map and the first original convolution kernel, while z1, z2, z3, and z4 are arranged in sequence in different channels of the output feature map of the second convolution layer. This proves that the output feature map of the second convolution layer can be regarded as a folded format of the convolution result of the first input feature map and the first original convolution parameters. In addition, it can be seen that the four 3×3 convolutions in the 4×4 area before the operation transformation at the first convolution layer (each convolution uses 9 points out of the 16 points in the 4×4 area) are converted to one 2×2 convolution on the 2×2 area.

[0124] In another specific implementation, step 1204 includes:

[0125] When the first original step value is 1 and the convolution kernel size of the first original convolution kernel is an even number, the first original convolution kernel is respectively subjected to fifth edge padding, sixth edge padding, seventh edge padding, and eighth edge padding;

[0126] Rearranging data of the first original convolution kernel padded with the fifth edge, the first original convolution kernel padded with the sixth edge, the first original convolution kernel padded with the seventh edge, and the first original convolution kernel padded with the eighth edge, respectively, to generate four first target convolution kernels;

[0127] Among them, the fifth edge filling is used to fill the right edge and the lower edge of the first original convolution kernel with 2 pixels respectively, and the filled pixel values ​​are all 0, the sixth edge filling is used to fill the left edge and the right edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, and fill the lower edge of the first original convolution kernel with 2 pixels and the filled pixel values ​​are all 0, the seventh edge filling is used to fill the upper edge and the lower edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, and fill the right edge of the first original convolution kernel with 2 pixels and the filled pixel values ​​are all 0, the eighth edge filling is used to fill the upper edge, lower edge, left edge, and right edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, and the preset step value is 1. According to the preset step value, the four convolution results obtained by convolving the target feature map with the four first target convolution kernels are stacked along the channel direction to obtain the stacked result as the output feature map of the second convolution layer.

[0128] When the first original step value is 1 and the convolution kernel size of the first original convolution kernel is an even number, the right edge and the bottom edge of the first original convolution kernel can be padded with 2 pixels respectively, and the padded pixel values ​​are all set to 0 to obtain the first original convolution kernel padded with the fifth edge; the left edge and the right edge of the first original convolution kernel can be padded with 1 pixel respectively, and the padded pixel values ​​are all set to 0, and the bottom edge of the first original convolution kernel can be padded with 2 pixels, and the padded pixel values ​​are all set to 0 to obtain the first original convolution kernel padded with the sixth edge; similarly, the first original convolution kernel padded with the seventh edge and the first original convolution kernel padded with the eighth edge can also be obtained. Next, the data rearrangement method introduced above can be used to rearrange the data of the first original convolution kernel filled with the fifth edge, the first original convolution kernel filled with the sixth edge, the first original convolution kernel filled with the seventh edge, and the first original convolution kernel filled with the eighth edge, respectively, to generate a first target convolution kernel that can be regarded as a folded format of the first original convolution kernel filled with the fifth edge, a first target convolution kernel that can be regarded as a folded format of the first original convolution kernel filled with the sixth edge, a first target convolution kernel that can be regarded as a folded format of the first original convolution kernel filled with the seventh edge, and a first target convolution kernel that can be regarded as a folded format of the first original convolution kernel filled with the eighth edge, thereby obtaining a total of four first target convolution kernels. Based on the generated four first target convolution kernels, referring to the relevant introduction in the third implementation of step 1204, the generation of the output feature map of the second convolution layer can be realized.

[0129] In one example, the first input feature map is Figure 4-1 As shown, the first original convolution kernel is Figure 4-9 As shown, by performing the fifth edge filling, the sixth edge filling, the seventh edge filling, and the eighth edge filling on the first original convolution kernel, and then combining the data rearrangement of the four first original convolution kernels after edge filling, four first target convolution kernels can be generated. These four first target convolution kernels can be referred to in turn. Figure 4-10 , Figure 4-11 , Figure 4-12 and Figure 4-13 In addition, by rearranging the data of the first input feature map, the generated target feature map can be seen in Figure 4-5 .

[0130] If the first original convolution kernel is convolved with the 2*2 area in the upper left corner of the first input feature map, the result y5=1×1+1×2+1×9+0×10=12. If the first original convolution kernel is convolved with the area after sliding one pixel to the right along the 2*2 area in the upper left corner of the first input feature map, the result y6=1×2+1×3+1×10+0×11=15. If the first original convolution kernel is convolved with the area after sliding one pixel downward along the 2*2 area in the upper left corner of the first input feature map, the result y7=1×9+1×10+1×17+0×18=36. If the first original convolution kernel is convolved with the area after sliding one pixel to the right and downward along the 2*2 area in the upper left corner of the input feature map, the result y8=1×10+1×11+1×18+0×19=39.

[0131] As shown in the figure Figure 4-10 The first target convolution kernel in is convolved with the 2*2 area in the upper left corner of the target feature map, and the result z5=[1×1+0×3+0×17+0×19]+[1×2+0×4+0×18+0×20]+[1×9+0×11+0×25+0×27]+[0×10+0×12+0×26+0×28]=12. Figure 4-11 The first target convolution kernel in is convolved with the 2*2 area in the upper left corner of the target feature map, and the result z6=[0×1+1×3+0×17+0×19]+[1×2+0×4+0×18+0×20]+[0×9+0×11+0×25+0×27]+[1×10+0×12+0×26+0×28]=15. Figure 4-12The first target convolution kernel in is convolved with the 2*2 area in the upper left corner of the target feature map, and the result z7=[0×1+0×3+1×17+0×19]+[0×2+0×4+0×18+0×20]+[1×9+0×11+0×25+0×27]+[1×10+0×12+0×26+0×28]=36. Figure 4-13 The first target convolution kernel in is convolved with the 2*2 area in the upper left corner of the target feature map, and the result z8=[0×1+0×3+0×17+0×19]+[0×2+0×4+1×18+0×20]+[0×9+1×11+0×25+0×27]+[1×10+0×12+0×26+0×28]=39.

[0132] Obviously, y5 is the same as z5, y6 is the same as z6, y7 is the same as z7, and y8 is the same as z8. The only difference is that y5, y6, y7, and y8 are located in the same channel of the convolution result of the first input feature map and the first original convolution kernel, while z5, z6, z7, and z8 are located in different channels of the output feature map of the second convolution layer. This proves that the output feature map of the second convolution layer can be regarded as the folded format of the convolution result of the first input feature map and the first original convolution kernel. In addition, it can be seen that the four 2×2 convolutions in the 3×3 area before the operation transformation at the first convolution layer are converted into one 2×2 convolution on the 2×2 area.

[0133] In an embodiment of the present disclosure, with reference to the first original convolution kernel and the first original step value obtained based on the first original convolution parameters of the first convolution layer, a suitable method can be used to generate a first target convolution kernel so as to obtain the corresponding first target convolution parameters, thereby ensuring the equivalence between the convolution operation of the first input feature map and the convolution operation of the target feature map when the first target convolution parameters are used to perform the convolution operation of the target feature map.

[0134] In an optional example, based on the network layers other than the first convolutional layer and the second convolutional layer in the neural network model to be compiled, generating a target neural network model through a compilation process includes:

[0135] In the case where the first original step value is 1, and there is a third convolution layer that meets the preset conditions in the network layers other than the first convolution layer in the neural network model to be compiled, the second target convolution parameters are determined based on the second original convolution parameters of the third convolution layer and the preset step value; wherein the third convolution layer meets the preset conditions means that the second original step value in the second original convolution parameters is 2, and one of the following two items is satisfied: the third convolution layer uses the output result of the first convolution layer; the third convolution layer is located after the first convolution layer and there are other network layers between the third convolution layer and the first convolution layer, and the other network layers maintain the data arrangement and dimension value of the output result of the first convolution layer;

[0136] The convolution operation of the original second input feature map of the third convolutional layer is transformed into the convolution operation of the new second input feature map of the third convolutional layer to obtain a fourth convolutional layer; wherein the original second input feature map and the new second input feature map refer to the input feature map of the third convolutional layer before and after the convolution operation of the first input feature map is transformed, respectively, and the convolution operation of the new second input feature map uses the second target convolution parameter;

[0137] Based on the network layers except the first convolution layer and the third convolution layer in the neural network model to be compiled, as well as the second convolution layer and the fourth convolution layer, a target neural network model is generated through compilation processing.

[0138] In an embodiment of the present disclosure, when the first original step value is 1, it can be determined whether the next network layer of the first convolutional layer in the neural network model to be compiled (i.e., the network layer using the output result of the first convolutional layer) is a convolutional layer.

[0139] If the next network layer is a convolutional layer, it can be determined that the next network layer meets the preset condition, and the next network layer is used as the third convolutional layer.

[0140] If the next network layer is not a convolutional layer, it can be determined whether the next network layer can maintain the data arrangement and dimensionality of the output result of the first convolutional layer, and whether the next network layer of the next network layer (which can be called the next network layer) is a convolutional layer. If the next network layer is a ReLU layer, and the next network layer is a convolutional layer, since the ReLU layer only assigns negative values ​​in the data it processes to 0 and keeps other values ​​unchanged, and does not change the dimensionality of the data it processes, it can be determined that the next network layer meets the preset conditions, and the next network layer is used as the third convolutional layer. Optionally, when the next network layer is a leaky ReLU (which is a variant of the ReLU operation) layer and the next network layer is a convolutional layer, the next network layer can be used as the third convolutional layer.

[0141] Next, the second target convolution kernel can be determined based on the second original convolution parameters used by the third convolution layer and the preset step value. It should be noted that the parameters included in the second original convolution parameters can refer to the introduction of the parameters included in the first original convolution parameters. Based on the second original convolution parameters and the preset step value, the method for determining the second target convolution kernel can refer to the method for determining the first target convolution kernel introduced above, which will not be repeated here.

[0142] Afterwards, the convolution operation of the original second input feature map of the third convolutional layer can be transformed into the convolution operation of the new second input feature map of the third convolutional layer to obtain the fourth convolutional layer. Assuming that the third convolutional layer is the next network layer of the first convolutional layer, the original second input feature map is the output feature map of the first convolutional layer, and the new second input feature map is the output feature map of the second convolutional layer.

[0143] Afterwards, the compiler backend can perform compilation processing based on the network layers except the first convolution layer and the third convolution layer in the neural network model to be compiled, as well as the second convolution layer and the fourth convolution layer, so as to generate a binary target neural network model. The specific compilation processing method can adopt any feasible method according to actual needs, and this disclosure will not go into details.

[0144] Assuming that the third convolutional layer is the next network layer of the first convolutional layer, the number of channels of the second original convolution kernel in the second original convolution parameters of the third convolutional layer should match the number of output channels of the first convolutional layer. From the latter specific implementation of step 1204 above, it can be seen that when the first original step value is 1, the output feature map of the second convolutional layer (i.e., the new second input feature map) can be regarded as a folded format of the convolution result of the first input feature map and the first original convolution parameters, and the number of channels of the new second input feature map is 4 times the number of output channels of the first convolutional layer. Then, the number of channels of the new second input feature map is obviously not matched with the number of channels of the second original convolution kernel, and the new second input feature map and the second original convolution kernel cannot perform convolution operations normally.

[0145] In view of this, in the embodiments of the present disclosure, the convolution operation of the original second input feature map of the third convolution layer can be transformed into the convolution operation of the new second input feature map, and the convolution operation of the new second input feature map uses the second target convolution parameters. Since the second target convolution parameters include the second target convolution kernel, the second target convolution kernel can be regarded as the folded format of the first target convolution kernel, and the new second input feature map and the second target convolution kernel of the same folded format can perform convolution operations normally. In this way, when the first original step value is 1, it is only necessary to perform the operation transformation at the first convolution layer, and then perform the corresponding operation transformation at the third convolution layer that meets the preset conditions, so that the convolution operation at the third convolution layer can be realized by an equivalent operation, and the network layer after the third convolution layer can perform operations based on its own input feature map normally.

[0146] In an optional example, the neural network model to be compiled includes three network layers, Conv1, ReLU and Conv2. The input channel of Conv1 is 3, the output channel is 32, and the input feature map of Conv1 is Input. Then:

[0147] (1) If the stride used by Conv1 is 2, the data of Input can be rearranged, the data of the convolution kernel used by Conv1 can be rearranged, and the operation transformation can be performed at Conv1;

[0148] (2) If the stride used by Conv1 is 1 and the stride used by Conv2 is 2, the input data can be rearranged, the convolution kernel used by Conv1 can be rearranged, and the convolution kernel used by Conv2 can be transformed. In addition, the transformation can be performed at Conv1 and Conv2 respectively.

[0149] In an optional example, if the first original step value is 1, and there is no third convolutional layer that meets the preset conditions in the neural network model to be compiled, then the operation transformation may not be performed at the first convolutional layer, and compilation processing may be performed directly based on all network layers included in the neural network model to be compiled to obtain a compiled neural network model.

[0150] In an optional example, if the first original step value is 1, and there is no third convolutional layer that meets the preset conditions in the neural network model to be compiled, the operation transformation can be performed normally at the first convolutional layer, and, in the actual operation of the compiled neural network model, after the second convolutional layer obtains the corresponding output feature map through operation processing, the output feature map of the second convolutional layer can be unfolded in a manner reverse to the data rearrangement method described above (the unfolding result is the convolution result of the first input feature map and the first original convolution parameter), and then the feature map obtained after unfolding is provided to the next network layer of the second convolutional layer for processing. Optionally, the unfolding of the output feature map can be accomplished by memory transfer.

[0151] In summary, in the embodiments of the present disclosure, by rearranging the data of the input feature map and the data of the convolution kernel, it is possible to ensure that the number of input and output channels of the convolution layer are both integer multiples of 64, so that the computing power of the neural network accelerator can be more fully utilized, thereby improving the operating efficiency of the neural network model on the neural network accelerator. For example, by adopting the embodiments of the present disclosure to increase the number of output channels of the first network layer of yolov3 from 32 to 64, the operating efficiency of the neural network accelerator for the first network layer of yolov3 can be increased to twice the original operating efficiency. For another example, by adopting the embodiments of the present disclosure to increase the number of output channels of the first network layer of another neural network model from 16 to 64, the operating efficiency of the neural network accelerator for the first network layer of another neural network model can be increased to four times the original operating efficiency.

[0152] Any method for compiling a neural network model provided in the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any method for compiling a neural network model provided in the embodiments of the present disclosure may be executed by a processor, such as the processor executing any method for compiling a neural network model mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0153] Exemplary Devices

[0154] Figure 5 is a schematic diagram of a structure of an apparatus for compiling a neural network model provided by an exemplary embodiment of the present disclosure, Figure 5 The device shown includes a first determination module 510 , a second determination module 520 , a transformation module 530 and a generation module 540 .

[0155] A first determination module 510 is used to determine a first convolutional layer in the neural network model to be compiled; wherein the first convolutional layer does not meet the optimal number of channels supported by the neural network accelerator;

[0156] A second determination module 520, configured to determine a first target convolution parameter based on a first original convolution parameter of the first convolution layer and a preset step value;

[0157] A transformation module 530 is used to transform the convolution operation of the first input feature map of the first convolution layer into a convolution operation of the target feature map to obtain a second convolution layer; wherein the target feature map is obtained by rearranging data of the first input feature map, the number of channels of the target feature map is greater than that of the first input feature map, the convolution operation of the target feature map uses the first target convolution parameter, and the second convolution layer meets the optimal number of channels supported by the neural network accelerator;

[0158] The generation module 540 is used to generate a target neural network model through compilation processing based on the network layers other than the first convolutional layer and the second convolutional layer in the neural network model to be compiled.

[0159] In an optional example, the target feature map includes: a first feature map, a second feature map, a third feature map, and a fourth feature map arranged in sequence along the channel direction; wherein,

[0160] The number of channels of the first feature map, the second feature map, the third feature map, and the fourth feature map is the same as the number of channels of the first input feature map;

[0161] The first feature map includes: each element in the first input feature map located in odd rows and odd columns;

[0162] The second feature map includes: each element in the first input feature map located in odd rows and even columns;

[0163] The third feature map includes: each element in the first input feature map located in an even row and an odd column;

[0164] The fourth feature map includes: each element in the first input feature map located in an even row and an even column;

[0165] The relative positions of any two elements in any one of the first feature map, the second feature map, the third feature map, and the fourth feature map are consistent with the relative positions of the two elements in the first input feature map.

[0166] In an alternative example,

[0167] When the width and height of the first input feature map are both odd numbers, the target feature map is obtained by performing a first edge padding on the first input feature map and then rearranging data of the first input feature map after performing the first edge padding on the first input feature map;

[0168] The first edge filling is used to fill the right edge and the bottom edge of the first input feature map with 1 pixel respectively, and the filled pixel values ​​are all 0.

[0169] In an alternative example, Figure 6 As shown, the second determination module 520 includes:

[0170] An acquisition submodule 5202 is used to acquire a first original convolution kernel and a first original step value based on a first original convolution parameter of the first convolution layer;

[0171] A first generating submodule 5204, configured to generate a first target convolution kernel based on the first original convolution kernel and the first original step value;

[0172] The first determination submodule 5206 is used to determine first target convolution parameters; wherein the first target convolution parameters include: a first target convolution kernel and a preset step value.

[0173] In an optional example, the first generating submodule 5204 includes:

[0174] A first edge filling unit, used for performing first edge filling on the first original convolution kernel when the first original step value is 2 and the convolution kernel size of the first original convolution kernel is an odd number;

[0175] A first generating unit, configured to rearrange data of the first original convolution kernel filled with the first edge to generate a first target convolution kernel;

[0176] Among them, the first edge filling is used to fill the right edge and the bottom edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0. The preset step value is 1. According to the preset step value, the convolution result obtained by convolving the target feature map with the first target convolution kernel is used as the output feature map of the second convolution layer.

[0177] In an optional example, the first generating submodule 5204 includes:

[0178] a second generating unit, configured to rearrange data of the first original convolution kernel to generate a first target convolution kernel when the first original step value is 2 and the convolution kernel size of the first original convolution kernel is an even number;

[0179] Among them, the preset step value is 1. According to the preset step value, the convolution result obtained by convolving the target feature map with the first target convolution kernel is used as the output feature map of the second convolution layer.

[0180] In an optional example, the first generating submodule 5204 includes:

[0181] A second edge filling unit is used to perform first edge filling, second edge filling, third edge filling, and fourth edge filling on the first original convolution kernel respectively when the first original step value is 1 and the convolution kernel size of the first original convolution kernel is an odd number;

[0182] a third generating unit, configured to rearrange data of the first original convolution kernel filled with the first edge, the first original convolution kernel filled with the second edge, the first original convolution kernel filled with the third edge, and the first original convolution kernel filled with the fourth edge, respectively, to generate four first target convolution kernels;

[0183] Among them, the first edge filling is used to fill the right edge and the bottom edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, the second edge filling is used to fill the left edge and the bottom edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, the third edge filling is used to fill the right edge and the upper edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, the fourth edge filling is used to fill the left edge and the upper edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, and the preset step value is 1. According to the preset step value, the four convolution results obtained by convolving the target feature map with the four first target convolution kernels are stacked along the channel direction to obtain the stacked result as the output feature map of the second convolution layer.

[0184] In an optional example, the first generating submodule 5204 includes:

[0185] A third edge filling unit is used for performing fifth edge filling, sixth edge filling, seventh edge filling, and eighth edge filling on the first original convolution kernel respectively when the first original step value is 1 and the convolution kernel size of the first original convolution kernel is an even number;

[0186] a fourth generating unit, configured to rearrange data of the first original convolution kernel filled with the fifth edge, the first original convolution kernel filled with the sixth edge, the first original convolution kernel filled with the seventh edge, and the first original convolution kernel filled with the eighth edge, respectively, to generate four first target convolution kernels;

[0187] Among them, the fifth edge filling is used to fill the right edge and the lower edge of the first original convolution kernel with 2 pixels respectively, and the filled pixel values ​​are all 0, the sixth edge filling is used to fill the left edge and the right edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, and fill the lower edge of the first original convolution kernel with 2 pixels and the filled pixel values ​​are all 0, the seventh edge filling is used to fill the upper edge and the lower edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, and fill the right edge of the first original convolution kernel with 2 pixels and the filled pixel values ​​are all 0, the eighth edge filling is used to fill the upper edge, lower edge, left edge, and right edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, and the preset step value is 1. According to the preset step value, the four convolution results obtained by convolving the target feature map with the four first target convolution kernels are stacked along the channel direction to obtain the stacked result as the output feature map of the second convolution layer.

[0188] In an optional example, the generating module 540 includes:

[0189] The second determination submodule is used to determine the second target convolution parameter based on the second original convolution parameter of the third convolution layer and the preset step value when the first original step value is 1 and there is a third convolution layer that meets the preset condition in the network layer other than the first convolution layer in the neural network model to be compiled; wherein the third convolution layer meets the preset condition means that the second original step value in the second original convolution parameter is 2, and one of the following two items is satisfied: the third convolution layer uses the output result of the first convolution layer; the third convolution layer is located after the first convolution layer and there are other network layers between the third convolution layer and the first convolution layer, and the other network layers maintain the data arrangement and dimension value of the output result of the first convolution layer;

[0190] A transformation submodule, used to transform the convolution operation of the original second input feature map of the third convolutional layer into the convolution operation of the new second input feature map of the third convolutional layer, so as to obtain a fourth convolutional layer; wherein the original second input feature map and the new second input feature map refer to the input feature map of the third convolutional layer before and after the convolution operation of the first input feature map is transformed, respectively, and the convolution operation of the new second input feature map uses the second target convolution parameter;

[0191] The second generation submodule is used to generate a target neural network model through compilation processing based on the network layers other than the first convolution layer and the third convolution layer in the neural network model to be compiled, as well as the second convolution layer and the fourth convolution layer.

[0192] Exemplary Electronic Devices

[0193] Below, reference Figure 7The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0194] Figure 7 A block diagram of an electronic device 700 according to an embodiment of the present disclosure is illustrated.

[0195] like Figure 7 As shown, the electronic device 700 includes one or more processors 701 and a memory 702 .

[0196] The processor 701 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 700 to perform desired functions.

[0197] The memory 702 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 701 may run the program instructions to implement the methods for compiling neural network models and / or other desired functions of the various embodiments of the present disclosure described above. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0198] In one example, the electronic device 700 may further include: an input device 703 and an output device 704, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0199] For example, when the electronic device 700 is the first device or the second device, the input device 703 may be a microphone or a microphone array. When the electronic device 700 is a stand-alone device, the input device 703 may be a communication network connector for receiving collected input signals from the first device and the second device.

[0200] In addition, the input device 703 may also include, for example, a keyboard, a mouse, and the like.

[0201] The output device 704 can output various information to the outside, and can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto.

[0202] Of course, to simplify, Figure 7 Only some of the components related to the present disclosure in the electronic device 700 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 700 may also include any other appropriate components.

[0203] Exemplary computer program products and computer-readable storage media

[0204] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the method for compiling a neural network model according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0205] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0206] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps in the method for compiling a neural network model according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0207] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0208] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0209] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0210] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0211] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0212] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0213] The above description of the disclosed aspects is provided to enable those skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0214] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for compiling a neural network model, characterized in that: include: Determine a first convolutional layer in the neural network model to be compiled; wherein the first convolutional layer does not meet the optimal number of channels supported by the neural network accelerator; Determine a first target convolution parameter based on a first original convolution parameter of the first convolution layer and a preset step value; The convolution operation of the first input feature map of the first convolution layer is transformed into a convolution operation of a target feature map to obtain a second convolution layer; wherein the target feature map is obtained by rearranging data of the first input feature map, the number of channels of the target feature map is greater than that of the first input feature map, the convolution operation of the target feature map uses the first target convolution parameter, and the second convolution layer meets the optimal number of channels supported by the neural network accelerator; Based on the network layers other than the first convolutional layer in the neural network model to be compiled, and the second convolutional layer, a target neural network model is generated through compilation processing.

2. The method according to claim 1, characterized in that The target feature map includes: a first feature map, a second feature map, a third feature map, and a fourth feature map arranged in sequence along the channel direction; wherein, The number of channels of each of the first feature map, the second feature map, the third feature map, and the fourth feature map is the same as the number of channels of the first input feature map; The first feature map includes: each element in the first input feature map located in odd rows and odd columns; The second feature map includes: each element in the first input feature map located in odd rows and even columns; The third feature map includes: each element in the first input feature map located in an even row and an odd column; The fourth feature map includes: each element in the first input feature map located in an even row and an even column; The relative positions of any two elements in any one of the first feature map, the second feature map, the third feature map, and the fourth feature map are consistent with the relative positions of the two elements in the first input feature map.

3. The method according to claim 1, characterized in that In a case where both the width and the height of the first input feature map are odd numbers, the target feature map is obtained by performing a first edge padding on the first input feature map and then rearranging data of the first input feature map after the first edge padding; The first edge filling is used to fill the right edge and the bottom edge of the first input feature map with 1 pixel respectively, and the filled pixel values ​​are all 0.

4. The method according to claim 1, characterized in that: The determining of the first target convolution parameter based on the first original convolution parameter of the first convolution layer and a preset step size includes: Based on the first original convolution parameter of the first convolution layer, obtaining a first original convolution kernel and a first original step value; Generate a first target convolution kernel based on the first original convolution kernel and the first original step value; Determine a first target convolution parameter; wherein the first target convolution parameter includes: the first target convolution kernel and the preset step value.

5. The method according to claim 4, characterized in that The step of generating a first target convolution kernel based on the first original convolution kernel and the first original step value includes: When the first original step value is 2 and the convolution kernel size of the first original convolution kernel is an odd number, performing a first edge padding on the first original convolution kernel; Rearranging data of the first original convolution kernel filled with the first edge to generate a first target convolution kernel; Among them, the first edge filling is used to fill the right edge and the bottom edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, the preset step value is 1, and according to the preset step value, the convolution result obtained by convolving the target feature map with the first target convolution kernel is used as the output feature map of the second convolution layer.

6. The method according to claim 4, characterized in that The step of generating a first target convolution kernel based on the first original convolution kernel and the first original step value includes: When the first original step value is 2 and the convolution kernel size of the first original convolution kernel is an even number, data rearrangement is performed on the first original convolution kernel to generate a first target convolution kernel; Among them, the preset step value is 1, and according to the preset step value, the convolution result obtained by performing a convolution operation on the target feature map and the first target convolution kernel is used as the output feature map of the second convolution layer.

7. The method according to claim 4, characterized in that The step of generating a first target convolution kernel based on the first original convolution kernel and the first original step value includes: When the first original step value is 1 and the convolution kernel size of the first original convolution kernel is an odd number, performing first edge padding, second edge padding, third edge padding, and fourth edge padding on the first original convolution kernel respectively; Rearranging data of the first original convolution kernel filled with the first edge, the first original convolution kernel filled with the second edge, the first original convolution kernel filled with the third edge, and the first original convolution kernel filled with the fourth edge, respectively, to generate four first target convolution kernels; Among them, the first edge filling is used to fill the right edge and the bottom edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, the second edge filling is used to fill the left edge and the bottom edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, the third edge filling is used to fill the right edge and the upper edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, the fourth edge filling is used to fill the left edge and the upper edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, and the preset step value is 1. According to the preset step value, the four convolution results obtained by convolving the target feature map with the four first target convolution kernels are stacked along the channel direction to obtain the stacked result as the output feature map of the second convolution layer.

8. The method according to claim 4, characterized in that The step of generating a first target convolution kernel based on the first original convolution kernel and the first original step value includes: When the first original step value is 1 and the convolution kernel size of the first original convolution kernel is an even number, performing fifth edge padding, sixth edge padding, seventh edge padding, and eighth edge padding on the first original convolution kernel respectively; Rearranging data of the first original convolution kernel filled with the fifth edge, the first original convolution kernel filled with the sixth edge, the first original convolution kernel filled with the seventh edge, and the first original convolution kernel filled with the eighth edge, respectively, to generate four first target convolution kernels; Among them, the fifth edge filling is used to fill the right edge and the lower edge of the first original convolution kernel with 2 pixels respectively, and the filled pixel values ​​are all 0, the sixth edge filling is used to fill the left edge and the right edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, and fill the lower edge of the first original convolution kernel with 2 pixels and the filled pixel values ​​are all 0, the seventh edge filling is used to fill the upper edge and the lower edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, and fill the right edge of the first original convolution kernel with 2 pixels and the filled pixel values ​​are all 0, the eighth edge filling is used to fill the upper edge, lower edge, left edge, and right edge of the first original convolution kernel with 1 pixel respectively, and the filled pixel values ​​are all 0, and the preset step value is 1. According to the preset step value, the four convolution results obtained by convolving the target feature map with the four first target convolution kernels are stacked along the channel direction to obtain the stacked result as the output feature map of the second convolution layer.

9. The method according to claim 7 or 8, characterized in that: The generating a target neural network model through compiling the network layers other than the first convolutional layer in the neural network model to be compiled and the second convolutional layer, comprises: In the case where the first original step value is 1, and there is a third convolution layer that meets the preset conditions in the network layers other than the first convolution layer in the neural network model to be compiled, based on the second original convolution parameter of the third convolution layer and the preset step value, the second target convolution parameter is determined; wherein the third convolution layer meets the preset condition means that the second original step value in the second original convolution parameter is 2, and one of the following two items is satisfied: the third convolution layer uses the output result of the first convolution layer; the third convolution layer is located after the first convolution layer and there are other network layers between the third convolution layer and the first convolution layer, and the other network layers maintain the data arrangement and dimension value of the output result of the first convolution layer; The convolution operation of the original second input feature map of the third convolutional layer is transformed into the convolution operation of the new second input feature map of the third convolutional layer to obtain a fourth convolutional layer; wherein the original second input feature map and the new second input feature map refer to the input feature map of the third convolutional layer before and after the convolution operation of the first input feature map is transformed, respectively, and the convolution operation of the new second input feature map uses the second target convolution parameter; Based on the network layers other than the first convolution layer and the third convolution layer in the neural network model to be compiled, as well as the second convolution layer and the fourth convolution layer, a target neural network model is generated through compilation processing.

10. A device for compiling a neural network model, characterized in that: include: A first determination module is used to determine a first convolutional layer in the neural network model to be compiled; wherein the first convolutional layer does not meet the optimal number of channels supported by the neural network accelerator; A second determination module, configured to determine a first target convolution parameter based on a first original convolution parameter of the first convolution layer and a preset step value; A transformation module, configured to transform a convolution operation of a first input feature map of the first convolutional layer into a convolution operation of a target feature map, so as to obtain a second convolutional layer; wherein the target feature map is obtained by rearranging data of the first input feature map, the number of channels of the target feature map is greater than that of the first input feature map, the convolution operation of the target feature map uses the first target convolution parameter, and the second convolutional layer meets the optimal number of channels supported by the neural network accelerator; A generation module is used to generate a target neural network model through compilation processing based on the network layers other than the first convolutional layer in the neural network model to be compiled, and the second convolutional layer.

11. An electronic device, characterized in that: include: A memory for storing a computer program product; A processor, configured to execute a computer program product stored in the memory, and when the computer program product is executed, the method for compiling a neural network model as described in any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method for compiling a neural network model described in any one of claims 1 to 9 is implemented.

13. A computer program product comprising computer program instructions, characterized in that When the computer program instructions are executed by a processor, the method for compiling a neural network model described in any one of claims 1 to 9 is implemented.

Citation Information

Patent Citations

  • Target detection method and device, electronic device and target detection model

    CN109670517A

  • ISAR image classification method based on spatial transformation three-channel convolution

    CN110969212A

  • Neural network processing method and device, electronic equipment and storage medium

    CN111680781A

  • Methods and systems for optimal transport of non-linear transformations

    US20210383589A1