Operator elimination method, system, electronic device and storage medium for neural network

Abstract

The present invention provides an operator elimination method, system, electronic device and storage medium for a neural network, and relates to the technical field of deep learning. It includes: determining a target subgraph structure with operator elimination requirements in a neural network model; determining an operator in an eliminateable state in the target subgraph structure as a target elimination operator; traversing all subgraph structures until all target subgraph structures with operator elimination requirements and corresponding target elimination operators are determined; calculating the buffer memory sum of the input and output of the target elimination operator; determining the target cache area of ​​the target elimination operator according to the buffer memory sum, so as to allocate the target elimination operator to the target cache area, and configuring the output of all target elimination operators with the same cache address; determining the jump offset of the target elimination operator; compiling the target elimination operator based on the jump offset to complete operator elimination. The present invention can realize operator elimination and reduce the reasoning time of the neural network.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning, and in particular to a method, system, electronic device and storage medium for eliminating operators of a neural network. Background Art

[0002] Operators in neural network learning are the basic computing units that constitute neural networks. Operators are responsible for performing various mathematical operations and processing on input data. With the rapid development of neural network learning, a large number of operator calculation operations will make the network structure of neural network learning more complex, and will also increase the memory and cache read and write requirements of the on-chip system. Therefore, by eliminating unnecessary operators, the amount of calculation, memory, and cache in neural network learning can be optimized.

[0003] Currently, we can generally eliminate reshaping operators (such as reshape / transpose / split operators) by modifying the shape information of the operator (the dimension or size information of the operator), or eliminate memory concatenation operators (such as the Concat operator) through special data movement logic to reduce the complexity of the algorithm. This can reduce the operator configuration, scheduling, and interaction time on the memory cache, thereby speeding up the running inference time.

[0004] However, the inventors found that the current operator elimination method cannot perform deep operator optimization based on the characteristics of the hardware due to the complex neural network structure. For example, the current operator elimination method cannot be combined with the model graph structure best memory allocation method (Best filt) to perform operator elimination, thus failing to achieve deep acceleration of the running reasoning time. Summary of the invention

[0005] The present invention provides a neural network operator elimination method, system, electronic device and storage medium, which are used to solve the problem that the current operator elimination method cannot perform deep operator optimization according to the characteristics of hardware.

[0006] According to one aspect of the present invention, the present invention provides an operator elimination method for a neural network, comprising: determining a target subgraph structure where operator elimination needs to be performed in a neural network model; determining an operator in an eliminating state in the target subgraph structure as a target elimination operator; traversing all subgraph structures until all target subgraph structures where operator elimination needs to be performed and corresponding target elimination operators are determined; before all inputs of the target elimination operator, calculating the buffer memory sum of the input and output of the target elimination operator; determining a target cache area of ​​the target elimination operator based on the buffer memory sum to allocate the target elimination operator to the target cache area, the target cache area at least including an on-chip cache area or a global memory cache area; configuring the outputs of all target elimination operators to the same cache address; determining a jump offset of the target elimination operator; and compiling the target elimination operator based on the jump offset to complete operator elimination.

[0007] According to some embodiments of the present invention, before all inputs of the target elimination operator, calculating the buffer memory sum of the input and output of the target elimination operator includes: sorting all the target elimination operators based on preset rules; calculating the buffer memory sum of the input and output of each target elimination operator in turn; wherein, in the case where the input and output of the target elimination operator share the same buffer, the input and output of the target elimination operator are calculated only once.

[0008] According to some embodiments of the present invention, determining a target cache area of ​​a target elimination operator based on a buffer memory sum to allocate the target elimination operator to the target cache area includes: judging the size of the buffer memory sum and the memory of an on-chip cache area; when the buffer memory sum is smaller than the memory size of the on-chip cache area, allocating the target elimination operator to the on-chip cache area; when the buffer memory sum is not smaller than the memory size of the on-chip cache area, allocating the target elimination operator to a global memory cache area.

[0009] According to some embodiments of the present invention, the target subgraph structure is a network topology structure subgraph.

[0010] According to another aspect of the present invention, the present invention provides an operator elimination system for a neural network, which includes an operator processing module, a memory processing module and an operator elimination module. The operator processing module determines a target subgraph structure with operator elimination requirements in a neural network model, determines the operators in the target subgraph structure in an erasable state as target elimination operators, and traverses all subgraph structures until all target subgraph structures with operator elimination requirements and corresponding target elimination operators are determined. The memory processing module calculates the buffer memory sum of the input and output of the target elimination operator before all inputs of the target elimination operator; determines the target cache area of ​​the target elimination operator according to the buffer memory sum, so as to allocate the target elimination operator to the target cache area, the target cache area at least including an on-chip cache area or a global memory cache area; and configures the outputs of all target elimination operators to the same cache address. The operator elimination module determines the jump offset of the target elimination operator; compiles the target elimination operator based on the jump offset to complete the operator elimination.

[0011] According to some embodiments of the present invention, a memory processing module sorts all target elimination operators based on preset rules; the memory processing module calculates the buffer memory sum of the input and output of each target elimination operator in turn; wherein, when the input and output of the target elimination operator share the same buffer, the memory processing module calculates the input and output of the target elimination operator only once.

[0012] According to some embodiments of the present invention, the memory processing module determines the size of the total memory of the buffer zone and the memory of the on-chip cache area; when the total memory of the buffer zone is smaller than the memory size of the on-chip cache area, the memory processing module allocates the target elimination operator to the on-chip cache area; when the total memory of the buffer zone is not smaller than the memory size of the on-chip cache area, the memory processing module allocates the target elimination operator to the global memory cache area.

[0013] According to some embodiments of the present invention, the target subgraph structure is a network topology structure subgraph.

[0014] According to another aspect of the present invention, the present invention further provides an electronic device. The electronic device includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors can implement the operator elimination method described above.

[0015] According to another aspect of the present invention, the present invention further provides a non-volatile computer-readable storage medium having a computer program stored thereon, which can implement the operator elimination method described above when executed by a processor.

[0016] The operator elimination method provided by the present invention determines the target elimination operator, calculates the buffer memory sum of the input and output of the target elimination operator, and can combine the optimal memory allocation method to preferentially allocate the target elimination operator to the on-chip cache area, and configures the output of the target elimination operator to the same cache address, and compiles the operator based on the jump offset, thereby realizing operator elimination.

[0017] The operator elimination method provided by the present invention has the characteristics of combining the best memory allocation method to perform memory allocation, and can preferentially execute the network structure in the on-chip cache area. In addition, the operator elimination method can realize the operator elimination of unnecessary operators, which can make the network structure clearer and simpler, and can significantly reduce the reasoning time of the neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram showing a flow chart of an operator elimination method according to an embodiment of the present invention;

[0020] Figure 2 Another schematic diagram showing a flow chart of the operator elimination method according to an embodiment of the present invention;

[0021] Figure 3 Another schematic diagram showing a flow chart of the operator elimination method according to an embodiment of the present invention;

[0022] Figure 4 A schematic diagram showing a jump offset according to an embodiment of the present invention;

[0023] Figure 5a A schematic diagram showing an existing operator calculation;

[0024] Figure 5b A schematic diagram showing operator elimination according to an embodiment of the present invention;

[0025] Figure 6 A schematic diagram of an operator elimination system according to an embodiment of the present invention is shown.

[0026] Description of reference numerals:

[0027] Operator elimination system 1; operator processing module 10; memory processing module 20; operator elimination module 30. DETAILED DESCRIPTION

[0028] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the present invention will be comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0029] The described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of these specific details, or other modes, components, materials, devices, etc. may be adopted. In these cases, known structures, methods, devices, implementations, materials or operations will not be shown or described in detail.

[0030] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0031] The terms "first", "second" and the like in the specification, claims and the above drawings of the present invention are used to distinguish different objects rather than to describe a specific order.

[0032] The following is a clear and complete description of the technical solution of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0033] The full names of the English abbreviations involved in the present invention and their Chinese meanings are:

[0034] OP: operator input;

[0035] buffer: buffer memory;

[0036] CPU: Central Processing Unit, central processing unit;

[0037] GPU: Graphics Processing Unit, graphics processor;

[0038] Bin: Binary, binary file;

[0039] output buffer: output buffer area.

[0040] Figure 1 A schematic flow chart of an operator elimination method according to an embodiment of the present invention is shown.

[0041] According to one aspect of the present invention, the present invention provides an operator elimination method based on a neural network. Figure 1 As shown, the operator elimination method may include steps S100-S800.

[0042] Exemplarily, the algorithm elimination method may be executed by an operator elimination system having computing capabilities.

[0043] According to an example embodiment, in step S100, the operator elimination system determines a target subgraph structure where operator elimination is required in a neural network model.

[0044] Optionally, the target subgraph structure may be a network topology subgraph. The network topology subgraph is a specific subgraph structure in a complex network.

[0045] Exemplarily, the operator elimination system responds to a user instruction and determines a network topology subgraph on which operator elimination can be performed in the graph structure of the neural network model.

[0046] In S200 , the operator elimination system determines an operator in an erasable state in a target subgraph structure as a target elimination operator.

[0047] For example, in response to user instructions, the operator elimination system marks the relevant operators (such as the memory concatenation operator Concat) in the network topology subgraph as an eliminateable state (or an optimizable state), and determines the operators marked as an eliminateable state (or an optimizable state) as target elimination operators to perform subsequent operator elimination operations on the target elimination operator.

[0048] Exemplarily, the operator elimination system responds to a user instruction and marks the memory concatenation operator and its OP (input of the operator) as being in an eliminateable state (or an optimizeable state).

[0049] In step S300 , the operator elimination system traverses all subgraph structures until all target subgraph structures with operator elimination requirements and corresponding target elimination operators are determined.

[0050] For example, the operator elimination system traverses all subgraph structures in the current neural network model to determine whether the current target subgraph structure is the last subgraph structure with a target elimination operator. If not, the operator elimination system loops through steps S100 to S300, so that the operator elimination system can determine all target subgraph structures with target elimination operators and all corresponding target elimination operators, so as to perform subsequent operator elimination operations on all target elimination operators.

[0051] In step S400 , the operator elimination system calculates the buffer memory sum of the input and output of the target elimination operator before all the inputs of the target elimination operator.

[0052] For example, the operator elimination system advances the calculation of the memory splicing operator to before all the inputs of the memory splicing operator, so that the size of the buffer (buffer memory) of all inputs and outputs related to the memory splicing operator can be calculated in advance, thereby obtaining the total buffer memory required for the input and output of the memory splicing operator.

[0053] Figure 2 Another schematic flow chart of the operator elimination method according to the embodiment of the present invention is shown.

[0054] Alternatively, if Figure 2 As shown, in step S400, step S400 may further include step S410 and step S420.

[0055] In step S410, the operator elimination system sorts all target elimination operators based on preset rules.

[0056] In step S420, the operator elimination system sequentially calculates the buffer memory sum of the input and output of each target elimination operator.

[0057] In the case where the input and output of the target elimination operator share the same buffer, the input and output of the target elimination operator are calculated only once.

[0058] For example, the preset rule may be a depth-first rule set according to user requirements. The operator elimination system sorts all target elimination operators based on the depth-first rule, and sequentially calculates the input and output buffer sizes of each target elimination operator based on the sorting order, thereby calculating the total buffer memory of the input and output of the target elimination operator.

[0059] In the present invention, when the input and output of the target elimination operator share the same buffer, the input and output of the target elimination operator are calculated only once. With such a setting, the present invention can save a certain amount of computing memory, and avoid multiple repeated allocation calculations of the target elimination operator, and can provide a computing basis for subsequent operators to share the cache.

[0060] In step S500, the operator elimination system determines a target buffer area of ​​a target elimination operator according to the buffer memory sum, so as to allocate the target elimination operator to the target buffer area. The target buffer area includes at least an on-chip buffer area or a global memory buffer area.

[0061] For example, the operator elimination system may allocate the target elimination operator to the on-chip cache area in priority according to the size of the total buffer memory, and when the size of the total buffer memory does not meet the requirements for allocation to the on-chip cache area, the operator elimination system allocates the target elimination operator to the global memory cache area.

[0062] For example, on-chip cache is an important component in modern processor design, used to improve data access speed and system performance. On-chip cache is usually located inside the processor chip and shares the same physical location as the CPU, so it can be accessed quickly, thereby reducing external bus activity and improving system performance. Global memory cache is a cache mechanism used to speed up access to global memory in computer model systems. Global memory cache can exist in different hardware architectures, such as GPU and CPU systems.

[0063] It can be understood here that the on-chip cache area is closest to the computing unit compared to the global memory cache area, and can perform efficient data reading and storage, and has relatively high operating efficiency.

[0064] Figure 3 Another schematic flow chart of the operator elimination method according to the embodiment of the present invention is shown.

[0065] Alternatively, if Figure 3 As shown, in step S500, step S500 may further include steps S510-S530.

[0066] In step S510 , the operator elimination system determines the size of the total buffer memory and the memory of the on-chip cache area.

[0067] In step S520 , the operator elimination system allocates the target elimination operator to the on-chip cache area when the total memory of the buffer area is smaller than the memory size of the on-chip cache area.

[0068] In step S530 , the operator elimination system allocates the target elimination operator to the global memory cache area when the total memory of the buffer area is not less than the size of the memory of the on-chip cache area.

[0069] For example, the operator elimination system determines the target buffer area of ​​the target elimination operator according to the size of the total buffer memory. If it is determined that the input operators of all memory splicing operators can be placed in the on-chip buffer area, the operator elimination system allocates all memory splicing operators to the on-chip buffer area; otherwise, the operator elimination system allocates all memory splicing operators to the global memory buffer area.

[0070] Through the above embodiment, the operator elimination system preferentially allocates the target elimination operator to the on-chip cache area according to the size of the total buffer memory of the target elimination operator. In this way, the present invention can realize the memory allocation of operators in combination with the optimal memory allocation method, and realize that the memory splicing operator and related operators can be executed on the cache area, which can save data transfer time, thereby accelerating the calculation and reasoning speed of the neural network model.

[0071] In step S600, the operator elimination system configures the outputs of all target elimination operators to the same cache address.

[0072] For example, the operator elimination system calculates the output memory of all memory splicing operators, and configures the same cache address for the output of all memory splicing operators that need to be spliced ​​based on the order after the graph adjustment.

[0073] In step S700 , the operator elimination system determines a jump offset of a target elimination operator.

[0074] In step S800 , the operator elimination system compiles the target elimination operator based on the jump offset to complete the operator elimination.

[0075] Figure 4 A schematic diagram showing a jump offset according to an embodiment of the present invention.

[0076] For example, Figure 4 As shown in the figure, for OP1 of memory splicing operator 1 and OP2 of memory splicing operator 2, the operator elimination system calculates the offset of the first input of each memory splicing operator in turn according to the order of the target elimination operator, and configures the jump offset (also known as the line break offset, corresponding to the jump address) of the target elimination operator after completing a splicing dimension based on the offset. After the operator is compiled, an executable Bin file is generated to complete the operator splicing fusion and realize operator elimination.

[0077] The following uses memory splicing operator 1 (56, 56, 128) and memory splicing operator 2 (56, 56, 256) as examples to provide an exemplary introduction to operator elimination.

[0078] First, the operator elimination system applies for a cache address memory addr0 of size 56*56*384. The first address position where the memory splicing operator 1 will be stored is addr1=addr0+offset(0); the first address position where the memory splicing operator 2 will be stored is addr2=addr0+offset(128).

[0079] Then, 128 data from memory splicing operator 1 are taken to fill addr1, then addr1=addr0+128; and 256 data from memory splicing operator 2 are taken to fill addr2, then addr2=addr0+128+256. After memory splicing operator 2 stores the 256 data, the next storage address of memory splicing operator 1 is updated to addr1=addr1+256, where 256 is the jump offset; and the next storage address of memory splicing operator 2 is updated to addr2=addr2+128.

[0080] By analogy, repeat the above operation 56*56 times to complete the splicing of memory splicing operator 1 (56, 56, 128) and memory splicing operator 2 (56, 56, 256), and obtain memory splicing operator 3 (56, 56, 384).

[0081] Figure 5a A schematic diagram showing an existing operator calculation; Figure 5b A schematic diagram showing operator elimination according to an embodiment of the present invention is shown.

[0082] For example, Figure 5a As shown, the input 1 (OP1) of the memory concatenation operator 1 and the input 2 (OP2) of the memory concatenation operator 2 pass through their respective output buffers to obtain the input 3 (OP3) of Chancel Concat (the memory concatenation operator 3 in the channel dimension). Then, the memory concatenation operator 3 passes through the output buffer to obtain the input 4 (OP4) of the memory concatenation operator 4.

[0083] And as Figure 5b As shown, input 1 (OP1) of memory splicing operator 1 and input 2 (OP2) of memory splicing operator 2 can directly pass through the output buffer to obtain input 4 (OP4) of memory splicing operator 4.

[0084] Compared with the calculation of the memory splicing operator in the prior art, the operator elimination method provided by the present invention can realize the operator elimination function, reduce the reading and writing process and related configuration of an operator, and thus reduce the reasoning time of the neural network model.

[0085] It can be understood here that the operator elimination method provided by the present invention can not only eliminate the memory splicing operator of the channel dimension, but also eliminate the memory splicing operator of any dimension.

[0086] Through the above embodiments, the operator elimination method provided by the present invention determines the target elimination operator, calculates the sum of the buffer memory of the input and output of the target elimination operator, and can combine the optimal memory allocation method to preferentially allocate the target elimination operator to the on-chip cache area, and configures the output of the target elimination operator to the same cache address, and compiles the operator based on the jump offset, thereby realizing operator elimination.

[0087] The operator elimination method provided by the present invention has the characteristics of combining the best memory allocation method to perform memory allocation, and can preferentially execute the network structure in the on-chip cache area. In addition, the operator elimination method can realize the operator elimination of unnecessary operators, which can make the network structure clearer and simpler, and can significantly reduce the reasoning time of the neural network model.

[0088] According to another aspect of the present invention, the present invention provides an operator elimination system for a neural network. Figure 6 FIG. 2 is a schematic diagram showing an operator elimination system according to an embodiment of the present invention. Figure 6 As shown, the operator elimination system 1 includes an operator processing module 10 , a memory processing module 20 and an operator elimination module 30 .

[0089] According to an example embodiment, the operator processing module 10 determines a target subgraph structure where operator elimination is required in the neural network model.

[0090] Optionally, the target subgraph structure may be a network topology subgraph. The network topology subgraph is a specific subgraph structure in a complex network.

[0091] Exemplarily, the operator processing module 10 responds to a user instruction and determines a network topology subgraph in the graph structure of the neural network model on which operators can be eliminated.

[0092] According to an example embodiment, the operator processing module 10 determines an operator in an erasable state in the target subgraph structure as a target elimination operator.

[0093] For example, the operator processing module 10 responds to user instructions, marks the relevant operators (such as the memory concatenation operator Concat) in the network topology subgraph as an eliminable state (or an optimizable state), and determines the operator marked as an eliminable state (or an optimizable state) as a target elimination operator to perform subsequent operator elimination operations on the target elimination operator.

[0094] Exemplarily, the operator processing module 10 responds to a user instruction and marks the memory concatenation operator and its OP (input of the operator) as being in an eliminable state (or an optimizable state).

[0095] According to an exemplary embodiment, the operator processing module 10 traverses all subgraph structures until all target subgraph structures with operator elimination requirements and corresponding target elimination operators are determined.

[0096] For example, the operator processing module 10 traverses all subgraph structures in the current neural network model to determine whether the current target subgraph structure is the last subgraph structure with a target elimination operator. If not, the operator processing module 10 loops through the above process, so that the operator processing module 10 can determine all target subgraph structures with target elimination operators and all corresponding target elimination operators, so as to perform subsequent operator elimination operations on all target elimination operators.

[0097] According to an example embodiment, the memory processing module 20 calculates the buffer memory sum of the inputs and outputs of the target elimination operator before all the inputs of the target elimination operator.

[0098] For example, the memory processing module 20 advances the calculation of the memory splicing operator to before all the inputs of the memory splicing operator, so that the size of the buffer (buffer memory) of all inputs and outputs related to the memory splicing operator can be calculated in advance, thereby obtaining the total buffer memory required for the input and output of the memory splicing operator.

[0099] Optionally, the memory processing module 20 sorts all target elimination operators based on a preset rule. The memory processing module 20 sequentially calculates the buffer memory sum of the input and output of each target elimination operator. In the case where the input and output of the target elimination operator share the same buffer, the memory processing module 20 only calculates the input and output of the target elimination operator once.

[0100] For example, the preset rule may be a depth-first rule set according to user requirements. The memory processing module 20 sorts all target elimination operators based on the depth-first rule and calculates the input and output buffer size of each target elimination operator, thereby calculating the total buffer memory of the input and output of the target elimination operator.

[0101] In the present invention, when the input and output of the target elimination operator share the same buffer, the input and output of the target elimination operator are calculated only once. With such a setting, the present invention can save a certain amount of computing memory, and avoid multiple repeated allocation calculations of the target elimination operator, and can provide a computing basis for subsequent operators to share the cache.

[0102] According to an exemplary embodiment, the memory processing module 20 determines the target cache area of ​​the target elimination operator according to the buffer memory sum to allocate the target elimination operator to the target cache area. The target cache area includes at least an on-chip cache area or a global memory cache area.

[0103] For example, the memory processing module 20 can preferentially allocate the target elimination operator to the on-chip cache area according to the size of the total buffer memory, and when the size of the total buffer memory does not meet the requirements for allocation to the on-chip cache area, the memory processing module 20 allocates the target elimination operator to the global memory cache area.

[0104] For example, on-chip cache is an important component in modern processor design, used to improve data access speed and system performance. On-chip cache is usually located inside the processor chip and shares the same physical location as the CPU, so it can be accessed quickly, thereby reducing external bus activity and improving system performance. Global memory cache is a cache mechanism used to speed up access to global memory in computer model systems. Global memory cache can exist in different hardware architectures, such as GPU and CPU systems.

[0105] It can be understood here that the on-chip cache area is closest to the computing unit compared to the global memory cache area, and can perform efficient data reading and storage, and has relatively high operating efficiency.

[0106] Optionally, the memory processing module 20 determines the size of the memory sum of the buffer area and the memory of the on-chip cache area. If the memory sum of the buffer area is less than the memory size of the on-chip cache area, the memory processing module 20 allocates the target elimination operator to the on-chip cache area; if the memory sum of the buffer area is not less than the memory size of the on-chip cache area, the memory processing module 20 allocates the target elimination operator to the global memory cache area.

[0107] For example, the memory processing module 20 determines the target buffer area of ​​the target elimination operator according to the size of the total buffer memory. When it is determined that the input operators of all memory splicing operators can be placed in the on-chip buffer area, the memory processing module 20 allocates all memory splicing operators to the on-chip buffer area; otherwise, the memory processing module 20 allocates all memory splicing operators to the global memory buffer area.

[0108] Through the above embodiment, the memory processing module 20 preferentially allocates the target elimination operator to the on-chip cache area according to the size of the total buffer memory of the target elimination operator. In this way, the present invention can realize the memory allocation of the operator in combination with the optimal memory allocation method, and realize that the memory splicing operator and related operators can be executed on the cache area, which can save data transfer time, thereby accelerating the calculation and reasoning speed of the neural network model.

[0109] According to an example embodiment, the memory processing module 20 configures the outputs of all target elimination operators to the same cache address.

[0110] For example, the memory processing module 20 calculates the output memory of all memory splicing operators, and configures the same cache address for the outputs of all memory splicing operators that need to be spliced ​​based on the order after the graph adjustment.

[0111] According to an example embodiment, the operator elimination module 30 determines a jump offset of a target elimination operator. The operator elimination module 30 compiles the target elimination operator based on the jump offset to complete the operator elimination.

[0112] For example, for OP1 of memory splicing operator 1 and OP2 of memory splicing operator 2, the operator elimination module 30 calculates the offset of the first input of each memory splicing operator in turn according to the order of the target elimination operator, and configures the jump offset (also called line break offset, corresponding to the jump address) of the target elimination operator after completing a splicing dimension based on the offset. After the operator is compiled, an executable Bin file is generated to complete the splicing fusion of the operator to achieve operator elimination.

[0113] Compared with the calculation of memory splicing operators in the prior art, the present invention can realize the operator elimination function, which can reduce the reading and writing process and related configuration of an operator at one time, thereby reducing the reasoning time of the neural network model.

[0114] It can be understood here that the present invention can not only eliminate the memory splicing operator of the channel dimension, but also eliminate the memory splicing operator of any dimension.

[0115] Through the above embodiments, the present invention determines the target elimination operator, calculates the sum of the buffer memory of the input and output of the target elimination operator, and can combine the optimal memory allocation method to preferentially allocate the target elimination operator to the on-chip cache area, and configures the output of the target elimination operator to the same cache address, and compiles the operator based on the jump offset, thereby realizing operator elimination.

[0116] The present invention has the characteristics of combining the best memory allocation method to perform memory allocation, and can preferentially execute the network structure in the on-chip cache area. The present invention can also eliminate operators of unnecessary operators, make the network structure clearer and simpler, and significantly reduce the reasoning time of the neural network model.

[0117] According to another aspect of the present invention, an electronic device is provided, which includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors can implement the operator elimination method described above.

[0118] According to another aspect of the present invention, a non-volatile computer-readable storage medium is provided, wherein a computer program is stored on the storage medium, and when the computer program is executed by a processor, the operator elimination method described above can be implemented.

[0119] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention is described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions of the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for eliminating operators of a neural network, characterized in that: include: Determine the target subgraph structure where operators need to be eliminated in the neural network model; Determining an operator of an erasable state in the target subgraph structure as a target elimination operator; Traverse all subgraph structures until all target subgraph structures with operator elimination requirements and the corresponding target elimination operators are determined; Before all inputs of the target elimination operator, calculate the buffer memory sum of the input and output of the target elimination operator; Determine a target buffer area of ​​the target elimination operator according to the buffer memory sum, so as to allocate the target elimination operator to the target buffer area, wherein the target buffer area at least includes an on-chip buffer area or a global memory buffer area; All outputs of the target elimination operators are configured with the same cache address; Determining a jump offset of the target elimination operator; The target elimination operator is compiled based on the jump offset to complete operator elimination.

2. The operator elimination method according to claim 1, characterized in that: The step of calculating the buffer memory sum of the input and output of the target elimination operator before all the inputs of the target elimination operator comprises: Sort all target elimination operators based on preset rules; Calculate the total buffer memory of the input and output of each target elimination operator in turn; Wherein, in the case where the input and output of the target elimination operator share the same buffer, the input and output of the target elimination operator are calculated only once.

3. The operator elimination method according to claim 1, characterized in that: Determining the target buffer area of ​​the target elimination operator according to the buffer memory sum to allocate the target elimination operator to the target buffer area includes: Determine the size of the total memory of the buffer and the memory of the on-chip cache area; In the case where the sum of the buffer memory is smaller than the size of the memory of the on-chip cache area, allocating the target elimination operator to the on-chip cache area; In a case where the sum of the buffer memory is not less than the size of the memory of the on-chip cache area, the target elimination operator is allocated to the global memory cache area.

4. The operator elimination method according to claim 1, characterized in that: The target subgraph structure is a network topology subgraph.

5. A neural network operator elimination system, characterized in that: include: An operator processing module determines in the neural network model a target subgraph structure where an operator elimination requirement exists, determines an operator in an eliminateable state in the target subgraph structure as a target elimination operator, and traverses all subgraph structures until all target subgraph structures where an operator elimination requirement exists and corresponding target elimination operators are determined; A memory processing module, which calculates the buffer memory sum of the input and output of the target elimination operator before all the inputs of the target elimination operator; Determine a target buffer area of ​​the target elimination operator according to the buffer memory sum, so as to allocate the target elimination operator to the target buffer area, wherein the target buffer area at least includes an on-chip buffer area or a global memory buffer area; and configuring the outputs of all the target elimination operators to the same cache address; The operator elimination module determines a jump offset of the target elimination operator; and compiles the target elimination operator based on the jump offset to complete the operator elimination.

6. The operator elimination system according to claim 5, characterized in that: The memory processing module sorts all target elimination operators based on preset rules; The memory processing module sequentially calculates the buffer memory sum of the input and output of each target elimination operator; Wherein, in the case where the input and output of the target elimination operator share the same buffer, the memory processing module calculates the input and output of the target elimination operator only once.

7. The operator elimination system according to claim 5, characterized in that: The memory processing module determines the size of the sum of the buffer memory and the memory of the on-chip cache area; The memory processing module allocates the target elimination operator to the on-chip cache area when the sum of the buffer memory is smaller than the size of the memory of the on-chip cache area; The memory processing module allocates the target elimination operator to the global memory cache area when the total memory of the buffer area is not less than the size of the memory of the on-chip cache area.

8. The operator elimination system according to claim 5, characterized in that: The target subgraph structure is a network topology subgraph.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the operator elimination method as described in any one of claims 1 to 4.

10. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the operator elimination method according to any one of claims 1 to 4 is implemented.

Images