Processing method and device of neural network model, equipment and medium

Abstract

The application relates to the technical field of convolution operation, and discloses a neural network model processing method and device, equipment and a medium. The neural network model processing method comprises the following steps: obtaining a neural network model to be processed, and determining all initial loop structures in the neural network model; traversing the initial loop structures, determining the loop times of each initial loop structure, and performing loop unrolling on the initial loop structures according to the loop times to obtain unrolled loop structures; in the case where the initial loop structures and the unrolled loop structures are determined to be different, updating the initial loop structures to the unrolled loop structures to obtain a processed neural network model. The above scheme can perform loop unrolling on the loop structures based on the loop times, thereby simplifying the structure of the neural network model, reducing the calculation amount and occupied memory space during neural model operation, and improving the operation efficiency of the neural network model.

Description

Technical Field

[0001] This application relates to the field of convolution operation technology, and in particular to a method, apparatus, device and medium for processing neural network models. Background Technology

[0002] In neural network models, loop nodes are nodes that differ from conventional matrix multiplication and convolution operations. Loop nodes can be conditional statements, loop statements, etc., and often need to make decisions based on the current state of the input data. Due to the existence of loop nodes, recursive function calls and repeated large loops may occur, making the execution order of some operators in the neural network model unpredictable. This increases the complexity of the neural network's computational graph, increases the memory usage of the neural network model, and may also cause unpredictable runtime delays during neural network operation, leading to memory overflow issues. Summary of the Invention

[0003] To address the issue that the presence of cyclic nodes increases the complexity of the computational graph of a neural network model and the memory it consumes, this application provides a method, apparatus, device, and medium for processing neural network models.

[0004] In a first aspect, embodiments of this application provide a method for processing a neural network model, applied to an electronic device. The method includes: acquiring a neural network model to be processed and determining all initial loop structures in the neural network model; traversing the initial loop structures, determining the number of loops for each initial loop structure, and performing loop expansion on the initial loop structures according to the number of loops to obtain an expanded loop structure; and updating the initial loop structure to the expanded loop structure when the initial loop structure and the expanded loop structure are different, thereby obtaining the processed neural network model.

[0005] In this embodiment, the initial loop structure in the neural network model whose loop count can be determined offline is determined before the neural network model runs, and the initial loop structure is expanded cyclically based on the loop count. This avoids determining the loop count of the initial loop structure with a fixed number of loops during the operation of the neural network model, thereby reducing the complexity of the computation graph during the operation of the neural network model and reducing the memory occupied by the neural network model.

[0006] In one possible implementation, determining the number of iterations of the initial loop structure includes: obtaining the actual parameters of the initial loop structure if all parameters of the conditional statements in the initial loop structure are constants; and inputting the actual parameters into the initial loop structure to determine the number of iterations of the initial loop structure.

[0007] In one possible implementation, a method for determining whether the parameters of the conditional statement of the initial loop structure are all constants includes: determining the expression of the conditional statement of the initial loop structure; determining the loop parameters of the expression, the loop parameters being used to determine the number of iterations of the initial loop structure; if the loop parameters can be determined offline, then the parameters of the conditional statement of the initial loop structure are all constants; if the loop parameters cannot be determined offline, then the parameters of the conditional statement of the initial loop structure include non-constant numbers.

[0008] In some embodiments, the loop parameter is used to determine the number of iterations of the initial loop structure and the final result value of the expanded loop structure.

[0009] In one possible implementation, inputting actual parameters into the initial loop structure to determine the number of iterations of the initial loop structure includes: configuring a loop counter; inputting actual parameters into the initial loop structure and determining the number of iterations of the initial loop structure using the loop counter.

[0010] In one possible implementation, traversing the initial loop structure includes: creating a list of loop structures to store the initial loop structures; and sequentially reading the initial loop structures from the list of loop structures until the list of loop structures is empty.

[0011] In one possible implementation, the method also includes creating an invalid unrolled structure list, which stores loop structures whose unrolled loop structures are identical to the initial loop structure, and removing initial loop structures from the loop structure list that are identical to the unrolled loop structures in the invalid unrolled structure list.

[0012] This application embodiment improves the processing efficiency of the neural network model by deleting the initial loop structure that is the same as the loop structure in the list of invalid expansion structures, thereby avoiding repeated loop expansion of the same initial loop structure in the neural network model.

[0013] One possible implementation includes: obtaining the i-th initial loop structure from the list of loop structures, where i is a natural number; determining the first loop count of the i-th initial loop structure, and performing loop expansion on the i-th initial loop structure based on the first loop count to obtain the i-th expanded loop structure; if the i-th expanded loop structure is found to be the same as the i-th initial loop structure, storing the i-th expanded loop structure in the list of invalid expanded structures; determining all initial loop structures in the neural network model to update the list of loop structures; and deleting the initial loop structure in the updated list that is the same as the i-th expanded loop structure in the list of invalid expanded structures.

[0014] In one possible implementation, if the number of loop nodes and the connection relationship of the loop nodes are the same in the initial loop structure and the expanded loop structure, the initial loop structure and the expanded loop structure are determined to be the same; if the number of loop nodes or the connection relationship of the loop nodes are different in the initial loop structure and the expanded loop structure, the initial loop structure and the expanded loop structure are determined to be different.

[0015] In one possible implementation, if the parameters of the conditional statement in the initial loop structure include non-constant numbers, the determined loop structure and the expanded loop structure are the same.

[0016] It is understandable that when the parameters of the conditional statement in the initial loop structure include non-constant numbers, the number of iterations of the initial loop structure cannot be determined offline, and therefore the initial loop structure cannot be expanded. Thus, determining the loop structure and expanding the loop structure are the same.

[0017] In one possible implementation, the conditional statement includes at least one of the IF statement and the IF ELSE statement.

[0018] In one possible implementation, the neural network model includes at least one of recurrent neural networks, gated recurrent units, long short-term memory networks, conditional generative models, and reinforcement learning models.

[0019] Secondly, embodiments of this application provide an apparatus including an acquisition module, an expansion module, and an update module. The acquisition module is used to acquire a neural network model to be processed and determine all initial loop structures in the neural network model. The expansion module is used to traverse the initial loop structures, determine the number of loops for each initial loop structure, and expand the initial loop structures according to the number of loops to obtain an expanded loop structure. The update module is used to update the initial loop structure to the expanded loop structure when the initial loop structure and the expanded loop structure are different, thereby obtaining the processed neural network model.

[0020] In this embodiment, the initial loop structure in the neural network model whose loop count can be determined offline is determined before the neural network model runs, and the initial loop structure is expanded cyclically based on the loop count. This avoids determining the loop count of the initial loop structure with a fixed number of loops during the operation of the neural network model, thereby reducing the complexity of the computation graph during the operation of the neural network model and reducing the memory occupied by the neural network model.

[0021] Thirdly, embodiments of this application provide an electronic device, including: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for implementing the processing method of any neural network model provided by the first aspect and various possible implementations of the first aspect.

[0022] Fourthly, embodiments of this application provide a computer-readable medium having instructions stored thereon, which, when executed, implement the processing method of any neural network model provided by the first aspect and various possible implementations of the first aspect. Attached Figure Description

[0023] Figure 1 According to an embodiment of this application, a schematic diagram of deploying a neural network model is shown;

[0024] Figure 2 According to an embodiment of this application, a schematic diagram of an initial loop structure and an expanded loop structure is shown;

[0025] Figure 3 According to an embodiment of this application, a first flowchart of a method for processing a neural network model is shown;

[0026] Figure 4 According to an embodiment of this application, a flowchart for determining the number of iterations of a loop structure is shown;

[0027] Figure 5 According to an embodiment of this application, a second flowchart of a method for processing a neural network model is shown.

[0028] Figure 6 According to an embodiment of this application, a third flowchart of a method for processing a neural network model is shown;

[0029] Figure 7 According to an embodiment of this application, a schematic diagram of the structure of a device 700 is shown;

[0030] Figure 8 According to an embodiment of this application, a schematic diagram of the structure of an electronic device 10 is shown. Detailed Implementation

[0031] The illustrative embodiments of this application include, but are not limited to, a method, apparatus, device, and medium for processing a neural network model.

[0032] The technical terms used in this application will be introduced below.

[0033] Loop structure: A loop structure is a control structure in programming languages ​​used to repeatedly execute a block of code. Loop structures include various types of loops, such as for loops, while loops, and do-while loops.

[0034] A loop node is an element in a flowchart or graphical programming language used to represent a loop in the program's control flow. A loop node indicates that the program will repeatedly execute a set of instructions until a certain condition is met. Loop nodes commonly appear in models such as recurrent neural networks (RNNs), gated recurrent units (GRUs), long short-term memory networks (LSTMs), conditional generative models, and reinforcement learning models. The output value of a loop node depends on the number of iterations and the computational logic within the loop node.

[0035] As you can understand, a loop node is a tool used to visualize and control loop structures in a programming environment.

[0036] The neural network model mentioned in the technical solution of this application can be a neural network model used in tasks such as image recognition, object detection, reinforcement learning, and semantic analysis. The types of neural network models are not limited to RNN, GRU, LSTM, conditional generative models, reinforcement learning models, Transformer (a model based on multi-head attention mechanism), etc.

[0037] In some embodiments, such as Figure 1 As shown, the neural network model 11 with a recurrent structure is first trained on the server 100. After the neural network model 11 is trained, the resulting model is further transferred to an inference model 12. Some variables in the structure of the inference model 12 may become constants. After the inference model 12 is deployed on the mobile phone 20, the mobile phone 20 can use the inference model 12 to recognize input data, such as image recognition. In some embodiments, the mobile phone 20 can also be replaced by any implementable terminal device such as a tablet computer or computer.

[0038] During image recognition of the input image by the mobile phone 20 based on inference model 12, the loop structure in inference model 12 has constant variables affecting the number of iterations, meaning the number of iterations is fixed. However, currently, during the execution of loop nodes, the loop continues based on the input image and the loop condition of the loop structure. This means the number of iterations is determined during the execution of inference model 12, which may lead to additional overhead, such as recursive calls and repeated large loops. This makes the execution order of some operators in inference model 12 uncertain, increasing the complexity of the computation graph of inference model 12, increasing the memory usage of inference model 12, and potentially causing unpredictable runtime delays due to the loop structure, leading to memory overflow issues.

[0039] To address the aforementioned problems, this application provides a method for processing neural network models. The method includes: before running the neural network model, determining the number of iterations of all initial loop structures in the model, and iteratively expanding the initial loop structures based on the iteration counts to obtain expanded loop structures. If the expanded loop structure differs from the initial loop structure, the initial loop structure is updated to the expanded loop structure, thereby simplifying the structure of the neural network model, reducing the computational load and memory usage during model execution, and improving the running efficiency of the neural network model.

[0040] It is understandable that loop unrolling can eliminate the branches in the loop structure and some code that manages inductive variables, thus amortizing some of the branch overhead of the loop structure. Furthermore, loop unrolling can actively schedule (or pipeline) loops to mask some of the delays.

[0041] In some embodiments, a method for determining the number of iterations in a loop structure includes: determining the expression of a conditional statement in the loop structure, wherein the conditional statement is used to determine whether the loop continues. The expression for the conditional statement contains formal parameters that determine the number of iterations. If all formal parameters determining the number of iterations are constants, meaning the number of iterations can be determined offline, then actual parameters are obtained and substituted into the loop structure to determine the number of iterations. If the formal parameters determining the number of iterations include non-constant numbers, meaning the number of iterations cannot be determined offline, and the loop structure cannot be expanded offline, the process ends.

[0042] For example, determine the expression of the conditional statement in the loop structure, such as the expression of the IF conditional statement, and then determine the formal parameters in the IF expression that determine the number of iterations. If the value of the formal parameters determining the number of iterations is not affected by the conditional branches of the IF conditional statement (i.e., the formal parameters determining the number of iterations can be determined offline, or all formal parameters determining the number of iterations are constants), then obtain the actual parameters and substitute them into the loop structure to determine the number of iterations. If the value of the formal parameters determining the output of the IF conditional statement is affected by the conditional branches of the IF conditional statement (i.e., the formal parameters determining the output of the conditional statement include non-constant numbers), then the formal parameters determining the number of iterations cannot be determined offline, and therefore the number of iterations in the loop structure cannot be determined offline, meaning the loop structure cannot be expanded offline, and the process ends.

[0043] In some embodiments, a method for determining whether an expanded loop structure is the same as an initial loop structure includes: determining whether the expanded loop structure is the same as the initial loop structure based on the number and connection relationships of the loop nodes. If the number or connection relationships of the loop nodes are different between the expanded loop structure and the initial loop structure, then the expanded loop structure is determined to be different from the initial loop structure. If both the number and connection relationships of the loop nodes are the same between the expanded loop structure and the initial loop structure, then the expanded loop structure is determined to be the same as the initial loop structure.

[0044] For example, such as Figure 2 As shown in (a), the initial loop structure includes four nodes: A, B, C, and D. Node A points to node B, node A points to node C, and nodes B and C point to node D. Figure 2 The expanded loop structure in (b) includes four nodes: A, B, C, and D. Node A points to node B, node B points to node C, and node C points to node D. That is, the connection relationship of the loop nodes in the expanded loop structure is different from that in the initial loop structure, thus confirming that the expanded loop structure is different from the initial loop structure.

[0045] It is understandable that the initial loop structure and the expanded loop structure are different, which means that the expanded loop structure is no longer a loop structure.

[0046] In some embodiments, the neural network model can be traversed to determine the initial loop structure in the neural network model, and the initial loop structure can be stored in list A (or loop structure list), and the initial loop structure can be read and expanded sequentially from list A.

[0047] If the expanded loop structure is the same as the initial loop structure, the expanded loop structure is stored in list B (or the list of invalid expanded structures). The neural network model is then re-traversed to update the initial loop structure recorded in list A and delete the loop structures in list A that are the same as the loop structure in list B. This avoids repeatedly expanding the same initial loop structure when expanding the initial loop structure recorded in list A again, thus improving processing efficiency.

[0048] For example, the neural network model is traversed for the first time, determining that the model includes initial loop structures Q, W, and E, and storing these structures in list A. First, the initial loop structure Q is iterated through to obtain the expanded loop structure q. Since the expanded loop structure q is different from the initial loop structure Q, the initial loop structure Q in the neural network model is updated to the expanded loop structure q.

[0049] A second traversal of the neural network model is performed to determine that the model includes an initial loop structure W and an initial loop structure E, which are then recorded in list A. First, the initial loop structure W is expanded to obtain an expanded loop structure W. Since the expanded loop structure W has the same structure as the initial loop structure W, it is stored in list B.

[0050] A third traversal is performed on the neural network model to determine that it includes initial loop structures W and E, which are then recorded in list A. If the expanded loop structure W in list B is found to be identical to the initial loop structures W and E in list A, the initial loop structures W and E recorded in list A are deleted.

[0051] It is understandable that when list A is empty, it means that all loop structures in the neural network model that have different structures before and after loop unrolling have been unrolled.

[0052] The processing method of the neural network model provided in the embodiments of this application will be described in detail below. The processing method of the neural network model in the embodiments of this application is applied to electronic devices. Figure 3 The diagram illustrates a processing method for a neural network model according to an embodiment of this application. The processing method for the neural network model includes:

[0053] 301: Obtain the neural network model to be processed and determine all the initial loop structures in the neural network model.

[0054] In this embodiment, each node of the neural network model can be traversed to determine all the loop structures in the neural network model, which are then used as the initial loop structure. The neural network model can be any neural network model such as RNN, GRU, LSTM, conditional generative model, or reinforcement learning model.

[0055] In some embodiments, the method for determining all initial loop structures in a neural network model includes: obtaining summary information of the neural network model to be processed, and determining all initial loop structures in the neural network model based on the summary information. The summary information includes the hierarchical structure and connection methods of the neural network model.

[0056] 302: Traverse the initial loop structure, determine the number of iterations for each initial loop structure, and expand the initial loop structure according to the number of iterations to obtain the expanded loop structure.

[0057] In this embodiment, the expression of the conditional statement in the initial loop structure can be determined first. The conditional statement determines whether the loop continues; it can be IF, IF ELSE, etc., and its output is true or false. Next, the formal parameters (or loop parameters) that determine the number of iterations in the conditional statement's expression are determined. If the formal parameters determining the number of iterations can be determined offline, the actual parameters are obtained and substituted into the loop structure to determine the number of iterations.

[0058] For example, determine the expression of the conditional statement in the loop structure, such as the expression of the conditional statement IF as (A, B, C). Then determine the formal parameters in the expression of the conditional statement that determine the number of iterations of the conditional statement. For example, in the conditional statement IF, A and B determine the number of iterations of IF, and C determines the calculation logic of IF. The formal parameters of A include m and n, and A is determined based on m and n; the formal parameters of B include p and q, and B is determined based on p and q; the formal parameters of C include r and t, and C is determined based on r and t. That is, the formal parameters that determine the number of iterations of the conditional statement IF include m, n, p, and q. When m, n, p, and q can be determined offline, that is, when m, n, p, and q are all constants, configure a loop counter, obtain the actual parameters of m, n, p, and q, and substitute them into the conditional statement IF to determine the number of iterations of the conditional statement IF.

[0059] As you can understand, a formal parameter (also called a placeholder parameter) is a parameter that appears in the function definition. It's a virtual parameter; no memory is allocated to it before the function is called, and it doesn't have an actual value in the function definition. The data is only received when the function is called. For the sake of function formality, it's called a formal parameter to represent the passed-in data. An actual parameter (also called a delimiter parameter) is a parameter that actually appears in the function. The value of an actual parameter can be a constant, variable, expression, class, etc. Actual parameters have definite values ​​and are placed in the same position as formal parameters.

[0060] In some embodiments, the loop parameter is used to determine the number of iterations of the initial loop structure and the final result value of the expanded loop structure.

[0061] 303: If the initial loop structure and the unrolled loop structure are different, update the initial loop structure to the unrolled loop structure to obtain the processed neural network model.

[0062] In this embodiment, it can be determined whether the expanded loop structure and the initial loop structure are the same based on the number and connection relationship of the loop nodes. If the number or connection relationship of the loop nodes in the expanded loop structure and the initial loop structure are different, then it is determined that the expanded loop structure and the initial loop structure are different. If the number and connection relationship of the loop nodes in the expanded loop structure and the initial loop structure are the same, then it is determined that the expanded loop structure and the initial loop structure are the same.

[0063] For example, such as Figure 2 As shown in (a), the initial loop structure includes four nodes: A, B, C, and D. Node A points to node B, node A points to node C, and nodes B and C point to node D. Figure 2 The expanded loop structure in (b) includes four nodes: A, B, C, and D. Node A points to node B, node B points to node C, and node C points to node D. That is, the connection relationship of the loop nodes in the expanded loop structure is different from that in the initial loop structure, thus confirming that the expanded loop structure is different from the initial loop structure.

[0064] It is understandable that the initial loop structure and the expanded loop structure are different, which means that the expanded loop structure is no longer a loop structure.

[0065] The embodiments of this application can determine the initial loop structure in the neural network model whose loop count can be determined offline before the neural network model runs, and expand the initial loop structure cyclically based on the loop count, thereby avoiding the need to determine the loop count of the initial loop structure with a fixed number of loops during the operation of the neural network model, thereby reducing the complexity of the computation graph during the operation of the neural network model and reducing the memory occupied by the neural network model.

[0066] The following section uses an initial loop structure in a neural network model, namely the A[0] loop structure, as an example to explain in detail the method for determining the number of loops in the initial loop structure. Figure 4 As shown, the methods for determining the number of iterations in the initial loop structure include:

[0067] 401: Determine if an IF node exists in the A[0] loop structure.

[0068] If the result is yes, meaning the parameters affecting the number of iterations of the loop structure can be determined by identifying the parameters of the condition node, then proceed to 402: find the formal parameters related to the IF condition; if the result is no, meaning the parameters affecting the number of iterations of the loop structure cannot be determined, then end.

[0069] In some embodiments, it can also be determined whether there are condition nodes such as IF ELSE nodes in the A[0] loop structure.

[0070] In this embodiment of the application, by determining whether there are condition nodes in the loop structure, the parameters affecting the number of iterations of the loop structure are determined, thus realizing the offline determination of the number of iterations of the loop structure.

[0071] 402: Identify the formal parameters related to the IF condition.

[0072] In the embodiments of this application, the formal parameters of the IF condition can be determined first, and then the formal parameters that affect the number of iterations of the IF condition can be determined.

[0073] For example, first, determine the expression of the IF condition node as (A, B, C). The formal parameters of A include m and n, and A is determined based on m and n; the formal parameters of B include p and q, and B is determined based on p and q; the formal parameters of C include r and t, and C is determined based on r and t. Second, determine the number of iterations of the IF condition node using A and B, and the calculation logic of the IF condition node using C. That is, determine the number of iterations of the IF condition statement using the formal parameters m, n, p, and q.

[0074] In some embodiments, the formal parameters of condition nodes such as IF ELSE nodes can be determined, and then the formal parameters that affect the number of iterations of the condition nodes can be determined.

[0075] 403: This error indicates whether the values ​​of all formal parameters are offline.

[0076] In this embodiment of the application, if the judgment result is yes, that is, all the formal parameters affecting the number of IF condition loops are determined offline, that is, all the formal parameters affecting the number of IF condition loops are constants, then proceed to 404: construct a new loop and introduce a loop calculator; if the judgment result is no, that is, at least one of the formal parameters affecting the number of IF condition loops is not determined offline, that is, when the formal parameters affecting the number of IF condition loops include non-constant numbers, the number of loops of the loop structure cannot be determined offline, and the process ends.

[0077] 404: Construct a new loop and introduce a loop counter.

[0078] In this embodiment, the new loop is the A[0] loop structure; the loop counter is used to calculate the number of iterations of the A[0] loop structure.

[0079] 405: Execute the new loop structure using actual parameters to obtain the actual number of loop iterations.

[0080] In this embodiment of the application, by substituting the actual parameters into the A[0] loop structure, the number of iterations of the A[0] loop structure can be determined, that is, the actual number of iterations of the A[0] loop structure.

[0081] It is understandable that since the parameters in the A[0] loop structure that affect the number of iterations of the IF condition are all constants, the number of iterations of the A[0] loop structure does not change with the change of the input value. Therefore, the actual number of iterations of the A[0] loop structure can be determined by substituting the actual parameters into the A[0] loop structure.

[0082] The embodiments of this application can determine the number of iterations of the loop structure offline or in advance, avoiding recursion during the process of the neural network model detecting input data, reducing the memory occupied by the neural network model during operation, and improving the operating efficiency of the neural network.

[0083] For some neural network models, the neural network model may include at least one identical loop structure. To improve the efficiency of loop unrolling in neural network models, embodiments of this application are based on... Figure 3 The neural network model processing method shown proposes a method for processing neural network models, such as... Figure 5 As shown, it includes:

[0084] 501: Find list A containing all loop structures.

[0085] In this embodiment, a neural network model to be processed can be obtained, and each node of the neural network model can be traversed to determine all loop structures (or initial loop structures) in the neural network model. All initial loop structures in the neural network model are then stored in list A. List A is used to store all initial loop structures in the neural network model.

[0086] 502: Create an empty list B.

[0087] In this embodiment of the application, list B is used to store invalid expanded structures, that is, expanded loop structures that are the same as the initial loop structure.

[0088] 503: Check if list A is empty.

[0089] If the judgment result is yes, that is, list A is empty, then the process ends; if the judgment result is no, then proceed to 504: take the first loop structure A[0] of list A.

[0090] 504: Take the first loop structure A[0] from list A.

[0091] In this embodiment of the application, the loop structures in list A can be traversed sequentially, such as first obtaining the first loop structure A[0] in list A that has a loop structure.

[0092] 505: Determine the actual number of iterations in the loop structure and perform loop expansion.

[0093] It is understandable that the method for determining the actual number of iterations in the loop structure and then unrolling the loop is... Figure 4 The embodiments shown are illustrated and will not be repeated here.

[0094] 506: Determine if the expanded loop structure is the same as the original loop structure. If the result is yes, go to 507: Record the invalid expanded structure in list B. If the result is no, update the original loop structure to the expanded loop structure and go to 508: Find the list A containing all loop structures.

[0095] In this embodiment, it can be determined whether the expanded loop structure is the same as the unexpanded one based on the number and connection relationships of the loop nodes between the expanded loop structure and the initial loop structure. If the number or connection relationships of the loop nodes in the expanded loop structure are different from those in the initial loop structure, then the expanded loop structure is determined to be different from the initial loop structure. If the number and connection relationships of the loop nodes in the expanded loop structure are the same as those in the initial loop structure, then the expanded loop structure is determined to be the same as the initial loop structure.

[0096] It is understandable that loop unrolling can eliminate branching and manage code for inductive variables. A loop structure is considered identical after unrolling if at least one of the following conditions is met: the loop structure has already been unrolled; or the loop structure contains non-constant numbers, making it impossible to unroll the loop structure.

[0097] 507: Record invalid expanded structures in list B.

[0098] In this embodiment of the application, for loop structures whose expanded loop structures are the same as those before expansion, the expanded loop structure is determined to be an invalid expanded structure, and the invalid expanded structure is recorded in list B.

[0099] 508: Find list A containing all loop structures.

[0100] In this embodiment, the neural network model is traversed again to update list A.

[0101] 509: Remove any loops in list B that occur in list A, and return to the previous state. 503: Determine if list A is empty.

[0102] This application embodiment improves processing efficiency by eliminating the loop structure of list B that appears in list A, thus avoiding the initial loop structure with the same structure being expanded repeatedly.

[0103] Based on the above Figures 3 to 5 The illustrated embodiment, taking a neural network model to be processed including initial loop structure Q, initial loop structure W, initial loop structure E, and initial loop structure R as an example, describes the processing method for the neural network model proposed in this application. Figure 6 As shown, it includes:

[0104] 601: Obtain the neural network model to be processed.

[0105] 602: Traverse the neural network model to be processed to determine the initial loop structure of the neural network model.

[0106] 603: Create list A and store all the initial loop structures of the neural network model in list A.

[0107] In this embodiment of the application, the initial loop structure of the neural network model includes an initial loop structure Q, an initial loop structure W, an initial loop structure E, and an initial loop structure R, which are stored in list A.

[0108] 604: Check if list A is empty. If the result is yes, end; if the result is no, go to 605: Read the i-th initial loop structure in list A and determine the number of iterations of the i-th initial loop structure.

[0109] 605: Read the i-th initial loop structure in list A and determine the number of iterations of the i-th initial loop structure.

[0110] In this embodiment of the application, the initial loop structure Q can be read first, and the number of iterations of the initial loop structure Q can be determined.

[0111] For example, first determine whether the initial loop structure Q includes a condition node. If the result is no, the process ends. If the result is yes, determine the formal parameters among the parameters of the condition node in the initial loop structure Q that affect the number of iterations of the condition node. If at least one of the formal parameters affecting the number of iterations of the condition node cannot be determined offline, the process ends. If all the formal parameters affecting the number of iterations of the condition node can be determined offline, configure a loop counter, obtain the actual parameters, and input the actual parameters into the initial loop structure Q to determine the number of iterations of the initial loop structure Q through the loop counter.

[0112] 606: Expand the i-th initial loop structure according to the number of iterations of the i-th initial loop structure to obtain the i-th expanded loop structure.

[0113] In this embodiment, the initial loop structure Q can be expanded by iterating through the number of iterations of the initial loop structure Q to obtain the expanded loop structure q. It can be understood that loop expansion can eliminate branches in the initial loop structure and some code that manages inductive variables, thus amortizing some branch overhead of the loop structure. Furthermore, loop expansion can actively schedule (or pipeline) the loop to mask some delays.

[0114] 607: Determine whether the i-th initial loop structure and the i-th expanded loop structure are the same, where i is a natural number.

[0115] If the judgment result is yes, then go to 608: store the i-th expanded loop structure in list B; if the judgment result is no, then go to 609: update the i-th initial loop structure to the i-th expanded loop structure.

[0116] It is understandable that when all formal parameters affecting the number of iterations of the condition nodes in the initial loop structure Q can be determined offline (or in other words, all formal parameters affecting the number of iterations of the condition nodes in the initial loop structure Q are constants), and the initial loop structure Q includes branches that do not affect the number of iterations of the initial loop structure Q, as well as some code managing inductive variables, the initial loop structure Q and the expanded loop structure q are different. When at least one formal parameter affecting the number of iterations of the condition nodes in the initial loop structure Q cannot be determined offline (or in other words, the formal parameters affecting the number of iterations of the condition nodes in the initial loop structure Q include non-constant numbers), or when the initial loop structure Q does not include branches affecting the number of iterations of the initial loop structure Q, as well as some code managing inductive variables, the initial loop structure Q and the expanded loop structure q are the same.

[0117] 608: Store the i-th expanded loop structure in list B.

[0118] 609: Update the i-th initial loop structure to the i-th expanded loop structure.

[0119] In this embodiment, the initial loop structure Q and the expanded loop structure q are different, so the initial loop structure Q is updated to the expanded loop structure q.

[0120] It can be understood that the initial loop structure Q and the expanded loop structure q are not the same, which means that the expanded loop structure q is no longer a loop structure.

[0121] 610: Determine the initial loop structure of the neural network model and store the initial loop structure in list A.

[0122] In this embodiment of the application, whenever an initial loop structure in list A is iterated and expanded, the initial loop structure of the neural network model is determined and stored in list A.

[0123] For example, since the expanded loop structure q is no longer a loop structure, the neural network model is traversed a second time to determine that list A includes the initial loop structure W, the initial loop structure E, and the initial loop structure R.

[0124] 611: Delete the initial loop structure in list A that is the same as the expanded loop structure in list B, and return 603: Determine if list A is empty.

[0125] In this embodiment, after the second traversal of the neural network model, the initial loop structure W can be expanded to obtain the expanded loop structure W. Since the expanded loop structure W and the initial loop structure W have the same structure, the expanded loop structure W can be stored in list B.

[0126] Then, the neural network model is traversed a third time to determine that it includes initial loop structures W, E, and R, and these initial loop structures are recorded in list A. If the expanded loop structure W in list B is found to be the same as the initial loop structures W and E in list A, then the initial loop structures W and E recorded in list A are deleted. That is, list A includes the initial loop structure R. The initial loop structure R is then iteratively expanded to obtain the expanded loop structure r. Since the expanded loop structure r is different from the initial loop structure R, the initial loop structure R is updated to the expanded loop structure r.

[0127] The neural network model is traversed for the fourth time to determine if it includes initial loop structures W and E, which are then recorded in list A. If the expanded loop structure W in list B is identical to the initial loop structures W and E in list A, the initial loop structures W and E recorded in list A are deleted. That is, list A is empty, and the process ends.

[0128] This application embodiment realizes the determination of the initial loop structure in the neural network model whose loop count can be determined offline before the neural network model runs, and the initial loop structure is expanded cyclically based on the loop count. This avoids determining the loop count of the initial loop structure with a constant loop count during the operation of the neural network model. Furthermore, this application embodiment avoids repeatedly expanding the same loop structure by deleting loop structures in list B that have the same expanded loop structure as the initial loop structure, thereby improving the processing efficiency of the neural network model.

[0129] like Figure 7 As shown in the figure, this application embodiment also provides an apparatus 700, including an acquisition module 701, an expansion module 702 and an update module 703.

[0130] The acquisition module 701 is used to acquire the neural network model to be processed and determine all the initial loop structures in the neural network model.

[0131] The expansion module 702 is used to traverse the initial loop structure, determine the number of iterations for each initial loop structure, and expand the initial loop structure according to the number of iterations to obtain the expanded loop structure.

[0132] The update module 703 is used to update the initial loop structure to the expanded loop structure when the initial loop structure and the expanded loop structure are different, so as to obtain the processed neural network model.

[0133] In this embodiment, the initial loop structure in the neural network model whose loop count can be determined offline is determined before the neural network model runs, and the initial loop structure is expanded cyclically based on the loop count. This avoids determining the loop count of the initial loop structure with a fixed number of loops during the operation of the neural network model, thereby reducing the complexity of the computation graph during the operation of the neural network model and reducing the memory occupied by the neural network model.

[0134] This application provides an electronic device, including: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for executing instructions to implement the processing method of the neural network model described above.

[0135] This application provides a computer-readable medium storing instructions that, when executed, implement the processing method of the aforementioned neural network model.

[0136] Figure 8 According to some embodiments of this application, a schematic diagram of the structure of an electronic device 10 is shown. For example... Figure 8 As shown, the electronic device 10 includes one or more processors 101, system memory 102, non-volatile memory (NVM) 103, communication interface 104, input / output (I / O) devices 105, and system control logic 106 for coupling the processor 101, system memory 102, NVM 103, communication interface 104, and input / output (I / O) devices 105. Wherein:

[0137] Processor 101 can be used to control an electronic device to execute the processing method of the neural network model of this application. Processor 101 may include one or more processing units, such as a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), microprocessor (MCU), artificial intelligence (AI) processor, or a field programmable gate array (FPGA) processing module or processing circuit, which may include one or more single-core or multi-core processors. System memory 102 is volatile memory, such as random-access memory (RAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc. System memory is used for temporary storage of data and / or instructions.

[0138] The non-volatile memory 103 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 103 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), a compact disc (CD), a digital versatile disc (DVD), a solid-state drive (SSD), etc. In some embodiments, the non-volatile memory 103 may also be a removable storage medium, such as a secure digital (SD) memory card, etc.

[0139] Specifically, system memory 102 and non-volatile memory 103 may each include a temporary copy and a permanent copy of instruction 107. Instruction 107 may include a processing method that, when executed by processor 101, causes electronic device 10 to implement the neural network model provided in the embodiments of this application.

[0140] The communication interface 104 may include a transceiver for providing a wired or wireless communication interface for the electronic device 10, thereby enabling communication with any other suitable device via one or more networks. In some embodiments, the communication interface 104 may be integrated into other components of the electronic device 10, for example, the communication interface 104 may be integrated into the processor 101. In some embodiments, the electronic device 10 may communicate with other devices through the communication interface 104; for example, the electronic device 10 may obtain the processing method of a neural network model to be run from other electronic devices through the communication interface 104.

[0141] Input / output (I / O) device 105 may include input devices such as keyboard, mouse, etc., and output devices such as monitor, etc. Users can interact with electronic device 10 through input / output (I / O) device 105.

[0142] System control logic 106 may include any suitable interface controller to provide any suitable interface to other modules of electronic device 10. For example, in some embodiments, system control logic 106 may include one or more memory controllers to provide an interface to system memory 102 and non-volatile memory 103.

[0143] In some embodiments, at least one of the processors 101 may be packaged together with the logic of one or more controllers for system control logic 106 to form a system in package (SiP). In other embodiments, at least one of the processors 101 may also be integrated on the same chip with the logic of one or more controllers for system control logic 106 to form a system on chip (SoC).

[0144] It is understood that electronic device 10 can be any electronic device or terminal device capable of running neural networks, including but not limited to mobile phones, wearable devices (such as smartwatches), tablets, desktops, laptops, handheld computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, etc., which are not limited in the embodiments of this application.

[0145] Understandable. Figure 8The structure of the electronic device 10 shown is merely an example. In other embodiments, the electronic device 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0146] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0147] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0148] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0149] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, compact disc-read-only memory (CD-ROMs), magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagated signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0150] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0151] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0152] It should be noted that, in the examples and description of this patent, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0153] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. A method for processing a neural network model, characterized in that, Applied to electronic devices, the method includes: Obtain the neural network model to be processed and determine all the initial loop structures in the neural network model, which is used to perform image recognition tasks or object detection tasks on the electronic device; Traverse the initial loop structure, determine the number of iterations for each initial loop structure, and expand the initial loop structure according to the number of iterations to obtain the expanded loop structure; If the initial loop structure and the expanded loop structure are determined to be different, the initial loop structure is updated to the expanded loop structure to obtain the processed neural network model. Before traversing the initial loop structure, the process includes: creating a loop structure list, which is used to store the initial loop structure; Create an invalid expansion structure list, which stores loop structures whose expanded loop structures are the same as the initial loop structure; Traversing the initial loop structure includes: sequentially reading the initial loop structures from the loop structure list; wherein, after reading the i-th initial loop structure from the loop structure list: Determine the first iteration number of the i-th initial loop structure, and perform loop expansion on the i-th initial loop structure according to the first iteration number to obtain the i-th expanded loop structure, where i is a natural number; If it is determined that the i-th expanded loop structure is the same as the i-th initial loop structure, the i-th expanded loop structure is stored in the list of invalid expanded structures. Determine all initial loop structures in the neural network model to update the loop structure list; Delete the initial loop structure in the updated loop structure list that is identical to the i-th expanded loop structure in the invalid expanded structure list; Continue until the list of loop structures is empty.

2. The method for processing neural network models according to claim 1, characterized in that, Determining the number of iterations of the initial loop structure includes: If all parameters of the conditional statements in the initial loop structure are constants, obtain the actual parameters of the initial loop structure. The actual parameters are input into the initial loop structure to determine the number of iterations of the initial loop structure.

3. The method for processing neural network models according to claim 2, characterized in that, A method for determining whether all parameters of the conditional statement in the initial loop structure are constants includes: The expression that determines the conditional statement of the initial loop structure; Determine the loop parameter of the expression, the loop parameter being used to determine the number of iterations of the initial loop structure; If the loop parameters can be determined offline, then the parameters of the conditional statements that determine the initial loop structure are all constants; If the loop parameters cannot be determined offline, then the parameters of the conditional statement that determines the initial loop structure include non-constant numbers.

4. The method for processing neural network models according to claim 2, characterized in that, The step of inputting the actual parameters into the initial loop structure to determine the number of iterations of the initial loop structure includes: Configure a loop counter; The actual parameters are input into the initial loop structure, and the loop counter determines the number of iterations of the initial loop structure.

5. The method for processing neural network models according to claim 1, characterized in that, include: If the number of loop nodes and the connection relationship of loop nodes are the same in the initial loop structure and the expanded loop structure, then the initial loop structure and the expanded loop structure are determined to be the same. If the number of loop nodes or the connection relationship of loop nodes are different between the initial loop structure and the expanded loop structure, it is determined that the initial loop structure and the expanded loop structure are different.

6. The method for processing neural network models according to claim 1, characterized in that, include: If the parameter of the conditional statement in the initial loop structure includes a non-constant number, then the loop structure and the expanded loop structure are determined to be the same.

7. The method for processing neural network models according to claim 6, characterized in that, The conditional statement includes at least one of the IF statement and the IF ELSE statement.

8. The method for processing a neural network model according to any one of claims 1-7, characterized in that, The neural network model includes at least one of the following: recurrent neural network, gated recurrent unit, long short-term memory network, conditional generation model, and reinforcement learning model.

9. A processing device for a neural network model, characterized in that, This includes the acquisition module, expansion module, and update module. The acquisition module is used to acquire the neural network model to be processed and determine all the initial loop structures in the neural network model, which is used to perform image recognition tasks or object detection tasks. The expansion module is used to traverse the initial loop structure, determine the number of iterations for each initial loop structure, and expand the initial loop structure according to the number of iterations to obtain the expanded loop structure. Create a list of loop structures to store the initial loop structure; Create an invalid expansion structure list, which stores loop structures whose expanded loop structures are the same as the initial loop structure; When traversing the initial loop structure, the unrolling module is used to sequentially read the initial loop structure from the loop structure list; and after reading the i-th initial loop structure from the loop structure list, the unrolling module is also used to: Determine the first iteration number of the i-th initial loop structure, and perform loop expansion on the i-th initial loop structure according to the first iteration number to obtain the i-th expanded loop structure, where i is a natural number; If it is determined that the i-th expanded loop structure is the same as the i-th initial loop structure, the i-th expanded loop structure is stored in the list of invalid expanded structures. The update module is configured to update the initial loop structure to the expanded loop structure when it is determined that the initial loop structure and the expanded loop structure are different. And delete the initial loop structure that is the same as the invalid unrolled structure in the updated loop structure list; The update module is further configured to, after the i-th initial loop structure is expanded, delete the initial loop structure in the updated loop structure list that is the same as the i-th expanded loop structure in the invalid expanded structure list.

10. An electronic device, characterized in that, It includes: a memory for storing instructions executed by one or more processors of the electronic device, and the processor, being one of the one or more processors of the electronic device, for executing the instructions to implement a processing method for a neural network model as described in any one of claims 1 to 8.

11. A computer-readable medium having instructions stored thereon, characterized in that, When the instruction is executed, it implements the processing method of the neural network model according to any one of claims 1 to 8.

Images